Weekly TMLR digest for Jul 05, 2026
TMLR
New certifications
==================
J2C Certification: Rethinking Reward Models for Multi-Domain Test-Time Scaling
Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song
https://openreview.net/forum?id=PgouBhL7IR
---
Survey Certification: Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries
Yubo Li
https://openreview.net/forum?id=cjU3YbcRr8
---
J2C Certification: Bigger Isn’t Always Memorizing: Early Stopping Overparameterized Diffusion Models
Alessandro Favero, Antonio Sclocchi, Matthieu Wyart
https://openreview.net/forum?id=P6r9OxZ1vm
---
J2C Certification: On Rate-Optimal Partitioning Classification from Observable and from Privatised Data
Balázs Csáji, László Györfi, Ambrus Tamás, Harro Walk
https://openreview.net/forum?id=KYYvIrtgK0
---
Expert Certification: Plain Transformers Can be Powerful Graph Learners
Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip Torr, Mark Coates
https://openreview.net/forum?id=bEmDvP0fdv
---
J2C Certification: Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection
Yihao Xue, Kristjan Greenewald, Youssef Mroueh, Baharan Mirzasoleiman
https://openreview.net/forum?id=6tlLISSgiu
---
Reproducibility Certification, J2C Certification: Towards Principled Task Grouping for Multi-Task Learning
Chenguang Wang, Xuanhao Pan, Tianshu Yu
https://openreview.net/forum?id=3DeSIpzuro
---
Expert Certification: What do near-optimal learning rate schedules look like?
Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg, George E. Dahl
https://openreview.net/forum?id=pEsAMnmq0L
---
Accepted papers
===============
Title: A quantitative analysis of semantic information in deep representations of text and images
Authors: Santiago Acevedo, Andrea Mascaretti, Riccardo Rende, Matéo Mahaut, Marco Baroni, Alessandro Laio
Abstract: It was recently observed that the representations of different models that process identical or semantically related inputs tend to align. We analyze this phenomenon using the Information Imbalance, an asymmetric rank-based measure that quantifies the capability of a representation to predict another, providing a proxy of the cross-entropy which can be computed efficiently in high-dimensional spaces. By measuring the Information Imbalance between representations generated by DeepSeek-V3 processing translations, we find that semantic information is spread across many tokens, and that semantic predictability is strongest in a set of central layers of the network, robust across six language pairs. We measure clear information asymmetries: English representations are systematically more predictive than those of other languages, and DeepSeek-V3 representations are more predictive of those in a smaller model such as Llama3-8b than the opposite. In the visual domain, we observe that semantic information concentrates in middle layers for autoregressive models and in final layers for encoder models, and these same layers yield the strongest cross-modal predictability with textual representations of image captions. Our results support the hypothesis of semantic convergence across languages, modalities, and architectures, while showing that directed predictability between representations varies strongly with layer-depth, model scale, and language.
URL: https://openreview.net/forum?id=sBnaFSIuGR
---
Title: Learning to Prompt for Generalizable Instance Segmentation via Bi-Level Optimization
Authors: Li Zhang, Pengtao Xie
Abstract: The Segment Anything Model has revolutionized image segmentation with its zero-shot capabilities, yet its reliance on manual prompts hinders fully automated deployment. While integrating object detectors as prompt generators offers a pathway to automation, existing pipelines suffer from two fundamental limitations: objective mismatch, where detectors optimized for geometric localization do not correspond to the optimal prompting context required by SAM, and alignment overfitting in standard joint training, where the detector simply memorizes specific prompt adjustments for training samples rather than learning a generalizable policy. To bridge this gap, we introduce BLO-Inst, a unified framework that aligns detection and segmentation objectives by bi-level optimization. We formulate the alignment as a nested optimization problem over disjoint data splits. In the lower level, the SAM is fine-tuned to minimize segmentation loss given the current detection proposals on a subset ($D_1$). In the upper level, the detector is updated to generate bounding boxes that explicitly minimize the validation loss of the fine-tuned SAM on a separate subset ($D_2$). This effectively transforms the detector into a segmentation-aware prompt generator, optimizing the bounding boxes not just for localization accuracy, but for downstream mask quality. Extensive experiments demonstrate that BLO-Inst achieves superior performance, outperforming standard baselines on tasks in general and biomedical domains.
URL: https://openreview.net/forum?id=zN1yKIIVxN
---
Title: Rethinking Reward Models for Multi-Domain Test-Time Scaling
Authors: Dong Bok Lee, Seanie Lee, Sangwoo Park, Minki Kang, Jinheon Baek, Dongki Kim, Dominik Wagner, Jiongdao Jin, Heejun Lee, Tobias Bocklet, Jinyu Wang, Jingjing Fu, Sung Ju Hwang, Jiang Bian, Lei Song
Abstract: The reliability of large language models (LLMs) during test-time scaling is often assessed with external verifiers or reward models that distinguish correct reasoning from flawed logic. Prior work has studied both outcome reward models (ORMs), which assess only the final answer, and process reward models (PRMs), which score intermediate reasoning steps. Although PRMs are often viewed as advantageous due to their finer-grained supervision, much of the supporting evidence comes from math-adjacent settings, and their relative benefits across broader domains remain unclear. We present the first unified evaluation of four reward model variants, discriminative ORM and PRM (DisORM, DisPRM) and generative ORM and PRM (GenORM, GenPRM), across 14 diverse domains. Contrary to conventional wisdom, we find that (i) DisORM performs on par with DisPRM, (ii) GenPRM is not competitive, and (iii) overall, GenORM is the most robust, yielding significant and consistent gains across every tested domain. We attribute this to PRM-style stepwise scoring, which inherits label noise from LLM auto-labeling and has difficulty evaluating long reasoning trajectories, including those involving self-correcting reasoning. Our theoretical analysis shows that step-wise aggregation compounds errors as reasoning length grows, and our empirical observations confirm this effect. These findings challenge the prevailing assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. We publicly release our code at this https://github.com/db-Lee/Multi-RM to facilitate future research in multi-domain settings.
URL: https://openreview.net/forum?id=PgouBhL7IR
---
Title: Beyond Freezing the Router: Rank-Aligned Post-Training Quantization for Mixture-of-Experts Models
Authors: Yi-Zeng Fang, Juinn-Dar Huang
Abstract: Quantizing Mixture-of-Experts language models remains a challenging problem because quantization noise propagates across layers and distorts downstream expert selection. Although common practice keeps the router in full precision, we show that this strategy is insufficient: quantization-induced errors in expert outputs still shift the logits of the next-layer router, and freezing the router removes the opportunity to compensate for these shifts. Motivated by this finding, we propose RouteQuant, a post-training quantization framework that explicitly embraces router quantization to correct for expert-level distortion. We analyze how quantization alters router rankings and rank flips, and provide a theoretical proof showing that deviations in expert outputs are bounded by both expert-selection and gap-preservation errors. These insights motivate two router-alignment objectives: (i) Rank-Aware Jaccard Loss, which aligns the top-$k$ routing sets between full-precision and quantized models, and (ii) Gap Hinge Loss, which preserves the margin between consecutive expert logits to suppress rank flipping. In addition to router alignment, we further introduce Expert-Aware Smoothing Factor, which assigns separate activation smoothing factors to heterogeneous experts. Across OLMoE, DeepSeek-MoE, and Qwen3-MoE, RouteQuant consistently improves perplexity on C4 and WikiText-2 and enhances zero-shot accuracy under W4A4 and W4A8 across diverse downstream tasks, demonstrating the effectiveness of the proposed framework.
URL: https://openreview.net/forum?id=bPsPPI65hf
---
Title: Provable Emergence of Deep Neural Collapse and Low-Rank Bias in $L^2$-Regularized Nonlinear Networks
Authors: Emanuele Zangrando, Piero Deidda, Simone Brugiapaglia, Nicola Guglielmi, Francesco Tudisco
Abstract: We present a unified theoretical framework connecting the first property of Deep Neural Collapse (DNC1) to the emergence of implicit low-rank bias in nonlinear networks trained with $L^2$ weight decay regularization. Our main contributions are threefold. First, we derive a quantitative relation between the Total Cluster Variation (TCV) of intermediate embeddings and the numerical rank of stationary weight matrices. In particular, we establish that, at any critical point, the distance from a weight matrix to the set of rank-$K$ matrices is bounded by a constant times the TCV of earlier-layer features, scaled inversely with the weight-decay parameter.
Second, for invertible nonlinearities, we prove global optimality of DNC1 in a constrained representation-cost setting for both feedforward and residual architectures, showing that zero TCV across intermediate layers minimizes the representation cost under natural architectural constraints.
Third, we establish a benign landscape property: for almost every interpolating initialization there exists a continuous, loss-decreasing path from the initialization to a globally optimal, DNC1-satisfying configuration. Our theoretical claims are validated empirically; numerical experiments confirm the predicted relations among TCV, singular-value structure, and weight decay. These results indicate that neural collapse and low-rank bias are intimately linked phenomena arising from the optimization geometry induced by weight decay.
URL: https://openreview.net/forum?id=PvmFUzchzY
---
Title: DP-FedSOFIM: Differentially Private Federated Stochastic Optimization using Regularized Fisher Information Matrix
Authors: Sidhant Nair, Tanmay Sen, Mrinmay Sen, Sayantan Banerjee
Abstract: Differentially private federated learning (DP-FL) often suffers from slow convergence under tight privacy budgets because the noise required for privacy preservation degrades gradient quality. Although second-order optimization can accelerate training, existing approaches for DP-FL face significant scalability limitations: Newton-type methods require clients to compute Hessians, while feature covariance methods scale poorly with model dimension. We propose \textbf{DP-FedSOFIM}, a simple and scalable second-order optimization method for DP-FL. The method constructs an online regularized proxy for the Fisher information matrix at the server using only privatized aggregated gradients, capturing useful curvature information without requiring Hessian computations or feature covariance estimation. Efficient rank-one updates based on the Sherman-Morrison formula enable communication costs proportional to the model size and require only $O(d)$ client-side memory. Because all curvature and preconditioning operations are performed at the server on already privatized gradients, \textbf{DP-FedSOFIM} introduces no additional privacy cost beyond the underlying privatized gradient release mechanism. Experiments on CIFAR-10 and PathMNIST show that \textbf{DP-FedSOFIM} converges faster and consistently achieves higher accuracy than DP-FedGD, DP-SCAFFOLD, and DP-FedFC across a range of privacy budgets, with particularly pronounced gains under stringent privacy constraints.
URL: https://openreview.net/forum?id=aDzj9DrwAR
---
Title: How Private is Your Attention? Bridging Privacy with In-Context Learning
Authors: Soham Bonnerjee, Kingsley Yeon, Anna Asch, Sagnik Nandy, PROMIT GHOSAL
Abstract: In-context learning (ICL)—the ability of transformer-based models to perform new tasks from examples provided at inference time—has emerged as a hallmark of modern language models. While recent works have investigated the mechanisms underlying ICL, its feasibility under formal privacy constraints remains largely unexplored. In this paper, we propose a differentially private pretraining algorithm for linear attention heads and present the first theoretical analysis of the privacy–accuracy trade-off for ICL in linear regression. Our results characterize the fundamental tension between optimization and privacy-induced noise, formally capturing behaviors observed in private training via iterative methods. Additionally, we show that our method is robust to adversarial perturbations of training prompts, unlike standard ridge regression. All theoretical findings are supported by extensive simulations across diverse settings.
URL: https://openreview.net/forum?id=M2qsrIba0L
---
Title: DuoShapley: Adaptive and Scalable Shapley Value Approximation for Federated Learning
Authors: Mengwei Yang, Baturalp Buyukates, Yanning Shen, Athina Markopoulou
Abstract: Federated Learning (FL) enables collaborative model training across decentralized users while preserving data privacy, but it also raises a fundamental challenge: how to efficiently and reliably quantify individual user contributions to the global model. The Shapley value (SV) provides a principled game-theoretic framework for contribution valuation, yet its exact computation is prohibitively expensive in realistic FL systems. Existing SV approximation methods face a trade-off between scalability and estimation fidelity, particularly under heterogeneous data distributions. In this work, we propose DuoShapley, an efficient and adaptive SV approximation tailored to large-scale FL that adaptively balances two complementary orders: Solo, capturing individual contributions, and Leave-One-Out (LOO), capturing marginal contributions relative to the full coalition. By adaptively weighting them during training based on the alignment between local and global model updates, DuoShapley achieves both computational efficiency and accurate contribution valuation across diverse FL scenarios, from independent and identically distributed (IID) to non-IID. Beyond contribution measurement, DuoShapley enables downstream applications such as robust user selection in the presence of users with noisy data, by prioritizing users with high estimated contributions. Such selective participation leads to enhanced robustness to noisy and low-quality updates, and reduced communication overhead. Extensive experiments show that DuoShapley is both computationally efficient and effective across diverse data distributions. Hence, DuoShapley provides a practical and scalable solution for evaluating and leveraging user contributions in FL.
URL: https://openreview.net/forum?id=zjgZFNEEHn
---
Title: Forking-Sequences: Statistically and Computationally Efficient Multi-Horizon Forecasting with Reduced Volatility
Authors: Willa Potosnak, Malcolm Wolff, Mengfei Cao, Ruijun Ma, Tatiana Konstantinova, Dmitry Efimov, Michael W. Mahoney, Boris N. Oreshkin, Kin G. Olivares
Abstract: While accuracy is a critical requirement for time series forecasting, an equally important desideratum is reasonable forecast volatility across forecast creation dates (FCDs). Even highly accurate models can produce erratic revisions between FCDs, undermining trust and disrupting downstream decision-making. To improve the volatility of forecast revisions, state-of-the-art models like MQCNN, MQT, and SPADE employ a powerful yet underexplored neural network architectural design: forking-sequences. This architectural design jointly encodes and decodes the entire time series across all FCDs, producing an entire multi-horizon forecast grid in a single forward pass. This approach contrasts with conventional neural forecasting methods that process FCDs independently, generating only a single multi-horizon forecast per forward pass. In this work, we formalize the forking-sequences design and motivate its broader adoption by introducing a metric for quantifying excess volatility in forecast revisions and by providing theoretical and empirical analysis. We theoretically motivate three key benefits of forking-sequences: (i) reduced forecast volatility through ensembling; (ii) gradient variance reduction, improving the statistical efficiency of the training procedure; and (iii) improved inference computational efficiency. We validate the benefits of forking-sequences
compared to baseline window-sampling on the M-series benchmark, using 16 datasets from the M1, M3, M4, and Tourism competitions. We observe median sCRPS improvements across datasets of 46.2%, 49.3%, 28.6%, 24.7%, and 6.4% for RNN, LSTM, CNN, Transformer, and State Space-based architectures, respectively. We then show that forecast ensembling during inference can reduce median forecast volatility by 13.2%, 13.0%, 10.9%, 10.2%, and 11.2% for these respective models trained with forking-sequences, while maintaining accuracy.
URL: https://openreview.net/forum?id=dXdycy7WCX
---
Title: Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries
Authors: Yubo Li
Abstract: Large language model agents increasingly store reusable procedures outside the model. These reusable procedures are often called \emph{skills}: they may be code functions, natural-language instructions, SKILL.md packages, workflow graphs, or learned adapters that a future agent can retrieve and invoke. This taxonomy-driven survey asks how such skill libraries change over time. Across a $124$-paper $2023$--$2026$ audit set, we synthesize dynamic skill systems as \emph{lifecycle-managed, verified, evolving artifact stores}: agents collect evidence from interaction, propose skill updates, verify and admit candidates, organize them for retrieval and composition, repair or prune stale entries, and govern sharing through provenance and rollback. We organize the literature around three survey tools. First, a $\text{six}$-sense taxonomy distinguishes the structurally different artifacts called ``skills'' in current papers. Second, an $\text{eight}$-stage lifecycle architecture identifies the recurring design decisions behind evidence acquisition, proposal, verification/admission, storage, retrieval/composition, maintenance, distillation/portability, and governance. Third, a lightweight skill-record schema and $\text{ten}$-operator vocabulary provide common terms for comparing library updates without elevating them into a separate method contribution. Using this structure, we synthesize evidence-graded patterns with explicit caveats: admission and repair are repeatedly important, verifier quality materially affects skill-aware RL, flat retrieval can degrade as libraries grow, and current benchmarks still under-report library trajectories, usage--utility gaps, and safety surfaces. We close with concrete reporting standards and open problems for evaluating dynamic skills as changing libraries rather than static prompt or tool collections.
URL: https://openreview.net/forum?id=cjU3YbcRr8
---
Title: Monte Carlo Multi-Feature Baseline Shapley (MMBS): An axiomatic attribution method for fine-grained explanations of image classification networks
Authors: Dirk Elias Schut, Robert van Liere, Tristan van Leeuwen
Abstract: This paper presents the Multi-Feature Baseline Shapley (MBS) attribution method for explaining the outcome of a neural network for a given input. MBS generalizes the Integrated Gradients (IG) and Baseline Shapley (BShap) methods by introducing a step size parameter. When the step size is set to one, MBS equals BShap, and when it is set to the number of features, MBS equals IG. MBS is an axiomatic method, which means that it was designed to satisfy certain axioms (mathematical properties). These axioms ensure that the attribution maps relate to the neural network in appealing ways, for example, by preserving linearity or symmetry. We prove that MBS satisfies eight axioms that are also satisfied by IG and BShap. To quickly approximate MBS, this paper presents the Monte Carlo Multi-Feature Baseline Shapley (MMBS) method, which is an unbiased estimator of MBS. On image classification tasks, we show that MMBS also approximates a Monte Carlo estimate for BShap while being orders of magnitude faster to compute. Furthermore, we compare MMBS to nine configurations of existing attribution methods on three image classification networks trained on either the Fashion MNIST or ImageNet1k dataset. MMBS has the best area under the deletion curve score on all three networks.
URL: https://openreview.net/forum?id=LLFIcr7zWh
---
Title: Training Verifiably Robust Agents Using Set-Based Reinforcement Learning
Authors: Manuel Wendl, Lukas Koller, Tobias Ladner, Matthias Althoff
Abstract: Reinforcement learning policies parametrized by deep neural networks have achieved strong performance for continuous control, yet even small input perturbations may lead to unpredictable behavior. This sensitivity limits their use in safety-critical domains, where robustness guarantees are required. Our work addresses this gap between state-of-the-art adversarial training methods and formal verification to train verifiably robust agents. Previous works train networks with individual adversarial perturbations, making them only robust against the specific adversarial attacks used. In contrast, our approach propagates entire perturbed input sets, enclosing all possible adversarial attacks within a single network pass. We leverage this to explicitly penalize the size of the output set (minimizing closed-loop uncertainty) and thereby make the actor robust against all possible attacks. This is realized by the use of set-based policy gradients, where each output within the set has a different gradient, thereby balancing the accuracy and robustness of the network. Doing so, we achieve formal verifiability across different verification frameworks for up to 9 times larger input perturbations compared to standard reinforcement learning and improve certified worst-case performance.
URL: https://openreview.net/forum?id=DhRPPhSjGA
---
Title: From Arithmetic to Logic: The Resilience of Logic and Lookup-Based Neural Networks Under Parameter Bit-Flips
Authors: Alan T. L. Bacellar, Sathvik Chemudupati, Shashank Nag, Allison Seigler, Priscila Machado Vieira Lima, Felipe M.G. França, Lizy Kurian John
Abstract: The deployment of Deep Neural Networks (DNNs) in safety-critical edge environments necessitates robustness against hardware-induced bit-flip errors. While empirical studies indicate that reducing Numerical Precision can improve fault tolerance, the theoretical basis of this phenomenon remains underexplored. In this work, we study resilience as a Structural Property of neural architectures rather than solely as a property of a dataset-specific trained solution. By deriving the Expected Squared Error (MSE) under independent parameter bit flips across multiple numerical formats and layer primitives, we show that lower precision, higher sparsity, bounded activations, and shallow depth are consistently favored under this corruption model. We then argue that Logic and Lookup-Based Neural Networks realize the joint limit of these design trends. Through ablation studies on the MLPerf Tiny benchmark suite, we show that the observed empirical trends are consistent with the theoretical predictions, and that LUT-based models remain highly stable in corruption regimes where standard floating-point models fail sharply. Furthermore, we identify a novel Even-Layer Recovery effect unique to logic-based architectures and analyze the structural conditions under which it emerges. Overall, our results suggest that shifting from continuous arithmetic weights to discrete Boolean lookups can provide a favorable Accuracy--Resilience trade-off for hardware fault tolerance.
URL: https://openreview.net/forum?id=ZZYvGZei5h
---
Title: Visual Explanations for Capsule Networks
Authors: Saja Tawalbeh, Jose Oramas
Abstract: The limited availability of explainability methods for Capsule Networks (CapsNets) restricts their adoption in critical domains such as clinical practice or legal document analysis. Although CapsNets offer structured and interpretable representations, existing explanation methods have primarily focused on more traditional Convolutional Neural Networks (CNNs) and are not directly applicable to capsule-based architectures. To address this issue, we propose a general method (Caps-CAM), which generates attribution maps to justify the predictions made by feed-forward CapsNet architectures. Unlike prior explanation methods for CapsNets that adapt techniques originally designed for CNNs, Caps-CAM explicitly employs gradient information that reflects the relevance of each capsule to a class of interest. As the gradient can help highlight the most relevant capsules, each selected capsule activation map is weighted by its corresponding gradient. The final attribution heatmap is then generated as a linear combination of weighted activation maps based on their contribution to the target class. Experiments show that Caps-CAM serves as an explanation method for CapsNets and compare the results of this method with other state-of-the-art explanation techniques previously introduced for CNNs. Empirical comparisons w.r.t. state-of-the-art explanation techniques previously introduced for CNNs, show that Caps-CAM can effectively serve as an explanation method for CapsNets. Experiments on standard and real-application data sets show the effectiveness of the introduced Caps-CAM.
URL: https://openreview.net/forum?id=eNQK9WJkid
---
Title: LibMoE: A Library for Comprehensive Research on Mixture of Experts in Large Language Models
Authors: Nam V. Nguyen, Thong T. Doan, Luong Tran, Van Nguyen, Quang Pham
Abstract: Mixture of experts (MoE) architectures have become a cornerstone for scaling up and are a key component in most large language models such as GPT-OSS, DeepSeek-V3, Llama-4, and Gemini-2.5. However, systematic research on MoE remains severely constrained by the prohibitive computational costs of training and evaluation, restricting large-scale studies accessible to most researchers. We introduce LibMoE, a unified framework for reproducible, efficient, and extensible MoE research that supports both pretraining and sparse-upcycling regimes. Beyond unified implementations, the framework provides transparent analytical tools for probing routing and expert dynamics. Leveraging this foundation, we conduct a comprehensive analysis along three dimensions: (i) routing dynamics, covering expert selection patterns, routing stability and optimality, and how routing entropy reveals task specialization and expert diversity; (ii) the effect of lightweight initialization on load balancing, demonstrating how subtle changes in router initialization shape early expert utilization; and (iii) training regime differences, revealing how sparse upcycling and full pretraining exhibit distinct routing patterns and stability profiles. By lowering the barrier to entry and standardizing evaluation, along with our comprehensive analysis, LibMoE broadens access to MoE research and establishes a reliable benchmark to guide future innovations.
URL: https://openreview.net/forum?id=PB2ju8tq0n
---
Title: MetaTT: A Global Tensor-Train Adapter for Parameter-Efficient Fine-Tuning
Authors: Javier Lopez-Piqueres, Pranav Deshpande, Archan Ray, Mattia Jacopo Villani, Marco Pistoia, Niraj Kumar
Abstract: We present MetaTT, a Tensor Train (TT) adapter framework for fine-tuning of pre-trained transformers. MetaTT enables flexible and parameter-efficient model adaptation by using a single shared TT to factorize transformer sub-modules. This factorization indexes key structural dimensions, including layer and matrix type, and can optionally incorporate heads and tasks. This design allows MetaTT’s parameter count to scale with the sum, rather than the product, of the modes, resulting in a substantially more compact adapter. Our benchmarks compare MetaTT with LoRA along with recent state-of-the-art matrix and tensor decomposition based fine-tuning methods. We observe that when tested on single-task standard language modeling benchmarks, MetaTT achieves competitive parameter efficiency to accuracy tradeoff. We further demonstrate that MetaTT performs competitively when compared to state-of-the-art methods on multi-task learning. Finally, we leverage the TT decomposition to design a rank adaptive optimizer inspired by the DMRG method from many-body physics. Our results demonstrate that integrating this approach with AdamW enhances optimization performance for a specified target rank.
URL: https://openreview.net/forum?id=1HdcPWfA9s
---
Title: Temporal Energy Transformer for Long Range Propagation in Continuous Time Dynamic Graphs
Authors: Parveena Shamim ABDUL SALAM, Abhishek A, Harsh Pandey
Abstract: Representation learning on temporal graphs is crucial for understanding dynamically varying real-world systems such as social media platforms, financial transactions, transportation networks, and communication systems. Existing self-attention based models encounter limitations in capturing long-range dependencies and lack clear theoretical foundations. Energy-based models offer a promising alternative, with a well-established theoretical foundation that avoids reliance on pseudo-losses. However, their application in this domain remains largely unexplored, primarily due to the challenge of designing energy functionals. In this work, we introduce the Temporal Energy Transformer (TET), a novel energy-based architecture that integrates with the Temporal Graph Network (TGN) framework. Our approach centres on a novel energy-based graph propagation module that leverages a specially designed energy functional to capture and preserve spatio-temporal information. This is achieved by modelling the temporal dynamics of irregular data streams with a continuous-time differential equation. Our temporal energy transformer (TET) layer employs a series of temporal energy attention layers and a dense associative memory model or a modern Hopfield network. This design demonstrably minimizes the energy functional that is tailored, enabling efficient retention of historical context while assimilating the incoming data. The efficacy of the model is comprehensively validated across a diverse range of temporal graph datasets, including those with long-range dependencies, demonstrating superior performance in both transductive and inductive scenarios for dynamic link prediction.
URL: https://openreview.net/forum?id=zg3bi0GRJk
---
Title: Advancing Counterfactual Prediction through Nonlinear Quantile Regression
Authors: Shaoan Xie, Biwei Huang, Bin Gu, Tongliang Liu, Peter Spirtes, Kun Zhang
Abstract: The ability to answer counterfactual “what if” questions is essential for understanding and leveraging causal relationships. Traditional counterfactual prediction under Pearl’s framework typically relies on access to, or estimation of, a structural causal model (SCM), which is often unavailable and difficult to identify in practice. While prior work has explored rank- and quantile-preserving approaches as an alternative, existing methods often rely on instrumental variables, high-dimensional density estimation, or optimal transport constructions, limiting their applicability in practice. In this paper, we develop a neural bi-level optimization framework that directly operationalizes the quantile-preservation principle for counterfactual prediction. The proposed bi-level framework learns the latent quantile associated with an observation and predicts counterfactual outcomes by preserving this quantile under intervention. We provide a theoretical analysis of the resulting formulation, establishing uniqueness of the population-level solution under mild assumptions and deriving finite-sample generalization guarantees for the conditional quantile estimator. Empirical evaluations across multiple datasets demonstrate the effectiveness of the proposed approach and support the theoretical results.
URL: https://openreview.net/forum?id=0T1Pd65fyD
---
Title: FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
Authors: Tao Feng, Haozhen Zhang, Zijie Lei, Pengrui Han, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jiaxuan You
Abstract: The rapid advancement of large language models (LLMs) has created a diverse landscape of models, each excelling at different tasks. This diversity drives researchers to employ multiple LLMs in practice, leaving behind valuable multi-LLM log data. This naturally leads to the question of whether such logs can be fully leveraged to fuse LLMs' complementary capabilities. Although prior work has explored various strategies for integrating multiple LLMs, we argue that practical fusion must meet two essential requirements: (1) compatibility with real-world serving scenarios (e.g., local and API-based serving), and (2) flexibility to operate at different stages of the LLM pipeline to meet varied user needs (e.g., fine-tuning and inference stages). To this end, we introduce LLMFusionBench, a large-scale benchmark for LLM fusion that spans 14 tasks across six domains, with responses from 20 open-source LLMs (8B-671B) totaling 103M tokens. Building on LLMFusionBench, we propose FusionFactory, a systematic framework with three elaborated levels: (1) query-level fusion via tailored LLM routers, (2) thought-level fusion leveraging retrieved abstract reasoning templates, and (3) model-level fusion via distillation from top-ranked responses. Experiments show that FusionFactory consistently outperforms the best individual LLM across all 14 benchmarks, with the optimal fusion configuration varying across benchmarks, highlighting the promise of multi-LLM log data as a practical foundation for fusing diverse LLM capabilities.
URL: https://openreview.net/forum?id=N951scS3yE
---
Title: Freeze, Prompt, and Adapt: A Framework for Source-free Unsupervised GNN Prompting
Authors: Peyman Baghershahi, Sourav Medya
Abstract: Prompt tuning has become a key mechanism for adapting pre-trained Graph Neural Networks (GNNs) to new downstream tasks. However, existing approaches are predominantly supervised, relying on labeled data to optimize the prompting parameters and typically fine-tuning a task-specific prediction head—practices that undermine the promise of parameter-efficient adaptation. We propose Unsupervised Graph Prompting Problem (UGPP), a challenging new setting where the pre-trained GNN is kept entirely frozen, labels on the target domain are unavailable, the source data is inaccessible, and the target distribution exhibits covariate shift. To address this, we propose UGPrompt, the first fully unsupervised GNN prompting framework. UGPrompt leverages consistency regularization and pseudo-labeling to train a prompting function, complemented with diversity and domain regularization to mitigate class imbalance and distribution mismatch. Our extensive experiments demonstrate that UGPrompt consistently outperforms state-of-the-art supervised prompting methods with access to labeled data, demonstrating the viability of unsupervised prompting as a practical adaptation paradigm for GNNs.
URL: https://openreview.net/forum?id=9KKgIQwCLO
---
Title: CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Authors: Zhaoyi Joey Hou, Alvin Zhang, Yining Lu, Bhiman Kumar Baghel, Anneliese Brei, Ximing Lu, Meng Jiang, Faeze Brahman, Snigdha Chaturvedi, Haw-Shiuan Chang, Daniel Khashabi, Xiang Lorraine Li
Abstract: Creativity is often seen as a hallmark of human intelligence. While large language models(LLMs) are increasingly perceived as generating creative text, there is still no cross-domain and scalable framework to evaluate their creativity across diverse scenarios.
Existing methods of LLM creativity evaluation either heavily rely on humans, limiting speed and scalability, or are fragmented across different domains and different definitions of creativity. To address this gap, we propose CreativityPrism, an evaluation and analysis framework that consolidates eight tasks from three domains: divergent thinking, creative writing, and logical reasoning, into a taxonomy of creativity that emphasizes three dimensions: quality, novelty, and diversity of LLM generations. The framework is designed to be scalable with reliable automatic evaluation judges that have been validated against human annotations. We evaluate 17 state-of-the-art (SoTA) LLMs on CreativityPrism and find that while frontier-scale LLMs dominate creative writing and logical reasoning tasks by a .10 (or 15%) lead over locally-deployable open models, they offer no significant advantage in divergent thinking, a domain much less explored in existing post-training regimes. Our analysis also shows that high performance in one creative dimension or domain rarely generalizes to others; specifically, novelty metrics often show weak or negative correlations with other metrics. This fragmentation confirms that a cross-domain, multi-dimensional framework like CreativityPrism is essential for any meaningful assessment of LLM creativity.
URL: https://openreview.net/forum?id=3pfsQcEtNC
---
Title: Towards Overcoming Reasoning Shortcuts in Neurosymbolic Learning via Efficient Generative Proxies
Authors: Panagiotis Lymperopoulos, Liping Liu
Abstract: Symbol grounding, the task of linking high-dimensional sensory inputs to symbolic representations in neurosymbolic AI (NeSy), often suffers from reasoning shortcuts, where inputs are mapped to unintended concepts due to limited supervision. Reconstruction-based training can help mitigate these ambiguities, but its effectiveness depends strongly on the quality and capacity of the reconstruction model. In this work, we propose a new grounding framework, Efficient Generative Proxies (EGP), that cleanly integrates reconstruction-based training into a generative modeling perspective. EGP subsumes several existing grounding approaches as special cases. We further argue that the role of reconstruction should be to capture the underlying structure of the data rather than to faithfully reconstruct inputs. Accordingly, we design a reconstruction term that leverages the principle that similar inputs should correspond to similar concept labels, thereby substantially reducing grounding ambiguity. We also develop extensions that incorporate additional inductive biases through this reconstruction term, improving robustness in more complex tasks. We evaluate our approach on tasks susceptible to reasoning shortcuts from the RSbench benchmark, as well as on the multi-concept ObjectMath dataset, integrating EGP into state-of-the-art neurosymbolic learning frameworks. Experimental results demonstrate that EGP significantly improves grounding accuracy and effectively mitigates reasoning shortcuts across diverse settings. The code of EGP is available at https://github.com/tufts-ml/egp-towards-overcoming-reasoning-shortcuts.
URL: https://openreview.net/forum?id=Sl2aC9hiaN
---
Title: FairSAM: Fair Classification on Corrupted Image Data Through Sharpness-Aware Minimization
Authors: Yucong Dai, Jie Ji, Xiaolong Ma, Yongkai Wu
Abstract: Image classification models trained on clean data often degrade sharply when exposed to corrupted test or deployment data, such as images with impulse noise, Gaussian noise, or environmental noise. This degradation reduces overall performance and disproportionately affects demographic subgroups, raising algorithmic bias concerns. Although robust learning algorithms such as Sharpness-Aware Minimization improve overall robustness and generalization, they do not address biased performance degradation across demographic subgroups. Existing fairness-aware machine learning methods reduce performance disparities but struggle to maintain robust and equitable accuracy across demographic subgroups under data corruption. This limitation reveals an inherent tension between robustness and fairness under corrupted data. To address these challenges, we introduce a metric to assess performance degradation across subgroups under data corruption. We propose FairSAM, a framework that integrates Fairness-oriented strategies into SAM to equalize performance across demographic groups under corrupted conditions. Experiments on multiple real-world datasets and prediction tasks show that FairSAM balances robustness and fairness in corrupted image classification. The framework yields a structured solution for fair and robust image classification in the presence of data corruption.
URL: https://openreview.net/forum?id=W2QKvn57yw
---
Title: Bigger Isn’t Always Memorizing: Early Stopping Overparameterized Diffusion Models
Authors: Alessandro Favero, Antonio Sclocchi, Matthieu Wyart
Abstract: Diffusion probabilistic models have become a cornerstone of modern generative AI, yet the mechanisms underlying their generalization remain poorly understood. In fact, if these models were perfectly minimizing their training loss, they would just generate data belonging to their training set, i.e., memorize, as empirically found in the overparameterized regime. We revisit this view by showing that, in highly overparameterized diffusion models, generalization in natural data domains is progressively achieved during training before the onset of memorization. Our results, ranging from image to language diffusion models, systematically support the empirical law that memorization time is proportional to the dataset size, consistent with a kernel-regression bound on the time required to fit the empirical score at low noise. Generalization vs. memorization is then best understood as a competition between time scales. We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules, where generalization corresponds to the hierarchical acquisition of deeper grammar rules as training time grows, and the generalization cost of early stopping can be characterized. We summarize these results in a phase diagram. Overall, our results support that a principled early-stopping criterion – scaling with dataset size – can effectively optimize generalization while avoiding memorization, with direct implications for hyperparameter transfer and privacy-sensitive applications.
URL: https://openreview.net/forum?id=P6r9OxZ1vm
---
Title: On Rate-Optimal Partitioning Classification from Observable and from Privatised Data
Authors: Balázs Csáji, László Györfi, Ambrus Tamás, Harro Walk
Abstract: In this paper we revisit the classical method of partitioning classification and prove novel convergence rates under relaxed conditions, both for observable (non-privatised) and for privatised data. We consider the problem of classification in a $d$ dimensional Euclidean space. Previous results on the partitioning classifier worked with the strong density assumption (SDA), which is restrictive, as we demonstrate through simple examples. Here, we study the problem under much milder assumptions. We presuppose that the distribution of the inputs is a mixture of an absolutely continuous and a discrete distribution, such that the absolutely continuous component is concentrated on a $d_a$ dimensional subspace. In addition to the standard Lipschitz and margin conditions, a novel characteristic of the absolutely continuous component is introduced, by which the convergence rate of the classification error probability is computed, both for the binary and for the multi-class cases. This bound can reach the minimax optimal convergence rate achievable using SDA, but under much milder distributional assumptions. Interestingly, this convergence rate depends only on the intrinsic dimension of the continuous inputs, $d_a$, and not on $d$. Under privacy constraints, the data cannot be directly observed, and the constructed classifiers are functions of the randomised outcome of a suitable local differential privacy mechanism. In this paper we add Laplace distributed noises to the discretisations of all possible locations of the feature vector and to its label. Again, tight upper bounds on the convergence rate of the classification error probability can be derived, without using SDA, such that this rate depends on $2d_a$.
URL: https://openreview.net/forum?id=KYYvIrtgK0
---
Title: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
Authors: Louie Hong Yao, Nicholas Jarvis, Tiffany Zhan, Saptarshi Ghosh, Linfeng Liu, Tianyu Jiang
Abstract: Standard LLM evaluation practices compress diverse abilities into single scores, obscuring their inherently multidimensional nature. We present JE-IRT, a geometric item-response framework that embeds both LLMs and questions in a shared space. For question embeddings, the direction encodes semantics and the norm encodes difficulty, while correctness on each question is determined by the geometric interaction between the model and question embeddings. This geometry replaces a global ranking of LLMs with topical specialization and enables smooth variation across related questions. Building on this framework, our experimental results reveal that out-of-distribution behavior can be explained through directional alignment, and that larger norms consistently indicate harder questions. Moreover, JE-IRT naturally supports generalization: once the space is learned, new LLMs are added by fitting a single embedding. The learned space further reveals an LLM-internal taxonomy that only partially aligns with human-defined subject categories. We also show that simple linear probes of the embedding space recover cross-subject ability directions, such as an arithmetic axis that highlights quantitatively demanding questions in seemingly distant subjects like virology and global facts. JE-IRT thus establishes a unified and interpretable geometric lens that connects LLM abilities with the structure of questions, offering a distinctive perspective on model evaluation and generalization.
URL: https://openreview.net/forum?id=VQe6p9wn5g
---
Title: Neural Logic Networks for Interpretable Classification
Authors: Vincent Perreault, Katsumi Inoue, Richard Labib, Alain Hertz
Abstract: Traditional neural networks have an impressive classification performance, but what they learn cannot be inspected, verified or extracted. Neural Logic Networks on the other hand have an interpretable structure that enables them to learn a logical mechanism relating the inputs and outputs with AND and OR operations. We generalize these networks with NOT operations and biases that take into account unobserved data and develop a rigorous logical and probabilistic modeling in terms of concept combinations to motivate their use. We also propose a novel factorized IF-THEN rule structure for the model as well as a modified learning algorithm. Our method improves the state-of-the-art in Boolean networks discovery and is able to learn relevant, interpretable rules in tabular classification, notably on examples from the medical and industrial fields where interpretability has tangible value.
URL: https://openreview.net/forum?id=CmJXxeimUk
---
Title: Improved denoising diffusion probabilistic models with efficient non-diagonal covariance modeling
Authors: Rui Xia, Ayan Das, Artem Artemev, Andi Zhang, Guillaume Hennequin, Alberto Bernacchia
Abstract: The sampling process of Denoising Diffusion Probabilistic Models (DDPMs) can be accelerated by leveraging second-order information in the form of approximations to the denoising posterior covariance -- allowing samples of acceptable quality to be produced in fewer but larger sampling steps. Previous attempts at using such information have used drastic (e.g. diagonal) simplifications of the covariance. These do not do justice to the peculiar statistical structure of natural images, which exhibit strong non-diagonal correlations between pixels and color channels, and a slow-decaying power-law frequency spectrum. Here, we develop a novel covariance model that captures these features. Our Kronecker-DCT (K-DCT) model uses a Kronecker-factored decomposition of inter-color covariances and spatial covariances modeled in the frequency domain using the Discrete Cosine Transform (DCT). The use of the DCT reduces the computational complexity from quadratic to log-linear, resulting in negligible computational and memory overhead in each denoising step. By learning K-DCT-structured amortizations of the denoising posterior covariance using pre-trained score models on CIFAR-10, Celeb-A, ImageNet and LSUN datasets, we show improved performance compared to previous SOTA denoising samplers, both in terms of FID and likelihoods, especially in the regime of few denoising steps.
URL: https://openreview.net/forum?id=V6FBm4kfML
---
Title: Leveraging Vision-Language Models for Resource Constrained Settings
Authors: Anna Bair, Madan Ravi Ganesh, Devin Willmott, J Zico Kolter
Abstract: Vision-language models (VLMs) such as CLIP have emerged as extremely strong zero-shot and few-shot image classifiers.
However, these models are often too expensive or cumbersome for resource constrained downstream applications.
In this work, we examine how to best leverage the strength of pretrained VLMs: by extracting $\textit{task-specific}$ information in order to obtain a small model that can be deployed in a very specific and low-resource setting. We present the SIDCLIP method, a training pipeline which drastically improves the performance of small, efficient models, such as EfficientNet B0. The pipeline includes three components that are critical to obtaining strong performance: 1) augmenting the classifier with $\textit{synthetic data}$ generated by leveraging CLIP itself; 2) $\textit{initializing}$ the modeling process using a smaller CLIP model pretrained on the target architecture; and 3) incorporating $\textit{knowledge distillation}$ to maximally mimic the performance of the larger model. SIDCLIP improves the performance of an EfficientNet B0 model by an average of $55$ percentage points on 1-shot versions of four datasets and by an average of $29$ points on the 8-shot versions, relative to directly trained networks, additionally approaching CLIP's linear probe performance while using a model with less than $2\%$ of the parameters of CLIP ViT-L/14's image encoder. Our work is intended to be a practical guide for leveraging the power of foundation models in downstream data-scarce and budget constrained settings. Code is released at https://github.com/annaebair/sidclip.
URL: https://openreview.net/forum?id=cYOKSg60jC
---
Title: Plain Transformers Can be Powerful Graph Learners
Authors: Liheng Ma, Soumyasundar Pal, Yingxue Zhang, Philip Torr, Mark Coates
Abstract: Transformers have attained outstanding performance across various modalities, owing to their simple but powerful scaled-dot-product (SDP) attention mechanisms.
Researchers have attempted to migrate Transformers to graph learning, but most advanced Graph Transformers (GTs) have strayed far from plain Transformers, exhibiting major architectural differences either by integrating message-passing or incorporating sophisticated attention mechanisms.
These divergences hinder the easy adoption of training advances for Transformers developed in other domains.
Contrary to previous GTs, this work demonstrates that the plain Transformer architecture can be a powerful graph learner.
To achieve this, we propose to incorporate three simple, minimal, and easy-to-implement modifications to the plain Transformer architecture to construct our Powerful Plain Graph Transformers (PPGT):
(1) simplified $L_2$ attention for measuring the magnitude closeness among tokens; (2) adaptive root-mean-square normalization to preserve token magnitude information; and (3) a simple MLP-based stem for graph positional encoding.
Consistent with its theoretical expressivity, PPGT demonstrates noteworthy realized expressivity on the empirical graph expressivity benchmark, comparing favorably to more complicated alternatives such as subgraph GNNs and higher-order GNNs.
Its empirical performance across various graph datasets also justifies the effectiveness of PPGT.
This finding underscores the versatility of plain Transformer architectures and highlights their strong potential as a unified backbone for multimodal learning across language, vision, and graph domains.
URL: https://openreview.net/forum?id=bEmDvP0fdv
---
Title: Graph Unitary Message Passing
Authors: Haiquan Qiu, Quanming Yao
Abstract: Unitarity is a useful principle for stabilizing deep neural networks, but in graph neural networks (GNNs) instability is induced not only by learnable parameters but also by the graph propagation operator. Motivated by this distinction, we propose Graph Unitary Message Passing (GUMP), a message-passing framework that uses a unitary propagation operator on a transformed graph to avoid graph-induced exponential decay under repeated propagation. GUMP combines (i) a graph transformation that maps an input graph to an Eulerian line-graph construction admitting unitary adjacency matrices, and (ii) a practical unitary projection procedure based on Newton-Schulz iteration. Theoretical analysis clarifies that, under standard analysis assumptions, unitary propagation keeps the graph-propagation term depth-stable, while vanilla normalized propagation exhibits exponential decay in its non-trivial spectral components. Across synthetic long-range tasks, TUDataset benchmarks, and LRGB datasets, GUMP improves over vanilla message passing and achieves competitive or superior performance against strong baselines. Code is available at https://github.com/ucker/gump_code.
URL: https://openreview.net/forum?id=dvNMDkSBIA
---
Title: On Hamming–Lipschitz Type Stability of the Subdominant (Minmax) Ultrametric: Theory and Simple Proofs
Authors: Alokendu Mazumder, Arnab Roy, Punit Rathore
Abstract: The subdominant (minmax) ultrametric is a canonical tree-structured summary of a dissimilarity matrix, arising equivalently as the ultrametric induced by single-linkage clustering. While its classical stability theory is usually formulated in $\ell_\infty$ or Gromov--Hausdorff terms, such bounds are poorly suited to sparse perturbations that alter only a few pairwise distances. We develop an $\ell_0$-type stability theory for this operator. Our analysis shows that sparse edits propagate only through the $\emph{minimum spanning tree}$ (MST): a pairwise ultrametric value can change only if its tree path crosses an edited edge or a cut newly exposed by an edited off-tree edge. This yields a sharp per-edit exposed-cut score and a tree-only global envelope, leading to Hamming--Lipschitz bounds on the number of ultrametric entries that can change. We also prove sharpness results showing that this dependence on tree geometry is unavoidable: under strict cut separation, the tree-edge bound is attained exactly, and for off-tree edits, there are explicit families in which one edited distance changes $\Theta(n^2)$ ultrametric entries. In addition, we prove a conditional near-additivity principle for multiple edits under certified large per-edit changed regions and negligible aggregate overlap. Experiments on deep-embedding graphs show that the resulting structural scores provide useful vulnerability diagnostics for hierarchical representations.
URL: https://openreview.net/forum?id=R4ASOCp3uM
---
Title: Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection
Authors: Yihao Xue, Kristjan Greenewald, Youssef Mroueh, Baharan Mirzasoleiman
Abstract: Large Language Models (LLMs) often hallucinate, limiting their reliability in sensitive applications. In black-box settings, several self-consistency-based techniques have been proposed for hallucination detection. We empirically show that these methods perform nearly as well as a supervised (black-box) oracle, leaving limited room for further gains within this paradigm. To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM. With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods. We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases. It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier. We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights. Extensive experiments on QA-style hallucination detection benchmarks show that this approach maintains high detection performance while significantly reducing computational cost.
URL: https://openreview.net/forum?id=6tlLISSgiu
---
Title: Enhancing Graph Representations with Neighborhood-Contextualized Message-Passing
Authors: Brian Godwin Lim, Galvin Brice Lim, Renzo Roel Tan, Irwin King, Kazushi Ikeda
Abstract: Graph neural networks (GNNs) have become an indispensable tool for analyzing relational data. Classical GNNs are broadly classified into three variants: convolutional, attentional, and message-passing. While the standard message-passing variant is expressive, its typical pair-wise messages only consider the features of the center node and each neighboring node individually. This design fails to incorporate contextual information contained within the broader local neighborhood, potentially hindering its ability to learn meaningful relationships within the entire set of neighboring nodes—a critical limitation for complex domains like financial network anomaly detection and molecular property prediction. To address this, the paper first refines the concept of neighborhood-contextualization within GNNs, leveraging ideas from set-based aggregation methods and a key property of the attentional variant. This then serves as the basis for generalizing the message-passing variant to the proposed neighborhood-contextualized message-passing (NCMP) framework. To demonstrate its utility, a simple, mathematically grounded method to parametrize and operationalize NCMP is presented, leading to the development of the proposed Soft-Isomorphic Neighborhood-Contextualized Graph Convolution Network (SINC-GCN). Across a diverse set of synthetic and benchmark datasets, SINC-GCN strikes a highly favorable balance between expressivity and efficiency. Notably, while more complex models incur significant computational overhead, SINC-GCN delivers substantial performance gains with considerable effect sizes over baseline GNN models while maintaining a highly efficient asymptotic runtime complexity, further underscoring the distinctive utility of neighborhood-contextualization. Overall, by integrating multiset neighborhood context, the proposed NCMP framework serves as a practical and scalable path toward enhancing the graph representational power of classical GNNs.
URL: https://openreview.net/forum?id=6nxdSt5pSn
---
Title: Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning
Authors: Guangming Huang, Yunfei Long, Cunjin Luo
Abstract: Supervised contrastive learning has achieved remarkable success by leveraging label information; however, determining positive samples in multi-label scenarios remains a critical challenge. In multi-label supervised contrastive learning (MSCL), multi-label relations are not yet fully defined, leading to ambiguity in identifying positive samples and formulating contrastive loss functions to construct the representation space. To address these challenges, we: (i) systematically formulate multi-label relations in MSCL, (ii) propose a novel Similarity-Dissimilarity Loss, which dynamically re-weights samples based on similarity and dissimilarity factors, (iii) further provide theoretically grounded proofs for our method through rigorous mathematical analysis that supports the formulation and effectiveness, and (iv) offer a unified form and paradigm for both single-label and multi-label supervised contrastive loss. We conduct experiments on both image and text modalities and further extend the evaluation to the medical domain. The results show that our method consistently outperforms baselines in comprehensive evaluations, demonstrating its effectiveness and robustness.
URL: https://openreview.net/forum?id=W445zcqThv
---
Title: Learning to Understand Videos From Encoded Bytes
Authors: AJ Piergiovanni, Ganesh Satish Mallya, Dahun Kim, Anelia Angelova
Abstract: We present an approach to understand video from encoded bytes, e.g., mp4s. These compressed videos are 99% smaller than the RGB pixel representations which are currently commonly used for video understanding. Encoded videos are able to compress the pixels by taking advantage of the redundant information across the frames using special encoding, such as key frames and motion residuals to handle this. However, standard video understanding models do not take advantage of this significant compression already available for each video, and instead either heavily subsample the frames or only work on short segments of the video. Here, we present an approach to understanding video from encoded bytes directly. We note that simply applying existing models, e.g., Transformers or State-Space models, to video byte sequences does not work, both due to difficulty in handling very long video byte sequences and easy overfitting. To address these challenges, we design a StateSpace model with sequence parallelism to handle very long byte sequences, reaching 15 million tokens in training, and essentially unlimited tokens in inference. We also propose a multilevel SSM activation fusion that reduces sequence length, which we find also benefits video understanding. We evaluate on six video understanding benchmarks including long, high-fps and video + audio understanding tasks and demonstrate competitive performance, illustrating, for the first time, the feasibility of learning from compressed video byte representations.
URL: https://openreview.net/forum?id=LXCs9GFkst
---
Title: GCN-DevLSTM: Path Development for Skeleton-Based Action Recognition
Authors: Lei Jiang, Weixin Yang, Xin Zhang, Hao Ni
Abstract: Skeleton-based action recognition (SAR) in videos is an important but challenging task in computer vision. The recent state-of-the-art (SOTA) models for SAR are primarily based on graph convolutional neural networks (GCNs), which are powerful in extracting the spatial information from skeleton data. However, their ability to capture temporal dynamics remains limited. To address this, we propose the G-Dev layer, which leverages path development—a principled and parsimonious representation for sequential data based on Lie group structures—to enhance temporal modeling. By integrating the G-Dev layer, the proposed DevLSTM module summarizes local temporal dynamics, reducing the time dimension while retaining high-frequency information. It can be conveniently applied to any temporal graph data, complementing existing advanced GCN-based models. Our empirical studies on the NTU-60, NTU-120 and Chalearn2013 datasets demonstrate that our proposed GCN-DevLSTM network consistently improves the strong GCN baseline models and achieves competitive performance. The code repository is publicly available at https://github.com/DeepIntoStreams/GCN-DevLSTM.
URL: https://openreview.net/forum?id=3o5seglmgn
---
Title: Overcoming label shift in target-aware federated learning
Authors: Edvin Listo Zec, Adam Breitholtz, Fredrik D. Johansson
Abstract: Federated learning enables multiple actors to collaboratively train models without sharing private data. Existing algorithms are successful and well-justified in this task when the intended target domain, where the trained model will be used, shares data distribution with the aggregate of clients, but this is often violated in practice. A common reason is label shift—that the label distributions differ between clients and the target domain. We demonstrate empirically that this can significantly degrade performance. To address this problem, we propose FedPALS, a principled and practical model aggregation scheme that adapts to label shifts to improve performance in the target domain by leveraging knowledge of label distributions at the central server. Our approach ensures unbiased updates under federated stochastic gradient descent, which yields robust generalization across clients with diverse, label-shifted data. Extensive experiments on image classification tasks demonstrate that FedPALS consistently outperforms baselines by aligning model aggregation with the target domain. Our findings reveal that conventional federated learning methods suffer severely in cases of extreme label sparsity on clients, highlighting the critical need for target-aware aggregation as offered by FedPALS.
URL: https://openreview.net/forum?id=dQAsAmb1Xb
---
Title: Can VLMs Reason Robustly? A Neuro-Symbolic Investigation
Authors: Weixin Chen, Antonio Vergari, Han Zhao
Abstract: Vision-Language Models (VLMs) have been applied to a wide range of reasoning tasks, yet it remains unclear whether they can reason robustly under distribution shifts. In this paper, we study covariate shifts in which the perceptual input distribution changes while the underlying prediction rules do not. To investigate this question, we consider visual deductive reasoning tasks, where a model is required to answer a query given an image and logical rules defined over the object concepts in the image. Empirically, we find that VLMs fine-tuned through gradient-based end-to-end training can achieve high in-distribution accuracy but fail to generalize under such shifts, suggesting that fine-tuning does not reliably induce the underlying reasoning function. This motivates a neuro-symbolic perspective that decouples perception from reasoning. However, we further observe that recent neuro-symbolic approaches that rely on black-box components for reasoning can still exhibit inconsistent robustness across tasks. To address this issue, we propose VLC, a neuro-symbolic method that combines VLM-based concept recognition with circuit-based symbolic reasoning. In particular, task rules are compiled into a symbolic program, specifically a circuit, which executes the rules exactly over the object concepts recognized by the VLM. Experiments on three simple visual deductive reasoning tasks with distinct rule sets show that VLC consistently achieves higher task accuracy on out-of-distribution data than other reasoning paradigms. Code is available at https://github.com/uiuctml/VLC.
URL: https://openreview.net/forum?id=4y6jiE6Q60
---
Title: Towards Principled Task Grouping for Multi-Task Learning
Authors: Chenguang Wang, Xuanhao Pan, Tianshu Yu
Abstract: Multi-task learning (MTL) aims to leverage shared information among tasks to improve learning efficiency and accuracy. However, MTL often struggles to effectively manage positive and negative transfer between tasks, which can hinder performance improvements. Task grouping addresses this challenge by organizing tasks into meaningful clusters, maximizing beneficial transfer while minimizing detrimental interactions.
This paper introduces a principled approach to task grouping in MTL, advancing beyond existing methods by addressing key theoretical and practical limitations. Unlike prior studies, our method offers a theoretically grounded approach that does not depend on restrictive assumptions for constructing transfer gains. We also present a flexible mathematical programming formulation that accommodates a wide range of resource constraints, thereby enhancing its versatility.
Experimental results across diverse domains, including computer vision datasets, combinatorial optimization benchmarks, and time series tasks, demonstrate the superiority of our method over extensive baselines, thereby validating its effectiveness and general applicability in MTL without sacrificing efficiency.
Code is available at \url{https://github.com/LOGO-CUHKSZ/Principled-Task-Grouping}.
URL: https://openreview.net/forum?id=3DeSIpzuro
---
Title: Domain Generalizable Adaptation of 3D Vision-Language Models via Regularized Fine-Tuning
Authors: Sneha Paul, Zachary Patterson, Nizar Bouguila
Abstract: Domain adaptation remains a central challenge in 3D vision, especially for multimodal foundation models that align 3D point clouds with visual and textual data. While these models demonstrate strong general capabilities, adapting them to downstream domains with limited data often leads to overfitting and catastrophic forgetting. To address this, we introduce ReFine3D, a regularized fine-tuning framework designed for domain-generalizable tuning of 3D large multimodal models (LMMs). ReFine3D combines selective layer tuning with two targeted regularization strategies: multi-view consistency across augmented point clouds and text diversity through synonym-based prompts generated by large language models. Additionally, we incorporate point-rendered vision supervision and a test-time scaling strategy to further enhance robustness. Extensive experiments across different 3D domain generalization benchmarks show that ReFine3D improves base-to-novel class generalization by 1.36%, cross-dataset transfer by 2.43%, robustness to corruption by 1.80%, and few-shot accuracy by up to 3.11%-outperforming prior state-of-the-art methods with minimal added computational overhead.
URL: https://openreview.net/forum?id=453uT7O7wc
---
Title: Confidence-Aware Explanations for 3D Molecular Graphs via Energy-Based Masking
Authors: Xufeng Liu, Wenhan Gao, Yi Liu
Abstract: Graph Neural Networks (GNNs) have become a powerful tool for modeling molecular data. To improve their reliability and interpretability, various graph explanation methods are proposed to identify key molecular substructures that drive model predictions. Many graph explainers introduce soft masks to enable gradient-based optimization, and then discretize the optimized masks to obtain explanatory subgraphs. While these methods perform well for 2D GNNs, there is a growing demand for 3D explanation techniques suited to 3D GNNs, which often surpass 2D GNNs in performance. However, existing explainers struggle with 3D GNNs because cutoff-based 3D graph construction yields denser graphs, with the number of edges growing quadratically with the number of atoms. Motivated by this, we identify key sources of explanation errors and derive an upper bound that decomposes the explanation error into two components: (i) the optimized soft-mask loss and (ii) the discrepancy introduced when discretizing the soft mask to form the explanatory subgraph. Our theoretical analysis shows that the second component is closely related to the soft-to-discrete mask gap and is amplified by graph density, making it particularly challenging for dense 3D graphs. To bridge this gap, we use an energy-based formulation and our method assigns two energy values to each atom, corresponding to importance and non-importance. The explanation model becomes more confident when the distinction between two states is clearer. By optimizing these energy values to distinguish the two cases, we minimize both components of the bound and identify a stable subgraph with high explanation fidelity. Experiments with various 3D backbone models on widely used datasets validate our method's effectiveness in providing accurate and reliable explanations for 3D molecular graphs. The code is publicly available at https://github.com/xufliu/EDMA.
URL: https://openreview.net/forum?id=V4oLqP0vJf
---
Title: What do near-optimal learning rate schedules look like?
Authors: Hiroki Naganuma, Atish Agarwala, Priya Kasimbeg, George E. Dahl
Abstract: A basic unanswered question in neural network training is: what is the best learning rate schedule shape for a given workload? The choice of learning rate schedule is a key factor in the success or failure of the training process, but beyond having some kind of warmup and decay, there is no consensus on what makes a good schedule shape. To answer this question, we designed a search procedure to find the best shapes within a parameterized schedule family. Our approach factors out the schedule shape from the base learning rate, which otherwise would dominate cross-schedule comparisons. We applied our search procedure to a variety of schedule families on three workloads: linear regression, image classification on CIFAR-10, and small-scale language modeling on Wikitext103. We showed that our search procedure indeed generally found near-optimal schedules. We found that warmup and decay are robust features of good schedules, and that commonly used schedule families are not optimal on these workloads. Finally, we explored how the outputs of our shape search depend on other optimization hyperparameters, and found that weight decay can have a strong effect on the optimal schedule shape. To the best of our knowledge, our results represent the most comprehensive results on near-optimal schedule shapes for deep neural network training, to date.
URL: https://openreview.net/forum?id=pEsAMnmq0L
---
Title: Domain Adaptation for Cold-Start Users in Sequential Recommendation
Authors: Lu Wang, Wenyu Zhang, Wang Chengke, Guimei Liu, Ye Luo
Abstract: Sequential recommendation tracks users' preferences over time based on users' historical activities and makes prediction on their next most probable action. However, this approach faces limitations when dealing with cold-start users who possess minimal interaction data, leading to difficulty in learning their preferences. To address this challenge, by taking regular users with longer interaction histories and cold-start users as two domains, this paper introduces domain adaptation techniques to narrow the performance gap caused by knowledge shifts in domains. We propose a dual-transformer framework with separate models for long (source) and short (target) sequences, collaboratively trained with shared item embeddings. To enable effective knowledge transfer, we introduce an emulated target domain by sampling short sequences from the source, and apply contrastive learning to align their contextual representations. To further improve adaptation under complex knowledge shifts, we reduce item popularity bias and incorporate user similarity into the contrastive loss. Experiments on five public datasets show consistent improvements over strong baselines, demonstrating the robustness of our approach under both length shifts and compounded shifts involving item distribution changes.
URL: https://openreview.net/forum?id=jrVmW6LfMa
---
Title: Modelling Complex Tabular Datasets with a Mixture of Diverse Generative Models
Authors: Faul Antoine, Xiao Zhou, Ossi Räisä, Mihaela van der Schaar, Cem Tekin
Abstract: Generative models are widely used, yet they often struggle to capture the multi-modal structure of complex tabular datasets. We address this challenge by introducing a novel framework that employs mixtures of diverse generators, each specialized to different regions of the data space. Our method proceeds in two stages: first, generators are assigned to data clusters via a compute-efficient bandit-based allocation strategy; second, cluster assignments are refined through an iterative procedure inspired by the Expectation–Maximization (EM)
framework. Crucially, our approach is designed for settings where the generators’ likelihoods are intractable and only generated data samples are accessible. In a simpler setting where clusters can be approximately identified, we derive theoretical results based on the robustness of the Maximum Mean Discrepancy (MMD). Empirical evaluations on both synthetic and real-world tabular datasets demonstrate that our approach produces high-quality synthetic data, validating its effectiveness in challenging generative modeling tasks.
URL: https://openreview.net/forum?id=3y3mHAldp7
---
Title: Extrapolating from Regularised Solutions for Solving Ill-Conditioned Linear Systems in Machine Learning
Authors: Disha Hegde, Jon Cockayne, Chris J. Oates
Abstract: Rapid prototyping of algorithms is a critical step in modern machine learning. Most algorithms exploit linear algebra, creating a need for lightweight numerical routines which -- while potentially sub-optimal for the task at hand -- can be rapidly implemented. For the numerical solution of ill-conditioned linear systems of equations, the standard solution for prototyping is Tikhonov-regularised inversion using a nugget. However, selection of the size of nugget is often difficult, and the use of data-adaptive procedures precludes automatic differentiation, introducing instabilities into end-to-end training. Further, while data-adaptive procedures perform multiple linear solves to select the size of nugget, only the result of one such solve is returned, which we argue is wasteful. This paper aims to circumvent the above difficulties, presenting `autonugget`; a `Python` package for automatic and stable numerical solution of linear systems suitable for rapid prototyping, and fully compatible with automatic differentiation using `JAX`. `autonugget` combines multiple linear solves using Richardson extrapolation to determine the solution of the ill-conditioned system, improving in accuracy over approximations based on a single nugget.
URL: https://openreview.net/forum?id=fqbkenUpRa
---
New submissions
===============
Title: The Scissors Effect: When Resize-Based Input Diversity Helps or Hurts Transfer Attacks
Abstract: Input Diversity (DI), which applies random resizing and padding at each attack iteration, is a near-default ingredient of transfer-based adversarial attacks because it is widely assumed to improve transferability. We show this assumption is regime-dependent and, for robustly trained surrogates, often reversed. Holding the attack fixed and varying only the surrogate, increasing the DI probability raises transfer success for standard surrogates but lowers it for robust ones: the two response curves separate like a pair of scissors, a pattern we call the Scissors Effect. The effect is strong and consistent on ImageNet: blind DI costs the robust source 10.3% attack success on average across CNN, ViT, Swin, and ConvNeXt targets, and the harm holds across ten attacks spanning 2018–2024; it is smaller on CIFAR-10 unless DI is made aggressive. A controlled robustness-strength sweep that fixes the ResNet-50 architecture and PGD-AT recipe and varies only the training $\epsilon$ shows the harm is graded rather than binary, crossing from beneficial to harmful already in the "little-robustness" regime. We trace it to gradient geometry: a resize/translation decomposition attributes roughly 67% of the harm to resize, and a direct source–target gradient-alignment measurement confirms the same resize operation improves alignment for standard surrogates but degrades it for robust ones. We summarize the regime with Local Gradient Consistency (LGC), a single input-space probe that cleanly separates the two surrogate types and tracks DI sensitivity at the regime level. A bias–variance theorem formalizes the mechanism qualitatively: a single crossover separates where DI helps from where its resize bias dominates. A training-free rule (CG-DI) that disables diversity when LGC is high then avoids the loss on robust surrogates while keeping DI's benefit on standard ones. We position the Scissors Effect as a DI-specific manifestation of the broader robustness–transferability trade-off, isolating which component of input diversity is responsible and when it should be disabled.
URL: https://openreview.net/forum?id=b4pCcgJM0M
---
Title: Sampling Triangulations and Calabi-Yau Threefolds with Autoregressive GNNs
Abstract: We introduce `dualGNN', an autoregressive message-passing GNN for sampling fine, regular triangulations of lattice polytopes. dualGNN operates on a generalization of the dual graph of a triangulation, with edges labeled by `signed circuits' --- combinatorial invariants from the theory of oriented matroids. We show that these circuits are necessary and sufficient to determine a triangulation's regularity from the graph, provided certain magnitude information is retained. The model is independent of the polytope's point count and invariant under its orientation-preserving symmetries ($\mathrm{SL}(d,\mathbb{Z}) \ltimes \mathbb{Z}^d$), and our masking procedure further guarantees that every rollout produces a fine triangulation (in 2D). On unseen polygons with $N_\mathrm{pts} \leq 40$, dualGNN is the only sampler we tested that is consistent with uniform sampling across all our diagnostics (KL divergence from uniformity, collision counts, and sample autocorrelation). The model is small ($\sim92$k parameters) and trains in $\sim7.5$ hours on a single consumer GPU. We apply dualGNN to string theory, sampling Calabi-Yau threefolds uniformly at $h^{1,1}=86$; we also sample CYs at $h^{1,1}=128$, observing no deviations from uniformity, but our diagnostics are weaker here. Code, training scripts, and pretrained models are available at \url{https://github.com/natemacfadden/dualGNN} (\code{pip install dualgnn}), and dualGNN is integrated into CYTools.
URL: https://openreview.net/forum?id=tlscqOYjmZ
---
Title: Rethinking Post-Hoc Calibration in Semantic Segmentation
Abstract: Reliable confidence estimates are essential in semantic segmentation, especially in safety-critical settings where overconfident errors can mislead downstream decisions. Yet modern segmentation models often remain miscalibrated. Post-hoc calibration offers a practical way to correct confidence estimates without retraining the segmentation model, but its use in dense prediction raises structural issues that are often overlooked. We study two such issues. First, adding a constant to all logits leaves the softmax probabilities unchanged, but several standard calibrators can still depend on this arbitrary offset. As a result, two logit representations encoding the same predictive distribution may yield different calibrated probabilities. We define translation-invariant (TI) calibrators as those whose outputs are unchanged under such shifts, characterize which common calibrators satisfy this property, and construct TI counterparts of shift-sensitive calibrators to isolate the effect of removing representation dependence. Second, post-hoc calibration is typically fitted by minimizing a likelihood-based objective, whereas segmentation models are trained with task-specific metrics such as Dice. This mismatch can cause calibration to alter class orderings and degrade the deployed segmentation map. We study decision-preserving calibration under argmax- and order-preservation constraints. Since enforcing these constraints collapses affine softmax calibrators to temperature scaling, we introduce class-conditional affine calibrators that can be made argmax- or order-preserving while retaining greater expressivity, allowing us to quantify the calibration--segmentation trade-off induced by decision preservation. Across natural-image and medical segmentation benchmarks, and under corruption-based covariate shift, matched comparisons show that TI variants generally improve calibration metrics, while decision-preserving variants prevent segmentation degradation and retain strong calibration performance. These results provide practical design principles for well-defined post-hoc calibration pipelines in semantic segmentation.
URL: https://openreview.net/forum?id=xwNoSNxgxV
---
Title: The Zero-Shot Illusion in the Wild: Diagnosing Boundary Failures in AIGC Detection
Abstract: Detecting AI-generated images remains a significant challenge due to the rapid proliferation of diverse generative models and workflows. While many detectors claim strong zero-shot generalization in curated benchmarks, they struggle to generalize to the more complex distribution of real-world AI images, where outputs are from workflows combining multiple models, LoRAs, adapters, post-processing, etc. We expose this \emph{zero-shot illusion} with \textbf{WildGen}, an in-the-wild benchmark built from 342K community-created AI images spanning 342 sources, together with 347K real images from 8 authentic datasets. On WildGen, detectors that perform well on existing benchmarks exhibit source-dependent failures: 11 of 12 training-based methods remain below 58\% balanced accuracy, and the best training-free method reaches only 75.21\% AUC. The generalization gap arises primarily from decision boundary misalignment rather than a fundamental deficiency in feature representation. We demonstrate that adapting off-the-shelf detectors to unseen models yields 97.22\% balanced accuracy (95.77\% if only 18 examples per generator are used) for AI-vs-real classification, and real-source identity is predictable at 84.33\% balanced accuracy. Detectors trained on WildGen can also be adapted to existing benchmarks such as GenImage and ForenSynths using few-shot target data. These results suggest that, rather than seeking a static universal AIGC detector, a more effective and realistic path is continuous few-shot adaptation that maintains detector boundaries as generators and content-creation workflows evolve.
URL: https://openreview.net/forum?id=9GU6P8xEnN
---
Title: SUPN: Shallow Universal Polynomial Networks
Abstract: Deep neural networks (DNNs) and Kolmogorov-Arnold networks (KANs) are popular methods for function approximation due to their flexibility and expressivity. However, they typically require a large number of trainable parameters to produce a suitable approximation. Beyond making the resulting network less transparent, overparameterization creates a large optimization space, likely producing local minima in training that have quite different generalization errors. In this case, network initialization can have an outsize impact on the model's out-of-sample accuracy. For these reasons, we propose shallow universal polynomial networks (SUPNs). These networks replace all but the last hidden layer with a single layer of polynomials with learnable coefficients, leveraging the strengths of DNNs and polynomials to achieve sufficient expressivity with far fewer parameters. We prove that SUPNs converge at the same rate as the best polynomial approximation of the same degree, and we derive explicit formulas for quasi-optimal SUPN parameters. We complement theory with an extensive suite of numerical experiments involving SUPNs, DNNs, KANs, and polynomial projection in one, two, and ten dimensions, consisting of over 13,000 trained models. On the target functions we numerically studied, for a given number of trainable parameters, the approximation error and variability are often lower for SUPNs than for DNNs and KANs by an order of magnitude. In our examples, SUPNs even outperform polynomial projection on non-smooth functions.
URL: https://openreview.net/forum?id=srHGkCrSTM
---
Title: EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models
Abstract: Large language models and vision transformers have shown impressive zero-shot capabilities, enabling significant transferability in downstream tasks. The fusion of these models has resulted in multi-modal architectures with enhanced instructional capabilities. Despite incorporating vast image and language pre-training, these multi-modal architectures often generate responses that deviate from the ground truth in the image data. These failure cases (false positives) are known as hallucinations. Current methods for mitigating hallucinations generally focus on regularizing the language component, improving the fusion module, or ensembling multiple visual encoders to improve visual representation. In this paper, we address the hallucination issue by directly enhancing the capabilities of the visual component. Our approach, named EAGLE, is fully agnostic to the LLM or fusion module and works as a post-pretraining stage that improves the grounding and language alignment of the visual encoder. We show that a straightforward reformulation of the original contrastive pre-training task results in an improved visual encoder that can be incorporated into the instructional multi-modal architecture without any additional instructional training. Extensive empirical validation shows that EAGLE significantly reduces hallucinations across six different instructional multi-modal models and four challenging benchmarks.
URL: https://openreview.net/forum?id=KlJiaLD6h8
---
Title: Practical and Private Hybrid ML Inference with Fully Homomorphic Encryption
Abstract: In contemporary cloud-based services, protecting users’ sensitive data and ensuring the confidentiality of the server’s model are critical. Fully homomorphic encryption (FHE) enables inference directly on encrypted inputs, but its practicality is hindered by expensive bootstrapping and inefficient approximations of non-linear activations. We introduce Safhire, a hybrid inference framework that executes linear layers under encryption on the server while offloading non-linearities to the client in plaintext. This design eliminates bootstrapping, supports exact activations, and significantly reduces computation cost. To safeguard model confidentiality despite client access to intermediate outputs, Safhire applies randomized shuffling, which obfuscates intermediate values and makes it practically impossible for a client to reconstruct the model. To further reduce inference latency, Safhire incorporates advanced optimizations such as fast ciphertext packing and partial extraction. Evaluations on multiple standard models and datasets show that Safhire achieves up to 10.7× and 7.4× lower inference latency than Orion and Concrete-ML, respectively, which are two state-of-the-art FHE baselines, with manageable communication overhead and comparable accuracy. These results underline the practicality of our hybrid FHE inference scheme.
URL: https://openreview.net/forum?id=AAneiBMYgS
---
Title: ODC: Orthogonal Drift Correction for Improved Text-to-Image Semantic Alignment at Inference
Abstract: Text-to-image models have achieved remarkable success in generating high-quality images from textual descriptions. However, they often struggle with ``semantic drift,'' where the generated output fails to precisely align with complex or nuanced text prompts. While recent approaches have attempted to address semantic errors regarding attribute binding or object presence, there remains a gap for a more holistic method that addresses these issues by directly refining the text embeddings of the initial user prompt. In this work, we introduce Orthogonal Drift Correction (ODC), an inference-time guidance method designed to mitigate semantic drift without requiring model retraining or additional user inputs. ODC guides image generation through a two-stage process. In the first stage, it identifies the semantic drift by evaluating the initially generated image against the user prompt in a shared vision-language embedding space. It then isolates the component of this drift vector that is orthogonal to the prompt's direction and translates it back into text via a vocabulary-based surrogate mechanism. In the second stage, it produces refined text conditioning for a second generation pass by feeding both the initial text embedding and the re-embedded drift representation into an adaptive rank-reduced concept removal module. Our experiments demonstrate the effectiveness of ODC in enhancing prompt-image alignment, yielding images that more accurately reflect detailed compositional instructions. As a plug-and-play module, ODC offers a practical and efficient method for improving the reliability of state-of-the-art text-to-image models.
URL: https://openreview.net/forum?id=BOdxfPdtjE
---
Title: Pessimistic Reward Modeling in RLHF for Robustness against Reward Hacking
Abstract: This work proposes `PET', a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard text generation datasets. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. The learned policy has a high KL divergence from the dataset distribution while having high performance in practice. We also observe that the length bias phenomenon in reward modeling is significantly mitigated by PET. While the proxy reward trained in traditional approaches shows bias to long responses, the pessimistic reward model finetuned by PET shows little bias. In summary, our work shows the feasibility of learning a pessimistic reward model through PET against reward hacking. The agent can greedily optimize a policy on the pessimistic reward without suffering from reward hacking. PET can also solve the length bias problem in reward modeling.
URL: https://openreview.net/forum?id=ep6ZnD1Zw5
---
Title: To Retain or to Adapt? Generalizing Continual Learning
Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on a pervasive, often unstated assumption: that a lifelong learner should approximate the Joint-Task Learning (JTL) solution and retain all previously acquired knowledge. We challenge this retention-centered premise, arguing that in non-stationary environments prioritizing retention can impede real-time adaptation. Shifting the focus to the Average Lifelong Error (ALE), we formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics. We introduce Transfer Efficiency as a quantitative measure of the tension between Instability, the bias inherited from conflicting past experience, and Transient Error, the optimization cost of learning new tasks from scratch. Under mild convergence conditions, holding across linear and neural network models, this decomposition yields a Critical Task Duration: a closed-form threshold beyond which historical knowledge transitions from a warm-start advantage to an optimization liability whenever retention induces a positive stationary bias. We validate these theoretical predictions on continual image classification and reinforcement learning benchmarks. Finally, by connecting continual learning to the online learning framework of predictable sequences, we show that JTL is only one instance of a broader family of objectives, and we propose a new general class of continual learning algorithms, which we call Predictive Continual Learning. Predictive CL algorithms optimize expected future performance under an explicit, dynamically updated model of future tasks. As a proof of concept, we analyze a Window algorithm that interpolates between JTL and Independent-Task Learning (ITL), outperforming both under controlled distributional drift.
URL: https://openreview.net/forum?id=C4zRmT9jYO
---
Title: Transformers with Selective Access to Early Representations
Abstract: Several recent Transformer architectures expose later layers to representations computed in the earliest layers, motivated by the observation that low-level features can become harder to recover as the residual stream is repeatedly transformed through depth. The cheapest among these methods add static value residuals: learned mixing coefficients that expose the first-layer value projection $V_1$ uniformly across tokens and heads. More expressive dense or dynamic alternatives recover finer-grained access, but at higher memory cost and lower throughput. The usefulness of V_1 is unlikely to be constant across tokens, heads, and contexts; different positions plausibly require different amounts of access to early lexical or semantic information. We therefore treat early-representation reuse as a retrieval problem rather than a connectivity problem, and introduce Selective Access Transformer (SATFormer), which preserves the first-layer value pathway while controlling access with a context-dependent gate. Across models from 130M to 1.3B parameters, SATFormer consistently improves validation loss and zero-shot accuracy over the static value-residual and Transformer baselines. Its strongest gains appear on retrieval-intensive benchmarks, where it improves over static value residuals by approximately 1.5 average points, while maintaining throughput and memory usage close to the baseline Transformer. Gate analyses suggest sparse, depth-dependent, head-specific, and category-sensitive access patterns, supporting the interpretation that SATFormer learns selective reuse of early representations rather than uniform residual copying. Our code is available at https://anonymous.4open.science/r/SATFormer-B09C/README.md
URL: https://openreview.net/forum?id=flq3oO3cNr
---
Title: Relational and Sequential Conformal Inference for Energy Time Series over Graphs via Foundation Models
Abstract: Accurate energy demand forecasting is essential for the reliable operation and planning of modern sustainable energy systems. Spatial-temporal graph neural networks (STGNNs) have recently achieved strong performance in point forecasting by jointly modeling temporal dynamics and relational dependencies across interconnected energy nodes. However, in real-world energy systems, accurate point forecasts alone are insufficient, as operators also require reliable uncertainty estimates to support risk-aware decision-making, grid stability, and operational planning under uncertainty. Conformal prediction provides a principled and model-agnostic framework for uncertainty quantification with statistical coverage guarantees, making it particularly attractive for safety-critical energy applications. However, existing conformal prediction approaches often fail to fully capture the complex spatial-temporal structure of energy systems. To address these limitations, we propose STOIC (Spatial-Temporal Graph Conformal Prediction with In-Context Learning), a novel framework that integrates graph-based forecasting with the zero-shot calibration capabilities of tabular foundation models. STOIC first generates point forecasts using an STGNN and subsequently reformulates spatial-temporal residuals into a tabular representation suitable for in-context learning. Leveraging a tabular foundation model, STOIC calibrates prediction intervals without task-specific retraining, effectively capturing both sequential and relational dependencies. We evaluate STOIC on five diverse benchmarks, including synthetic simulations as well as real-world electricity and district heating networks. Across all datasets, STOIC consistently outperforms existing conformal prediction baselines, delivering more reliable and robust uncertainty estimates for complex graph-structured energy time series.
URL: https://openreview.net/forum?id=pur0JbTRmv
---
Title: Reflection as a Criterion for Active Learning
Abstract: Active learning algorithms are commonly evaluated by classification error, but directly evaluating data acquisitions by their effect on expected classification error is generally difficult. Many principled acquisition criteria are therefore derived for cross-entropy, information gain, or model uncertainty.
We propose reflection as an intermediate criterion for this modeling problem: it scores a candidate batch by holding the previously acquired and candidate instances fixed and assessing their usefulness under alternative plausible label realizations. Given the current posterior over model parameters, reflection samples plausible models, resamples labels for both previously selected and candidate instances, retrains on the relabeled set, and evaluates the resulting classifier under the sampled model.
This formulation is closely aligned with the downstream objective, remains meaningful under point-estimate posteriors and is amenable to asymptotic analysis.
Under cross-entropy, reflection provides a derivation of the Fisher-information-ratio objective as used by BAIT, explaining why BAIT evaluates this ratio on the entire selected set comprising both previous and candidate instances.
To analyze reflection under classification error, we derive the asymptotic expected zero-one loss for histogram classifiers. The formula yields a budget-dependent shift from coverage-oriented to difficulty-driven acquisition. A kernelized relaxation turns this structure into a practical strategy that is competitive with strong baselines, including Margin, BADGE and TypiClust and robust to large batches, label noise, and duplicates.
URL: https://openreview.net/forum?id=TO5Hv4YiH9
---
Title: DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models
Abstract: Videos are inherently temporal sequences by their very nature. In this work, we explore the potential of modeling videos in a chronological and scalable manner with autoregressive (AR) language models, inspired by their success in natural language processing. We introduce \textbf{DiCoDe}, a novel approach that leverages \textbf{Di}ffusion-\textbf{Co}mpressed \textbf{De}ep Tokens to generate videos with a language model in an autoregressive manner. Unlike existing methods that employ low-level representations with limited compression rates, DiCoDe utilizes deep tokens with a considerable compression rate, approaching a 1000× reduction in token count when amortized over long chronological videos. This significant compression is made possible by a tokenizer trained through leveraging the prior knowledge of video diffusion models. Deep tokens enable DiCoDe to employ vanilla AR language models for video generation, akin to translating one visual ``language'' into another. By treating videos as temporal sequences, DiCoDe fully harnesses the capabilities of language models for autoregressive generation. DiCoDe is scalable using readily available AR architectures, and is capable of generating videos ranging from a few seconds to one minute using only 4 A100 GPUs for training. We evaluate DiCoDe both quantitatively and qualitatively, using generation quality mainly as evidence that such an aggressive amortized token compression remains usable for video synthesis under limited training resources. To showcase its scalability, we release a series of DiCoDe configurations with varying parameter sizes and observe a consistent improvement in performance as the model size increases from 100M to 3B. We believe that DiCoDe's exploration in academia represents a promising initial step toward scalable video modeling with AR language models, paving the way for the development of larger and more powerful video generation models.
URL: https://openreview.net/forum?id=B7a0IAIEXZ
---
Title: Aligning Latent Spaces with Flow Priors
Abstract: This paper presents a novel framework for aligning learnable latent spaces to arbitrary sample-defined prior distributions by leveraging flow-matching models as expressive priors. Our method first pretrains a flow-matching model on prior features to capture their conditional interpolation dynamics. The fixed flow model subsequently regularizes the latent space via an alignment loss, which reformulates the flow-matching objective by treating the learnable latents as candidate endpoints. We establish a score-space foundation for this objective: under the population-optimal conditional flow-matching field, the pointwise alignment loss is exactly a multiscale relative Fisher energy between candidate-induced Gaussian probes and Gaussian-smoothed prior marginals. For isotropic Gaussian priors, this energy is an affine transformation of the exact negative log-likelihood; at the aggregate level, its expectation decomposes into a proper multiscale Fisher discrepancy and an endpoint-recoverability term. These results provide a principled basis for using the alignment loss as a computationally tractable prior-likelihood surrogate. Notably, the proposed method eliminates expensive likelihood evaluations and avoids ODE solving during downstream optimization. In controlled experiments, the alignment-loss landscape closely approximates the negative-log-likelihood landscape of diverse two-dimensional priors. We further validate the effectiveness of our approach by regularizing autoencoder latent spaces with diverse visual and textual priors in large-scale ImageNet image generation, accompanied by detailed discussions and ablation studies. With both theoretical and empirical validation, our framework paves a new way for flexible latent-space alignment.
URL: https://openreview.net/forum?id=dtEVxNdKt4
---
Title: When Does Overlap Help? OSU-Mem and a Cell-Conditional Analysis of Trajectory Memory for LLM Agents
Abstract: Long-horizon large language model (LLM) agents accumulate interaction trajectories that quickly exceed any practical prompt budget, and existing memory methods either truncate aggressively and lose non-local evidence or retain boilerplate that degrades decision quality. We ask a mechanism question rather than claiming a better general-purpose memory system: when does organizing trajectory memory into overlapping semantic units (OSUs)—groups of related steps in which one step may belong to several units—help retrieval over flat or disjoint alternatives? We instantiate this in OSU-Mem, which retrieves from an overlapping OSU pool via budgeted coarse-to-fine expansion, and show its benefit is conditional: overlapping memory helps when the evidence steps a query needs share tool calls or entities, but hurts when those steps are fully heterogeneous and share neither. On a synthetic benchmark where evidence carries such shared structure by construction, OSU-Mem improves over the strongest baseline as the theory predicts; yet on a concatenated, constructed unaugmented $\tau$-bench setting its aggregate advantage over flat retrieval vanishes. Splitting queries by whether their evidence shares tools and entities shows this near-tie to be an artifact of mixing query types rather than a property of either method, and ToolBench, a controlled probe built to carry shared structure by design, corroborates the same mechanism via an overlap-vs.-disjoint construction contrast (under a coverage-guided variant), isolating the construction principle rather than validating the full default system. Because the relevant sharing is cheaply estimable from metadata, the analysis yields a metadata-based heuristic for predicting when overlap is likely to improve retrieval. We deliberately isolate the retrieval layer, assessed by retrieval quality and an LLM-mediated evidence-selection stage.
URL: https://openreview.net/forum?id=bfD0aYVGQl
---
Title: Understanding the Physics of Key-Value Cache Compression for LLMs through Attention Dynamics
Abstract: As context windows in large language models (LLMs) grow to hundreds of thousands of tokens, the key--value (KV) cache becomes the main memory cost during autoregressive decoding. Recent compression methods cut this cost by 80\%--90\% with little accuracy loss on standard benchmarks, which has encouraged a common assumption that the cache is largely redundant. We question that assumption --- attention is not only a store of past tokens; it also decides how information is routed to the decoder, so a key--value pair can survive compression yet still be unreachable when the answer is produced. We therefore study KV compression as a controlled change to this routing, separating whether a token is kept from whether it can still be used. Using small, tightly controlled datasets for multi-entity tracking, disambiguation, coreference, and multi-hop reasoning, we report four major findings - (1) Moderate compression damages internal representations while leaving answers largely correct, confirming that much of the cache is redundant, (2) Near 90\% compression, every model we test shows a sharp jump in hallucination, a ``safety cliff'', that coincides with spikes in a simple quantity we call the Global Eviction Ratio (GER): the fraction of answer tokens dropped from every head at once, (3) The layers where routing is most fragile sit at different depths in different model families, so depth-based pruning rules do not transfer reliably, (4) Finally, even when answer tokens survive, too much agreement among attention heads can leave the model unable to re-route to them, a second failure mode we call rigidity. What limits compression, then, is not how many tokens are kept but whether a usable route to the answer survives, an inference-time counterpart to lottery-ticket sparsity in attention.
URL: https://openreview.net/forum?id=JJcm7lt0FY
---
Title: T3R: Deeper Test-Time Adaptation for Graph Neural Networks via Gradient Rotation
Abstract: Graph Neural Networks (GNNs) deployed in real-world systems typically have fixed weights, often leading to degraded performance under distribution shifts. This issue can be mitigated by conventional fine-tuning, but in many real-world cases, collecting labeled data is expensive or infeasible. A potential approach is Test-Time Training (TTT), which adapts models' weights using unlabeled test data, yet it is typically limited to shallow updates that affect only a subset of model parameters. We propose T3R, leveraging multiple Rotograd matrices to improve task affinity between the target and auxiliary tasks, essential for effective test-time training. T3R further introduces a rotation technique that reorients self-supervised signals using these matrices to create surrogate gradients for the target task, allowing deeper adaptation across nearly the entire architecture. Empirically, T3R reduces MAE by 0.172 points over standard inference in regression datasets and achieves at least 9.37% relative improvement on cross-domain OGB classification benchmarks compared to models without adaptation.
These results highlight the potential to develop an adaptation pipeline for graph-based systems, particularly in settings where conventional fine-tuning or retraining is infeasible.
URL: https://openreview.net/forum?id=fOPlYqoSnN
---
Title: Reproducibility and Validity at Risk: Reporting and Evaluation Inconsistencies in Fake News Detection Studies
Abstract: The growing body of fake news detection (FND) research has led to the development of nu
merous machine learning and deep learning (ML/DL) approaches. However, the increasing
volume of publications has not always been accompanied by corresponding methodological
rigor. As a result, concerns regarding reproducibility, reliability, and validity have attracted
increasing attention. In particular, inconsistencies in dataset construction and reporting,
inadequate evaluation practices, methodological flaws, and insufficient experimental trans
parency may compromise the credibility of reported results and impede meaningful com
parison across studies. This study was motivated by observations made during a systematic
mapping study of the FND literature. The review followed a comprehensive search process
across major digital libraries, initially identifying 1,030 publications. After applying prede
fined inclusion and exclusion criteria, 138 studies remained for full-text assessment. During
the second screening stage, each excluded study was manually examined and its exclusion
reason was documented individually. This process resulted in the exclusion of 78 studies,
whose reasons were subsequently categorized into high-level groups. Although the analysis
was conducted within the context of FND, the identified methodological, reporting, and
evaluation deficiencies extend far beyond this particular research area. Since most empiri
cal ML/DL studies follow a common experimental pipeline involving dataset acquisition or
selection, model development, and performance evaluation, these shortcomings may affect
research conducted across diverse application domains. To address these recurring issues,
we propose a practical checklist derived from the systematic analysis of the excluded studies
to support the transparent, reproducible, and methodologically sound design, implemen
tation, and evaluation of empirical ML/DL research. The checklist is intended to assist
authors, reviewers, and editors in assessing the methodological quality, transparency, and
reproducibility of ML/DL studies.
URL: https://openreview.net/forum?id=TPMakZm1vD
---
Title: A Discrepancy-Based Perspective on Dataset Condensation
Abstract: Given a dataset of finitely many elements $\mathcal{T} = \{\mathbf{x}_i\}_{i = 1}^N$, the goal of dataset condensation (DC) is to construct a synthetic dataset $\mathcal{S} = \{\tilde{\mathbf{x}}_j\}_{j = 1}^M$ which is significantly smaller ($M \ll N$) such that a model trained from scratch on $\mathcal{S}$ achieves comparable or even superior generalization performance to a model trained on $\mathcal{T}$. Recent advances in DC reveal a close connection to the problem of approximating the data distribution represented by $\mathcal{T}$ with a reduced set of points. In this work, we present a unified framework that encompasses existing DC methods and extend the task-specific notion of DC to a more general and formal definition using notions of discrepancy, which quantify the distance between probability distribution in different regimes. Our framework broadens the objective of DC beyond generalization, accommodating additional objectives such as robustness, privacy, and other desirable properties.
URL: https://openreview.net/forum?id=xByeYLCMMG
---
Title: Does the Explanation Drift First? Early Warning of Graph Neural Network Degradation
Abstract: Machine-learning models quietly lose accuracy when data drift occurs, and the labels that would reveal it often arrive late or never. A label-free clue is the model's own explanation: when its reasoning changes, something has shifted. Prior work detects this \emph{explanation shift} in tabular data but does not analyse whether it occurs \emph{before} accuracy drops. This paper focuses on graphs. Lead-time between explanation drift and the model's accuracy is measured on fully controlled, synthetic graph streams. The explanation drifts first, by a median of $15$ steps, with no false alarms, and a controlled sweep confirms this reflects a real mechanism, not an artefact. The lead is over accuracy, not over every detector. The effect also holds, albeit more weakly, on a realistic IT infrastructure graph (RCAEval). Explanation drift can therefore serve as a simple, label-free early warning that a graph model should be revalidated and updated to new conditions.
URL: https://openreview.net/forum?id=XMcbv3VuAV
---
Title: BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression
Abstract: Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations. We introduce a layer representation framework that extends activation-aware factorization beyond linear layers, including standard and grouped convolutions. Within this framework, our whitening-based formulation is more general than prior ones, naturally covering rank-deficient activations, and yields an optimal low-rank projection that attains the reconstruction error of the best low-rank approximation to layer activations. The resulting singular spectrum provides a closed-form per-layer distortion proxy, which we use to allocate per-layer ranks under explicit FLOP or parameter-count budgets via a Lagrangian relaxation with negligible overhead. Together, these components form BALF, an end-to-end pipeline for efficient vision model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets and remains competitive with other fine-tuning-free compression techniques.
URL: https://openreview.net/forum?id=zvewqh2j7F
---
Title: ALPS: AnneaLed Posterior Sampling for Inverse Imaging via Diffusion‑to‑Energy Distillation
Abstract: Diffusion models have emerged as the dominant prior for inverse imaging problems. Existing diffusion posterior sampling algorithms are constrained to follow the reverse diffusion trajectory - i.e., the prior path from Gaussian noise to clean images - while enforcing data consistency. Methods with convergence guarantees typically rely on computationally expensive ordinary or stochastic differential solvers to map the current noisy latent to a denoised image at each iteration, resulting in a multi-kernel convergence analysis. We propose ALPS (AnneaLed Posterior Sampling), which departs from the above paradigm by defining an explicit family of posterior distributions indexed by noise scale. It then runs a preconditioned Metropolis-adjusted Langevin algorithm at a discrete sequence of noise scales along this posterior path. The key enabler of ALPS is the multi-scale energy-based model (EBM) distilled from a pre-trained diffusion model. The resulting energy formulation enables ALPS to unlock three capabilities: analytical composition of the negative log-prior and negative log-likelihood at each noise scale in the discrete annealing schedule, which eliminates latent-image transport at each iteration; a Metropolis-Hastings correction that guarantees convergence at each scale without multi-kernel analysis; and maximum-a-posteriori estimation without contraction assumptions on the score model. We establish guarantees on the geometric ergodicity of each annealed chain, convergence of the annealed sequence to the true posterior, and a non-asymptotic mixing bound in which preconditioning mitigates the dominant ill-conditioning induced by the forward operator. The pseudo-inverse solution initialization at the coarsest scale and geometry-aware preconditioner further accelerate convergence in practice. Experiments on image inpainting, Gaussian, motion deblurring, and accelerated MRI reconstruction demonstrate consistent improvements over diffusion-based sampling algorithms. Notably, these gains arise from the posterior formulation rather than a stronger prior, as the EBM is distilled from the same diffusion model used by the baselines.
URL: https://openreview.net/forum?id=vdvgs0Jf01
---
Title: Interlaced Autoregressive Image Generation
Abstract: Autoregressive (AR) models inherently impose a 1D sequential dependency graph, which creates a fundamental structural mismatch when applied to 2D spatial manifolds like natural images. We introduce InterlacedAR, an autoregressive image generator that replaces the traditional left-to-right causal chain with a 2D spatial hierarchy. Inspired by classical graphics interlacing (e.g., Adam7 in PNG), our method partitions the token lattice into interleaved sublattices and decodes them level by level: early levels establish a sparse, isotropic global context, while later levels perform highly parallelized local refinements conditioned on previously generated neighbors. This design strictly preserves an exact likelihood under our proposed level-wise factorization, avoiding the heuristic approximations or multi-scale token bloat of prior parallel AR methods. On ImageNet 256 X 256, InterlacedAR-XL achieves an FID of 3.26 in just 10 decoding steps, running over 16X faster than raster-scan baselines while retaining comparable fidelity. Ultimately, InterlacedAR demonstrates that structurally aligning the decoding schedule with the 2D nature of images unlocks massive parallelism without sacrificing the simplicity of standard decoder-only Transformer backbones.
URL: https://openreview.net/forum?id=n9y2UqZt9w
---
Title: From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades
Abstract: Standard LLM Cascades improve efficiency by deferring difficult queries from weak to strong models. However, these systems are typically static: when faced with repeated or semantically similar queries, they redundantly consult the expensive model, failing to adapt during inference. To address this, we propose Inter-Cascade, an online, interactive framework that transforms the strong model from a temporary helper into a long-term teacher.
In our approach, when the strong model resolves a deferred query, it generates a generalized, reusable problem-solving strategy. These strategies are stored in a dynamic repository and retrieved via similarity matching to augment the weak model's context for future queries. This enables the weak model to "learn" on the job without expensive parameter fine-tuning. We theoretically show that this mechanism improves the weak model's confidence calibration. Empirically, Inter-Cascade outperforms standard cascades on multiple benchmarks, improving weak model and overall system accuracy by up to 33.06% and 6.35%, while reducing strong model calls by up to 48.05% and saving fee by up to 49.63%. Inter-Cascade demonstrates effective in-context knowledge transfer between LLMs and provides a general, scalable framework applicable to both open-source and API-based LLMs.
URL: https://openreview.net/forum?id=KUcCF5qqFL
---
Title: Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence
Abstract: Efficient multimodal inference is increasingly constrained not only by model quality or FLOP count, but by the cost of preserving, moving, routing, caching, and quantizing multimodal representations under low latency, memory, and energy budgets. This paper discusses the recent works on efficient vision-language and multimodal large language models across visual token redundancy and compression,
MLLM and video token management, KV-cache optimisation, Mixture-of-Experts routing and serving, low-bit quantization, edge deployment, and hardware-aware benchmarking. The central argument is that these techniques cannot be treated as independent optimisations. Visual token compression changes the evidence and feature distributions seen by downstream language layers and MoE routers; routing instability changes expert traffic and quantization sensitivity; quantized router logits can alter expert assignment; KV-cache policies determine which multimodal evidence remains available during generation; and edge hardware constraints can turn local FLOP savings into memory movement, cache pressure, or evidence loss. We therefore organize the literature around a failure-propagation chain that makes the interfaces between compression, routing, quantization, cache management, and hardware execution explicit. Beyond cataloging methods, the paper synthesizes recurring design tensions in deployable multimodal systems: accuracy versus token budget, static compression versus content-adaptive retention, sparse routing efficiency versus expert collapse, low-bit serving versus modality-specific degradation, and benchmark convenience versus end-to-end edge realism. We further introduced Temporal Routing Consistency as a diagnostic for video MoE models and identify open problems in routing-aware compression, cross-modal cache retention, proactive failure detection, hardware-aware co-design, and unified multimodal edge benchmarking.
URL: https://openreview.net/forum?id=rbj6jsUJtR
---
Title: Globally Optimal Training of Spiking Neural Networks via Parameter Reconstruction
Abstract: Spiking Neural Networks (SNNs) have been proposed as biologically plausible and energy-efficient alternatives to conventional Artificial Neural Networks (ANNs). However, the training of SNN usually relies on surrogate gradients due to the non-differentiability of the spike function, introducing approximation errors that accumulate across layers. To address this challenge, we extend the work on convexification of parallel feedforward threshold networks to parallel recurrent threshold networks, which subsume parallel SNNs as a structured special case. Building on this theoretical framework, we propose a parameter reconstruction algorithm for SNN training that demonstrates consistent and significant advantages across various tasks, both as a standalone method and in combination with surrogate-gradient training. The ablations further demonstrate the data scalability and robustness to model configurations of our training algorithm, pointing toward its potential in large-scale SNN training.
URL: https://openreview.net/forum?id=0WFP4dtDGP
---
Title: HERALD: Harmfulness Propagation Dynamics Layer-wise Trajectories of Adversarial Intent in Large Language Models
Abstract: We identify Harmfulness Propagation Dynamics (HPD). For harmful prompts, the projection of the last-token hidden state onto a learned harm direction rises monotonically with transformer depth, whereas benign prompts remain flat or oscillatory. This cross-layer signature reflects harmful intent as a progressively resolved semantic property: surface form appears early, while pragmatic intent consolidates later, making the trajectory shape more informative than any single-layer snapshot. Moreover, LDA-based harm directions, learned per layer, remain stable across random splits (pairwise cosine similarity $>0.97$), supporting the projection sequence as a reproducible structured signal. Building on HPD, we introduce HERALD (Harmful Encoding Recognition via Activation Layer Dynamics). This lightweight input moderator extracts a seven-dimensional feature record, slope, curvature, monotonicity, onset layer, and related statistics from the cross-layer projection sequence and classifies it with a 288-parameter MLP. \herald{} stores one $d$-dimensional direction per layer ($262$\,KB for a 32-layer, $d{=}4096$ model), requires no gradient computation during training, and adds only $2.6{\times}10^{-6}$ prefill FLOPs at inference. Across eight prompt-harmfulness benchmarks and four model families, HERALD achieves an average F1 score of $89.3$ on OLMo2-7B, surpassing all tested guard models on adversarial jailbreak detection ($98.4$ vs. $96.9$ F1) and outperforming prior latent-based methods by $2.3$-$4.1$ F1 points across all backbones. Per-instance trajectories provide machine-readable audit records that reveal when and how harmfulness emerges, offering an interpretability advantage over single-layer approaches.
URL: https://openreview.net/forum?id=5wXrijv1FF
---
Title: A Positional Realizability Theory of Length Generalization in Single-Head Attention
Abstract: Why does length generalization (LG) depend so sharply on the positional encoding (PE) — relative schemes extrapolating fixed offsets, absolute schemes fixed positions, neither handling growing offsets? We give a realizability theory for a deliberately controlled abstraction — a single positional-only attention head — that derives this empirical folklore and says precisely where it holds and breaks. The device is a simulation transfer operator $T_L$ that re-evaluates a selected length-$L_0$-realizing positional kernel on the longer grid, with a scalar $\alpha_L \in [0,1]$ read off before any training from $(\Phi, t_n)$ alone. We are explicit upfront: $\alpha_L$ is not proposed as a stronger empirical predictor; it is the computable witness of that taxonomy's derivation. Our results (each adversarially verified): a relative-head deep-row characterization, an absolute/relative duality that derives the folklore, a learned-absolute underdetermination result, and a static expressivity bound; a sufficiency theorem bridges $\alpha_L$ to LG, and the general deterministic case is a precise open conjecture (correcting a tempting but invalid "outside-colspan" argument). Experiments confirm the structure: in the controlled single-head regime $\alpha_L$ exactly separates the taxonomy's generalizing and non-generalizing (PE, task) cells (2625 trained-model measurements; stable at vocab 64 / $d{=}128$). Together, these results turn the PE folklore into a positional-kernel transfer theorem, and identify the content and depth mechanisms that escape it.
URL: https://openreview.net/forum?id=DRwVtZF4b6
---
Title: Offline Contextual Bandits in the Presence of New Actions
Abstract: Automated decision-making algorithms drive applications in domains such as recommendation systems and search engines. These algorithms often rely on off-policy contextual bandits or off-policy learning (OPL). Conventionally, OPL selects actions that maximize the expected reward from an existing action set. However, in many real-world scenarios, actions, such as news articles or video content, change continuously, and the action space evolves over time compared to when the logged data was collected. We define actions introduced after deploying the logging policy as new actions and focus on the problem of OPL with new actions. Existing OPL methods identify optimal actions from the existing set effectively. However, these methods cannot learn and select new actions because no relevant data are logged. To address this limitation, we propose a new OPL method that leverages action features. In particular, we first introduce the Local Combination PseudoInverse (LCPI) estimator for the policy gradient, generalizing the PseudoInverse estimator initially proposed for off-policy evaluation of slate bandits. LCPI controls the trade-off between reward-modeling condition and the condition for data collection regarding the action features, capturing the interaction effects among different dimensions of action features. Furthermore, we propose a generalized algorithm called Policy Optimization for New Actions (PONA), which integrates LCPI, a component specialized for new action selection, with Doubly Robust (DR), which excels at learning within existing actions. We define PONA as a weighted sum of the LCPI and DR estimators, optimizing both the selection of existing and new actions, and allowing the proportion of new action selections to be adjusted by controlling the weight parameter. Through extensive experiments, we demonstrate that PONA efficiently selects new actions while maintaining the overall policy performance as opposed to most existing methods that cannot select new actions.
URL: https://openreview.net/forum?id=3YTlnocq4P
---
Title: An Axiomatic Theory of Transferability Metrics with Provable Risk Bounds for Source-Model Selection
Abstract: The proliferation of pre-trained model repositories has created a fundamental source-
selection problem: given a target task and K candidate models, which source should guide
downstream training? Existing transferability metrics (LEEP, LogME, H-Score, and others)
are proposed as independent heuristics, each with its own assumptions and none with formal
guarantees. We argue that the field lacks a foundational answer to a prior question: what
properties should any transferability metric satisfy?
This paper provides such a foundation. We make three contributions. First, we intro-
duce four axioms (monotonicity, invariance, boundedness, and consistency) that formalize
the minimal requirements for a principled compatibility metric. From these axioms, we
derive the Representation-Task Alignment (RTA) score, which provably satisfies all four.
Second, we prove a transfer risk bound: the excess risk of linear transfer decomposes into
an approximation error controlled by the discriminative alignment and an estimation error
controlled by the structural alignment, with each term rigorously bounded under explicit
assumptions. Third, we show that LEEP, LogME, and H-Score are recoverable as special
cases or relaxations of RTA, each violating at least one axiom. Our results provide the first
axiomatic foundation for transferability estimation and the first formal connection between
a transferability metric and downstream excess risk.
URL: https://openreview.net/forum?id=DJkiREbPQM
---
Title: Repulsive Pheromones-Based Action Selection for Multi-Objective Reinforcement Learning in Continuous State and Action Spaces
Abstract: Many real-world control tasks contain multiple conflicting objectives that need to be optimized simultaneously. Multi-objective reinforcement learning (MORL) is one method to solve these problems. One challenge in multi-objective environments with large-dimensional action spaces, such as those in robotics, is that it becomes difficult to properly explore the entire search space while training. This paper introduces four novel action selection strategies within the MORL/D framework for multi-objective control tasks which are designed to improve the exploration of the action space while training, and are compared in six bi-objective simulated control tasks. Two of the proposed approaches, LI-$\tau$ and LD-$\tau$, which promote more exploration at the end and at the beginning of training respectively, had superior performance in the majority of the tested environments, and an adaptive approach which adjusts exploration depending on whether the performance has improved in the previous episode had the best performance in the largest action spaces which were tested. The experimental results showed that by promoting more exploration at different stages of the learning process, agents are better able to explore the action space, resulting in better quality final Pareto-optimal fronts (POFs), and are also often able to find a better-quality approximation of the POF earlier on in the training process over the baseline method.
URL: https://openreview.net/forum?id=kvlDUUlrU1
---
Title: The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
Abstract: Vision–Language Models (VLMs) have achieved strong performance on (implicit and explicit) visual grounding and related tasks. However, such abilities are generally tested on simple, single-object phrases. We find that grounding performance degrades for complex, multi-object references. These limitations largely arise from training objectives that leverage image-caption alignment, where direct multi-object references are rare, number of possible such references is theoretically large (exponential in the number of objects) and attribution is difficult. To address this, without requiring any additional annotations, we propose Compositional Attention-Regularized Training (CompART) training, which decomposes captions into object-centric phrases and constructs composite phrases by pairing them with conjunctions. We then introduce a composition loss that encourages the attention induced by a composite phrase to equal the sum of the attentions of its constituent phrases, promoting balanced multi-object localization. We evaluate CompART across four VLM architectures, spanning both contrastive-based and generative-based models, on four benchmarks for multi-object grounding and two VQA benchmarks for general visual understanding. CompART consistently achieves improved grounding for both single- and multi-object references across diverse VLM architectures and datasets, and further demonstrates enhanced visual understanding, as evidenced by gains on VQA, despite not being explicitly trained for the task.
URL: https://openreview.net/forum?id=HDTtgFBnLI
---
Title: JuryProbe: A Consensus-Risk Guardrail for Reference-Free Factuality Judge Panels
Abstract: Model-based evaluation systems increasingly use panels of inexpensive LLM judges to make accept-or-escalate decisions. In factuality settings, accepting a claim because several reference-free judges agree can create a hidden safety risk: agreement may arise from shared
false-negative blind spots rather than independent evidence. We introduce JuryProbe, a consensus-risk guardrail that decides when reference-free panel accept decisions should be routed to grounded verification. JuryProbe estimates panel-level consensus risk from a
calibration probe using false-negative-only (FN-only) judge correlation and false-consensus lift, then protects high-risk accept decisions by escalating them to the same judge panel with reference grounding. The grounded verifier is not a stronger oracle model; it is the
same panel given a trusted reference and aggregated by grounded majority. Using frozen FEVER-supported claim pools, fixed-seed construction, and audited Number and Entity corruption families, we show that reference-free judge panels exhibit substantial correlated
false negatives. Specifically, FN-only correlation is 0.402 for Number and 0.368 for Entity, with false-consensus lift of $3.13\times$ and $18.13\times$, respectively. Grounded verification collapses this failure mode, reducing unanimous false consensus to zero in both families. In 10-seed held-out deployment evaluations, JuryProbe-Routed reduces false accepts to 0.000 while using 49.6% fewer verifier calls than Always Grounded for Number and 62.1% fewer for Entity. Disagreement-based and budget-matched random routing fail to eliminate false accepts, showing that JuryProbe’s benefit comes from routing on measured consensus risk rather than generic escalation.
URL: https://openreview.net/forum?id=dgBczhxcZY
---
Title: Stable Continuous-Time Consistency Distillation: An Empirical Study with a Multistep Extension
Abstract: Continuous-time consistency models are an attractive route to few-step generation: they remove the discretization error inherent to the timestep grid that discrete-time consistency models depend on. However, until recently, training them was unstable, and Lu & Song (2025) address this with the TrigFlow formulation and a coordinated set of architectural and training-objective changes. We conduct an independent empirical study of this framework on CIFAR-10 and ImageNet-64. No open-weight TrigFlow teachers exist at these scales, so we train them from scratch following Lu & Song (2025)'s prescription and then verify the consistency distillation (sCD) and consistency training (sCT) behavior of students initialized from these teachers. On CIFAR-10 our students reproduce the reported FID at one and two steps. On ImageNet-64 they fall short, a gap we trace to our teacher and to architectural choices the recipe leaves unspecified. We also report a parameter-count gap of about $5\%$ between the model size stated in the paper and a faithful re-implementation, which we attribute to the pixel-normalization in Adaptive Double Normalization.
We then propose Multistep sCD (MS-sCD), a continuous-time, segment-conditioned multistep generalization of sCD. The segment-conditioning idea is borrowed from the discrete-time multistep consistency distillation of Heek et al. (2024). The rest of the algorithm follows sCD. MS-sCD replaces sCD's fixed two-step sampling schedule with a trained segment boundary and recovers sCD at $M=1$. We report FID at $M=2,4,8$ and compare against the discrete-time MSCD baseline. Code, training scripts, and all teacher and student checkpoints are provided as anonymized supplementary material with this submission, and will be released publicly upon acceptance.
URL: https://openreview.net/forum?id=di6ofoWEU8
---
Title: GRAFT: Gradient-Aware Fast MaxVol Technique for Dynamic Data Sampling
Abstract: Training modern neural networks on large datasets is computationally and environmentally
costly. We introduce GRAFT, a scalable in-training subset selection method that (i) extracts
a low-rank feature representation of the training data, (ii) applies a Fast MaxVol sampler to
pick a small, diverse subset that spans the data’s dominant subspace, and (iii) dynamically
adjusts the subset size using a gradient-approximation criterion. Because selection operates
inside the optimizer’s mini-batches, its cost is independent of dataset size, and the same
volume-maximizing criterion extends without change to one-shot curation of a full data
pool, including instruction-tuning corpora for language models. By operating in low-rank
subspaces and training on carefully chosen examples instead of full batches, GRAFT preserves
the training trajectory while reducing wall-clock time, energy, and CO2 emissions. Across
CIFAR-10/100, FashionMNIST, TinyImageNet, Caltech256, and ImageNet-1K (ResNet-18),
GRAFT lies on the Pareto frontier of accuracy and emissions against standard integrated
subset-selection methods, and unlike them it carries over to language-model fine-tuning; its
selection criterion further extends, without labels or gradients, to instruction-data curation,
where we give a preliminary demonstration that it selects more diverse subsets than the
usual baselines. At matched accuracy it consistently lowers CO2, and its margin is largest in
the aggressive-pruning regime (5 to 25% of the data), where principled selection matters
most. On ImageNet-1K, GRAFT comes within 0.4 points of full-data ResNet-18 accuracy
(70.0% against 70.4%) using 30% of the data and under a third of the emissions. We also
delimit where GRAFT does not help, namely near-full data fractions and highly imbalanced
batches, so practitioners can predict when the selection overhead pays off. An open-source
implementation will be released to support reproducibility
URL: https://openreview.net/forum?id=DDLMlVKQMw
---
Title: Sufficiency-Driven Structured Conditioning for Verifier-Guided Image Generation: A Rate–Distortion Analysis of Budgeted Local Refinement
Abstract: Caption-to-image pipelines compress a scene into a string of text and then ask a generator to invert that compression. The text channel is narrow, so information about layout, geometry, and style is discarded before generation even begins. We study this loss directly. We model conditioning as a communication channel and introduce a Structured Multimodal State (SMS) that factorizes a scene into objects, relations, layout, and style, and we prove that under a stated sufficiency assumption the SMS preserves the task-relevant information of the image up to an arbitrarily small slack, whereas a finite-capacity caption cannot. On a controlled factor-model testbed where mutual information is computable in closed form, the structured state recovers 85.7% of image variance against 43.5% for a caption-only channel of comparable role. We then cast verifier-guided correction as budgeted local refinement: specialized verifiers localize violations, and a corrector re-renders only the flagged region under a compute budget. We give convergence guarantees—sublinear in general and linear under restricted strong convexity—a sample-complexity bound for the verifier ensemble, and a cost analysis showing that reaching target error $\varepsilon$ takes $\mathcal{O}(\log(1/\varepsilon) \cdot \mathbb{E}|\mathcal{M}|)$ work rather than the $\mathcal{O}(HW \cdot T)$ of full regeneration. Every theorem is checked against simulation: measured convergence rates, the $\log(1/\varepsilon)$ scaling, and the verifier sample bound all match their predicted forms, and budgeted refinement traces the latency–quality Pareto frontier. We release all code and seeds.
URL: https://openreview.net/forum?id=rzVTfxdrnz
---
Title: PromptWise: Extending Contextual Bandits to Multi-Round Prompt Routing for Cost-Efficient LLM Inference
Abstract: The rapid advancement of generative AI has provided users with a wide range of well-trained models to address diverse prompts, with substantial variation in both performance and service cost. When selecting a model for a given prompt, users should therefore consider not only expected output quality but also the monetary cost of querying the model. Many existing approaches assume fixed or single-shot model assignment per prompt, whereas practical deployments may require online adaptation and cost accumulation across multiple attempts. In this paper, we introduce PromptWise, an online learning framework for cost-aware prompt assignment that learns from sequential interactions between prompts and models. PromptWise maintains and continuously updates estimates of prompt–model compatibility based on observed feedback, and uses these estimates to make adaptive, cost-sensitive decisions under uncertainty. Unlike standard contextual bandits that make a one-shot decision per prompt, PromptWise employs a cost-aware contextual bandit structure that allows sequential model assignments per prompt and explicitly accounts for the cumulative service cost incurred across multiple attempts. Through numerical experiments on tasks such as code generation and translation, we demonstrate that PromptWise achieves performance comparable to baseline selection methods while substantially reducing overall service cost.
URL: https://openreview.net/forum?id=mjkSi2oFLX
---
Title: DATE-GCN: Dynamic Untangled Adjacency Learning with Adaptive Temporal Encoding for Skeleton-Based Action Recognition
Abstract: Graph convolutional networks (GCNs) are commonly used for skeleton-based action recognition and have achieved remarkable performance. However, existing GCN-based approaches still face two major limitations: (1) Most models use predefined or fixed adjacency matrices to represent inter-joint relationships, which restricts their ability to adapt to the dynamic and context-dependent nature of human actions, and (2) their temporal modeling modules often fail to explicitly disentangle motion dynamics from feature extraction, resulting in limited capacity to capture complex and non-linear temporal dependencies. In this work, we present two main contributions: (1) a dynamic untangled multi-scale adjacency mechanism (DUMA) that adaptively learns action-dependent spatial relationships between body joints, and (2) an adaptive temporal encoding module (ATEM) that improves temporal modeling by decoupling temporal and feature channel interactions. Our full model, DATE-GCN, achieves state-of-the-art performance in skeleton-based action recognition
across major benchmark datasets: NTU RGB+D 60, NTU RGB+D 120, and Northwestern-UCLA. The source code of this work is available at: https://anonymous.4open.science/r/DATE-GCN-784C.
URL: https://openreview.net/forum?id=ip5EsaJlKW
---
Title: A Spectral Theory of Normalized Corrected GNN Propagation
Abstract: We develop a spectral theory for normalized corrected GNN propagation. The object of study is the symmetric normalized adjacency with its degree-stationary component removed, matching the normalization used by standard GCN-style models while isolating the stationary direction most directly tied to oversmoothing. The central theoretical question is whether this corrected normalized operator preserves class-discriminative signal after many propagation layers. Our main result is a high-probability exact-recovery theorem for the binary Contextual Stochastic Block Model after $k=O(\log n)$ propagation steps in the dense polylogarithmic regime $p \ge C\log^B n/n$, for any fixed $B>4$, under explicit graph-signal and feature-SNR conditions. We also establish a multi-class partial recovery theorem showing contraction toward class centers for most nodes. Synthetic and real node-classification experiments are included as empirical checks of the theory's predicted dependence on depth, graph signal, and feature noise.
URL: https://openreview.net/forum?id=tNkfnUa2Qb
---
Title: Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Abstract: Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Reinforcement learning (RL) has recently emerged as a dominant paradigm for improving this ability, yet existing work largely focuses on reward engineering while diverse training data remains scarce. We revisit this problem from a data-centric perspective and show that a simple yet effective data recipe alone, paired with a minimal outcome-based GRPO setup, suffices to substantially improve long-context reasoning. Our recipe targets three complementary task families -- retrieval, multi-evidence synthesis, and reasoning -- for which we construct and curate eight datasets totaling ~14K examples. Experiments on three models (Qwen3-4B/8B/30B-A3B) yield average gains of +7.2/+3.2/+6.4 points across seven long-context benchmarks, surpassing prior RL training sets. We further demonstrate that these gains transfer to agentic tasks, where continuing RL training on an agent-tuned model with our data recipe improves GAIA by +4.8 and BrowseComp by +7.0 points. We will release our datasets to facilitate future research.
URL: https://openreview.net/forum?id=l0GadGvuyA
---
Title: Multiple Imputation Guided by Full Law and Target Law Identifiability
Abstract: The central challenges in missing data models concern the identifiability of two distributions: the target law and the full law. The target law refers to the joint distribution of the data variables, whereas the full law refers to the joint distribution of the data variables and their corresponding response indicators. However, the relationship between the identifiability of these two distributions and the feasibility of multiple imputation has not been clearly established. We present a procedure where the choice of the imputation method is guided by identifiability considerations. We show that imputations can be drawn from the correct conditional distributions for all possible missing data patterns if and only if the full law is identifiable. This result implies that standard multiple imputation methods---which keep observed values unchanged and replace missing values with imputed values---are invalid when the target law is identifiable but the full law is not. We demonstrate that alternative imputation strategies can sometimes enable the estimation of the target law in such cases. Specifically, we introduce factorizable imputation where certain observed values are also imputed and the imputed data are weighted in the analysis.
URL: https://openreview.net/forum?id=j7ICPoiGTi
---
Title: EVA-Flow: Learning Shared Chemistry for Unified Environment-Aware Molecular Conformation Generation
Abstract: Molecular 3D structures underpin simulations of thermodynamics, kinetics, crystallization, and nucleation, making their accurate prediction central to drug discovery and materials design. However, molecular geometry can change dramatically across environments. Existing models either ignore environmental effects or train separate models for each environment. We hypothesize that transfer learning between different contexts is possible in conformer generation and that a general model will perform better in environments with limited training data: A very common and important setting including ligand binding and crystallization. We introduce EVA-Flow, a unified framework for environment-aware molecular conformation generation that learns shared chemical structure and adapts it through environment-conditioned generative modeling. Across four environments, vacuum, protein-ligand docking, solvation, and crystal packing, EVA-Flow substantially improves generation accuracy via pretraining and cross-environment finetuning. Analysis of molecules observed in multiple environments further shows that EVA-Flow produces distinct, physically valid, environment-specific conformations rather than memorizing a single canonical geometry.
URL: https://openreview.net/forum?id=pD4Smou3Xp
---
Title: Approaching the Harm of Gradient Attacks While Only Flipping Labels
Abstract: Machine learning systems deployed in distributed or federated environments are highly susceptible to adversarial manipulations, particularly availability attacks -- rendering the trained model unavailable.
Prior research in distributed ML has demonstrated such adversarial effects through the injection of gradients or data poisoning. In this work, we ask whether comparable degradation is still possible under a substantially more constrained action space: the adversary may only flip a limited number of labels of existing training examples, without modifying features, injecting samples, or directly controlling gradients.
We analyze the extent of damage caused by constrained label flipping attacks against distributed learning under mean aggregation -- the dominant baseline in research and production.
Focusing on classification problems, (1) we propose a novel formalization of label flipping attacks as a per-round constrained optimization problem, derive a greedy label-selection rule for logistic regression, and empirically evaluate it beyond its derivation setting, including on MLPs and robust aggregators. The rule is provably per-epoch optimal for the attacker under the mean aggregator.
(2) Empirically, we show that optimized label flipping can cause substantial accuracy degradation while outperforming random label flipping under similar budgets.
(3) We shed light on an interesting interplay between what the attacker gains from more \emph{write-access} versus what they gain from more \emph{flipping budget}.
(4) Finally, although the attack is derived for mean aggregation, we find that it can transfer empirically to the coordinate-wise median and trimmed mean aggregators, where its effectiveness is comparable to the Little-is-Enough gradient attack. This demonstrates that even highly constrained label-flipping adversaries can pose a significant availability threat to distributed learning.
URL: https://openreview.net/forum?id=eAqBcljvsU
---
Title: BhiliVaani: Dialect-Faithful TTS for a Low-Resource Indian Tribal Language via Staged Cross-Lingual Augmentation
Abstract: Endangered languages often lack text-to-speech (TTS) systems due to scarce paired speech data. We study Bhili, an Indian tribal language without existing TTS resources, and introduce \textbf{BhiliVaani}, consisting of {Core} (community utterances) and {Bridge} (synthetic utterances) corpus. Bridge audio is highly intelligible (MOS$_{\text{Int}}$ = 4.86 $\pm$ 0.08) but exhibits Hindi-accented prosody, motivating a two-axis benchmark evaluating intelligibility and dialect authenticity. We benchmark VITS, GlowTTS, and AlignTTS across five regimes: Original, Synthetic, Mixed, Syn$\rightarrow$Org, and staged Org$\rightarrow$Syn with incremental synthetic scaling (+2,+4,+6 h). Evaluation includes community MOS$_{\text{Int}}$/MOS$_{\text{Dial}}$, MUSHRA, diagnostic metrics (VAD-SNR, $\Delta$C50, F0), and manual error analysis. Results show that increasing synthetic data improves clarity but can reduce dialect authenticity. The Org$\rightarrow$Syn staged strategy reveals a controllable trade-off frontier and a preservation-oriented approach balancing intelligibility and dialect fidelity.
URL: https://openreview.net/forum?id=Qq7kbooBLz
---
Title: Controlling Calibration Robustness in Split Conformal Prediction Under Adversarial Attacks
Abstract: Conformal prediction (CP) provides distribution-free, finite-sample coverage guarantees but critically relies on exchangeability, a condition often violated under distribution shift. We study the robustness of split conformal prediction under adversarial perturbations at test time, focusing on both coverage validity and the resulting prediction efficiency. Our theoretical analysis characterizes how the strength of adversarial perturbations during calibration affects the coverage gap relative to the nominal coverage level under adversarial test conditions. We further examine the impact of adversarial training at the model-training stage. Experiments support our theory: (i) Prediction coverage varies monotonically with the calibration-time attack strength, enabling the use of nonzero calibration-time attack to predictably control coverage under adversarial tests; (ii) the marginal coverage can remain within a user-specified tolerance band around the nominal coverage level and (iii) adversarial training at the training stage yields a strictly smaller upper bound on the prediction-set size and empirically produces tighter sets, improving efficiency while maintaining coverage validity.
URL: https://openreview.net/forum?id=WjY1UCyIgb
---
Title: Learning in Markovian bandits with non-observable states and constrained decision epochs
Abstract: This paper studies the problem of regret minimization in Markovian bandits with non-observable states and possibly constrained decision epochs.
The focus is restricted to a ``pure'' regret benchmark, that compares the performance of the learning algorithm to the best pure policy which---akin to optimal policies of stochastic bandits---picks the optimal arm from start to finish without ever switching.
We introduce a generalization of rested Markovian bandits, self-degrading Markovian bandits, for which pure policies are always asymptotically optimal.
We show that without prior knowledge on the underlying bandit, the regret of algorithms that switch arms rarely necessarily scales super-logarithmically for every bandit, i.e., as $\omega(\log(T))$, where $T$ is the learning horizon.
Despite the unreachability of the logarithmic regime, we design UCB-NOM, an optimistic algorithm inspired by UCB, of which the regret is nearly logarithmic.
Lastly, we show that given prior knowledge on the Markovian bandit in the form of a bound on the bias functions of its arm, a proper instantiation of UCB-NOM achieves $\OH(\log(T))$ regret.
We further show that this prior knowledge allows for a $\OH(\sqrt{T \log(T)})$ worst-case regret bound for UCB-NOM.
Notably, our regret bounds do not depend on the number of states of the underlying Markov chains.
Our findings suggest that the non-observability of states is a mild inconvenience in self-degrading Markovian bandits.
URL: https://openreview.net/forum?id=kR04xLlazS
---
Title: Managing What to Remember: A Semantics–Realization– Dynamics Framework for External Long-Term Memory Systems in Large Language Model Agents
Abstract: External memory systems have become a core component of Large Language Model (LLM)
agents for supporting long-context reasoning, personalization, continual interaction, and
persistent task execution. Existing abstraction frameworks mainly separate representation
from implementation, but do not explicitly isolate the operational memory mechanisms
through which memory is accessed, selected, indexed, updated, consolidated, forgotten,
and governed during agent execution. As a result, important runtime memory behaviors
are often mixed with representation or storage concerns, making systematic analysis of
external memory architectures difficult. This survey analyzes 81 external memory systems
for LLM agents through a three-layer architectural decomposition consisting of the Memory
Semantics Layer, the Memory Realization Layer, and the Memory Dynamics Layer. The
Semantics Layer characterizes what memory units represent and how they are logically
structured. The Realization Layer characterizes how memory is physically instantiated across
storage substrates and infrastructural backends. The Dynamics Layer characterizes how
memory is operated during runtime. Using this decomposition, the survey identifies several
architectural patterns and imbalances across current systems. Procedural memory is largely
absent from structured representational forms despite the prevalence of structured declarative
memory. Sparse indexing appears in only 1 of 81 surveyed systems, with most architectures
heavily favoring dense retrieval mechanisms. In addition, forgetting operations are primarily
governed through heuristic policies, while consolidation mechanisms are predominantly
model-driven, suggesting a systematic asymmetry between memory removal and memory
integration strategies in current LLM external memory architectures. By separating memory
operation, memory meaning, and physical realization into interacting architectural layers,
this survey provides a unified framework for analyzing external memory systems and clarifies
several open challenges in scalable long-term memory management for LLM agents.
URL: https://openreview.net/forum?id=jWHNMGrHmS
---
Title: Guarding Concept Bottlenecks: A Voting-Margin Defense Against Concept-Level Backdoors
Abstract: Concept Bottleneck Models (CBMs) make predictions through human-interpretable concepts, but the same semantic interface can be exploited by concept-level backdoors: poisoned concept patterns steer the downstream classifier to a target label while preserving benign accuracy. We study the corresponding defense problem and introduce ConceptGuard, a concept-space partition-aggregation framework for CBMs. ConceptGuard partitions the concept vocabulary, trains lightweight concept-to-label classifiers on concept subgroups, and aggregates their predictions by majority vote. Rather than attempting to identify a specific trigger, the defense aims to scatter trigger influence so that only a small number of subgroup classifiers are effectively corrupted. We give a conditional voting-margin analysis that states when the ensemble prediction is stable, and we connect the theory to measurable diagnostics: trigger scatter, corrupted sub-model count, partial-trigger learnability, and clean voting margin. Across CUB and AwA under CAT/CAT+ attacks, ConceptGuard substantially reduces attack success while preserving clean accuracy. Additional evaluations cover partition-size effects, generic defense baselines, partition-aware adaptive attacks, noisy concepts, larger datasets, end-to-end encoder training, embedding choices, and computational overhead. The resulting claim is deliberately scoped: disjoint partitions support a structural voting-margin analysis, while random-overlap partitioning is evaluated as an empirical mitigation for adaptive attackers rather than as part of the proof.
URL: https://openreview.net/forum?id=BLEobQ8J84
---
Title: BayMOTH: Bayesian optiMizatiOn with meTa-lookahead - a simple approacH
Abstract: Bayesian optimization (BO) has for sequential optimization of expensive black-box functions demonstrated practicality and effectiveness in many real-world settings. Meta-Bayesian optimization (meta-BO) focuses on improving the sample efficiency of BO by making use of information from related tasks. Although meta-BO is sample-efficient when task structure transfers, poor alignment between meta-training and test tasks can cause suboptimal queries to be suggested during online optimization. To this end, we propose a simple meta-BO algorithm that utilizes related task information when determined useful, falling back to lookahead BO otherwise, within a unified framework. We demonstrate competitiveness of our method with existing approaches on function optimization tasks, while retaining strong performance in low task-relatedness regimes where test tasks share limited structure with the meta-training set.
URL: https://openreview.net/forum?id=HI08DxLCgA
---
Title: BigO(Bench) - Can LLMs Generate Code with Controlled Time and Space Complexity?
Abstract: We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generating code with specified time and space complexities. This benchmark addresses the gap in current evaluations that often overlook the ability of models to comprehend and produce code constrained by computational complexity. BigO(Bench) includes tooling to infer the algorithmic complexity of any Python function from profiling measurements, including human- or LLM-generated solutions. BigO(Bench) also includes a set of 3,105 coding problems and 1,190,250 solutions from Code Contests annotated with inferred (synthetic) time and space complexity labels from the complexity framework, as well as corresponding runtime and memory footprint values for a large set of input sizes. We present results from evaluating multiple state-of-the-art language models on this benchmark, highlighting their strengths and weaknesses in handling complexity requirements. In particular, token-space reasoning models are unrivaled in code generation but not in complexity understanding, hinting that they may not generalize well to tasks for which no reward was given at training time. Upon acceptance, we will release the complete codebase, annotated dataset, generated runtime and memory measurements, and interactive leaderboard.
URL: https://openreview.net/forum?id=8mb8cItG2h
---
Title: Structured Flows for Causal Modelling and Statistical Inference
Abstract: Probabilistic Graphical Models (PGMs) specify the conditional-independence structure of a system, yet continuous normalizing flows (CNFs) are typically structure-blind. We present \textbf{StructFlow}, a framework that compiles an arbitrary directed PGM into a task-specific block-triangular CNF. From a single graphical specification, StructFlow produces either a \emph{causal-alignment} model---preserving the generative factorization for interventional and counterfactual queries---or an \emph{inference-alignment} model---reordered and moralized to capture the denser posterior dependencies induced by ``explaining away''. We formalize when each compilation admits tractable queries, show that block-triangular velocity fields preserve the directed acyclic graph (DAG) factorization, and support both static and dynamic conditioning paradigms. Rather than introducing a new flow primitive, our contribution is to \emph{unify} the causal and inference directions for arbitrary DAGs behind a single automated compiler, and to \emph{characterize} when structure helps: we show that the compiler's moralization step is necessary to represent collider posteriors, that sparse structure acts as a beneficial inductive bias in the low-data regime, and that the resulting inference engine is amortized---answering posterior queries roughly an order of magnitude faster than per-query markov chain monte carlo (MCMC). We demonstrate these properties on synthetic linear-Gaussian and nonlinear PGMs (up to eight nodes) and on MNIST. The implementation is released as open-source software.
URL: https://openreview.net/forum?id=5cFwygAUFt
---
Title: Particles of Thought: Aligning Process Supervision with Test-Time Search
Abstract: Large language models have demonstrated strong mathematical reasoning, yet effective test-time scaling remains challenging. Automated process supervision methods train value functions on unguided rollouts, but inference-time search selectively retains high-value states, creating a distribution mismatch that degrades guidance quality. We propose Particles of Thought (PoT), a framework that aligns process supervision with test-time search. PoT formulates reasoning as probabilistic inference, using Sequential Monte Carlo at both training and test time to approximate the posterior over correct reasoning paths. We train our guidance model on particles sampled from an Oracle SMC process that filters based on rollout success; at test time, this learned guidance model guides the search, aligning training and inference distributions. Empirically, Guided SMC is the strongest test-time decoder at a matched sampling budget, outperforming Best-of-N and beam search on Math500 and OlympiadBench. Next, any gain from distribution-matched guidance falls within the standard deviation across seeds, so distribution matching gains are inconclusive. We further find that roughly half of outcome-correct solutions do not faithfully derive their answer, which may hinder learned guidance and limit its transfer to a different policy family.
URL: https://openreview.net/forum?id=v6DJbnt3qe
---
Title: RLPR: Extrapolating RLVR to general domains without verifiers
Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that the LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected reward during training. We find that addressing the high variance of this noisy probability signal is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR surpasses strong verifier-model-dependent approaches General-Reasoner by 1.7 average points across seven benchmarks.
URL: https://openreview.net/forum?id=p7QC2F4duc
---
Title: CoVAE: Consistency Training of Variational Autoencoders
Abstract: Variational Autoencoders (VAEs) provide a principled framework for representation learning and generation, but are limited by the use of a single latent distribution, resulting in a trade-off between reconstruction quality and generative performance. This limitation is commonly addressed through multi-stage pipelines that decouple representation learning from generative modeling.
We introduce Consistency Training of Variational AutoEncoders (CoVAE), a single-stage training framework that builds on the VAE encoder-prior-decoder structure by learning a continuum of latent representations with increasing regularization. The encoder produces noise-controlled latent embeddings governed by a time-dependent regularization parameter, smoothly transitioning from near-deterministic encodings to an isotropic Gaussian prior. The decoder is trained with a consistency-based reconstruction loss that enforces agreement across latent regularization levels while retaining KL-based variational regularization.
Rather than optimizing a new likelihood-based ELBO, CoVAE should be understood as a variationally regularized consistency objective over a time-dependent latent path.
This formulation enables joint representation learning and generation within a single autoencoding model, without auxiliary priors or multi-stage training. CoVAE generates high-quality samples in one or few decoding steps, significantly outperforming comparable single-stage VAE baselines while maintaining strong reconstruction performance. Overall, CoVAE offers an alternative paradigm for training generative autoencoders through structured, time-dependent latent representations.
URL: https://openreview.net/forum?id=hxvWseITKP
---
Title: Can Interpretable Clustering Trees Be Certified? Provable Bounds under Discretized Splits
Abstract: Interpretable clustering trees provide explicit rules for cluster membership, but existing solvers typically rely on heuristic search, leaving the quality of the returned tree unknown within the prescribed tree class. We present Certified Interpretable Clustering Trees (CICT), a branch-and-bound method that maximizes a regularized Dunn Index over a finite class of axis-aligned clustering trees with bounded depth, minimum leaf size, and discretized split thresholds. CICT derives prefix-dependent upper bounds by combining an upper bound on achievable inter-cluster separation with lower bounds on the maximum within-cluster diameter that any completion must attain. These diameter bounds use order statistics based on the remaining tree depth and pairs of observations that cannot be separated by any feasible split. The resulting bounds safely prune tree prefixes whose completions cannot improve the incumbent solution. CICT returns an interpretable clustering tree together with a certified residual optimality gap for the prescribed finite tree class. Experiments on benchmark datasets show that CICT achieves competitive regularized Dunn Index values, prunes large portions of the search space, and proves optimality on most tested instances.
URL: https://openreview.net/forum?id=4Kga5VWFMv
---
Title: Enabling Energy-Efficient Simultaneous Multi-Task Reinforcement Learning through Spiking Neural Networks with Active Dendrites for Bio-inspired Generalist Agents
Abstract: Reinforcement learning (RL) has demonstrated remarkable capabilities in training agents to solve complex tasks autonomously, such as mobile robots, UAVs/UGVs, and game-playing agents). However, scaling RL to master multiple tasks simultaneously (i.e., so-called multi-task RL) remains a significant challenge. Such a multi-task RL capability especially is important for agents to adapt to changes in real-world operational environments. State-of-the-art works show that, training agents with neural networks and shared structures across tasks promises improved generalization in simultaneous multi-task RL. However, they still suffer from task interference and incur high energy consumption due to intensive computation. To address this, we propose MTSpark, a novel methodology that enables energy-efficient simultaneous multi-task RL using spiking neural networks (SNNs) equipped with active dendrites for bio-inspired generalist agents. Specifically, MTSpark enhances a Deep Spiking Q-Network (DSQN) with active dendrites, a dueling structure, and task-specific context signals to dynamically form specialized sub-networks for individual tasks, while exploiting sparse operations for energy-efficient network processing. Experimental results demonstrate that MTSpark achieves higher performance and efficiency compared to state-of-the-art by obtaining high scores across three Atari games (i.e., Pong: -5.4, Breakout: 0.6, and Enduro: 371.2), approaching human-level performance (i.e., Pong: -3, Breakout: 31, Enduro: 368), while incurring similar memory and about 2x lower energy than state-of-the-art. These results show that our MTSpark potentially advances the frontiers toward energy-efficient generalist agents by combining RL and SNNs.
URL: https://openreview.net/forum?id=DKgBL6Skm3
---
Title: FG-Attn: Leveraging Fine-Grained Sparse Attention in Video Diffusion Models
Abstract: Using diffusion transformers for media generation may require evaluating attention over extremely long sequences, with attention layers accounting for the majority of generation latency. Exploiting sparsity in attention maps offers a promising opportunity to reduce this cost. In this work, we show that attention maps in diffusion transformers exhibit significant fine-grained sparsity in video generation models. Existing sparse attention methods, however, are too coarse-grained, leaving a large fraction of redundant computation unaddressed, or incur high overheads at finer granularity. We propose FG-Attn, a novel, low-overhead fine-grained sparse attention mechanism that skips score computations at the granularity of a M × N tile, where N ≥ 1 and M ≥ 16, and where each block is the result of query-key dot products between M queries and N keys. FG-Attn addresses the key challenge of hardware underutilization in sparse attention kernels on GPUs, without incurring the overheads of irregular memory access and redundant operations. FG-Attn can fully supersede existing sparse attention methods and extend block sparse attention methods to finer granularities on modern GPUs. At 70% sparsity, FG-Attn is up to 2.45× faster than state-of-art FlashInfer, and reduces attention kernel time by 14.7% on average. FG-Attn speeds up end-to-end video generation times by up to 1.40× (1.18× on average) over Flash Attention 3.
URL: https://openreview.net/forum?id=CtvvwWxSpR
---
Title: Structure-Aligned Protein Language Model
Abstract: Protein language models (pLMs) pre-trained on vast protein sequence databases excel at various downstream tasks but often lack the structural knowledge essential for some biological applications. To address this, we introduce a method to enrich pLMs with structural knowledge by leveraging pre-trained protein graph neural networks (pGNNs). First, a latent-level contrastive learning task aligns residue representations from pLMs with those from pGNNs across multiple proteins, injecting inter-protein structural information. Additionally, a physical-level task integrates intra-protein information by training pLMs to predict structure tokens. Together, the proposed dual-task framework effectively incorporates both inter- and intra-protein structural knowledge into pLMs. Given the variability in the quality of protein structures in PDB, we further introduce a residue loss selection module that uses a small model trained on high-quality structures to select reliable yet challenging residue losses for the pLM to learn. Applying our structure alignment method as a simple, lightweight
post-training step to the state-of-the-art ESM2 and AMPLIFY yields notable performance gains on tasks where structural signal is directly relevant, including deep mutational scanning (DMS) fitness prediction and a 59% increase in P@L/5 for ESM2 650M contact prediction on
CASP16. We further show that these gains scale with model sizes from 8M to 650M, while broader functional-property benchmarks often remain statistically inconclusive or saturated.
URL: https://openreview.net/forum?id=IUtttwuLNA
---
Title: SPL: Orchestrating Workflows with Declarative Deterministic–Probabilistic Composition
Abstract: We present SPL (Structured Prompt Language), a declarative language that composes \emph{deterministic} and \emph{probabilistic} computation modes in a single specification. While existing frameworks separate these---orchestration systems (AutoGen, CrewAI, LangGraph) for LLM calls, symbolic tools (SymPy, SageMath, Lean) for computation---SPL unifies them. It provides $\texttt{GENERATE}/\texttt{EVALUATE}$ for probabilistic computation and $\texttt{SOLVE}/\texttt{ASSERT}$ for deterministic computation, sharing syntax, variable bindings, and runtime routing. A \texttt{.spl} specification runs unchanged across local nodes (Ollama), cloud APIs (OpenRouter, Anthropic), and distributed grids (Momagrid), with model and verifier selection deferred to invocation time.
We validate SPL through an extensive $78$-recipe cookbook and a controlled $1{,}200$-run experiment ($10 \text{ models} \times 20 \text{ problems} \times 2 \text{ arms} \times 3 \text{ repetitions}$; the $20$ problems span $6$ difficulty tiers). The solver arm achieves $82$--$93\%$ machine-verified correctness (sonnet-4-6: $85\%$, gemma4:e2b: $93\%$) while the LLM-only arm measures output production without mathematical verification, making the comparison one of \emph{verified correctness} against \emph{unverified fluency}. A backend difficulty gradient emerges (SymPy $78\%$, Sage $54\%$), and the dominant failure mode is $\texttt{solver\_error}$ (kernel-rejected expressions), not format non-compliance.
URL: https://openreview.net/forum?id=1cub1C6m3Z
---
Title: Divisive Normalization Shapes Low-Rank Slow Manifolds for Continuous Working Memory
Abstract: The ability to robustly maintain and update continuous variables is a hallmark of working memory. While classical continuous attractor networks suffer from severe fine-tuning fragility, standard artificial recurrent neural networks (RNNs) like GRUs and LSTMs typically fail to stably learn continuous manifolds, instead shattering the state space into discretized point attractors. To bridge this gap, we draw inspiration from divisive normalization, a canonical neural computation widely observed across cortical circuits, and propose the Recurrent Divisive Normalization Network (RDNN), a biologically motivated architecture that replaces standard additive gating with a dynamic, multiplicative divisive bottleneck. Through dynamical systems analysis on canonical working memory tasks, we demonstrate that the RDNN naturally converges to robust, high-fidelity slow manifolds. Furthermore, we mathematically and empirically reveal that divisive normalization acts as a powerful activity-dependent implicit regularizer during Backpropagation Through Time (BPTT). This mechanism induces an efficient self-compression of the network's effective rank, confining the recurrent dynamics to a tight, low-dimensional subspace while avoiding the optimization pathologies associated with explicit low-rank factorization. Finally, ablations demonstrate that while subtractive inhibition can maintain static memories, divisive normalization is mathematically essential to prevent manifold shattering under time-varying inputs. Our findings identify divisive normalization not merely as a biological artifact, but as a critical computational mechanism for learning high-fidelity continuous representations.
URL: https://openreview.net/forum?id=a43l19lyfC
---
Title: Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
Abstract: Consistency models have been proposed for fast generative modeling, achieving results competitive with diffusion and flow models. However, these methods exhibit inherent instability and limited reproducibility when training from scratch, motivating subsequent work to explain and stabilize these issues. While these efforts have provided valuable insights, the explanations remain fragmented, and the theoretical relationships remain unclear. In this work, we provide a theoretical examination of consistency models by analyzing them from a flow map-based perspective. This joint analysis clarifies how training stability and convergence behavior can give rise to degenerate solutions. Building on these insights, we revisit self-distillation as a practical remedy for certain forms of suboptimal convergence and reformulate it to avoid excessive gradient norms for stable optimization. We demonstrate that our strategy extends beyond image generation to diffusion-based policy learning, without reliance on pretrained diffusion models for initialization, illustrating its broader applicability.
URL: https://openreview.net/forum?id=Qx2FrHDVzp
---
Title: Free Lunch Alignment of Text-to-Image Diffusion Models without Preference Image Pairs
Abstract: Recent advances in diffusion-based text-to-image (T2I) models have led to remarkable success in generating high-quality images from textual prompts. However, ensuring accurate alignment between the text and the generated image remains a significant challenge for state-of-the-art diffusion models. To address this, existing studies often employ reinforcement learning with human feedback (RLHF) to align T2I outputs with human preferences. These methods, however, either rely directly on paired image preference data or require a learned reward function, both of which depend heavily on costly, high-quality human annotations and thus face scalability limitations. In this work, we introduce Text Preference Optimization (TPO), a novel framework that enables ``free-lunch'' alignment of T2I models, achieving alignment without the need for paired image preference data. TPO works by training the model to prefer matched prompts over mismatched prompts, which are constructed by perturbing original captions using a large language model (LLM). Our framework is general and compatible with existing preference-based algorithms. We extend both DPO and KTO to our setting, resulting in TDPO and TKTO. Quantitative and qualitative evaluations across multiple benchmarks show that our methods consistently outperform their original counterparts, yielding superior human preference scores and better text-to-image alignment.
URL: https://openreview.net/forum?id=PnhruSkx25
---
Title: World Action Models: A Survey
Abstract: World Action Models (WAMs) are embodied predictive-action models that make a forecast of the future available to action. Recent WAMs repurpose large video generation models, and a parallel line relies on language or vision-language backbones without a video-generation core. This rapid expansion has blurred the boundary among broad world models, video generation models, action-grounded video world models, Vision-Language-Action policies, and WAMs. This survey gives the field a common account. It first clarifies these boundaries, then organizes existing works through two complementary views. The first view asks what each method is required to generate, spanning rendered futures, latent futures, and video-generation-free action reasoning. The second view decomposes each method by predictive substrate, backbone, action coupling, and deployment regime. This anatomy supports a unified discussion of interactability, causality, persistence, physical plausibility, and generalization, followed by data, evaluation, and open challenges. Across these axes, a consistent design pattern emerges: WAMs are not simply video generators with action heads, but predictive-action methods whose design choices trade representational richness against compute, memory, latency, and action-label cost. The field is moving toward methods that generate less of the future while preserving what control requires.
URL: https://openreview.net/forum?id=aazAKr44WU
---
Title: Channel Location Constrains the Auditability of Subliminal Learning
Abstract: Subliminal learning lets a student inherit a teacher's hidden trait from distillation data that never names it. We ask when such transfer can be audited before training. The answer is not model identity or scale alone, but channel location: the causal carrier through which the trait reaches the student. We find three regimes. In a controlled initialization-dependent body channel, a pre-training screen works. Coverage, the cosine between the student's initial distillation update and the teacher's fine-tuning displacement, predicts held-out transfer (Spearman $\rho \approx 0.95$; AUROC 0.997). In pretrained language models, masked single-token traits instead ride convergent vocabulary geometry. This channel is initialization-independent, so initialization-alignment screens, including coverage, are not mechanistic; the useful handles are post-hoc detection and targeted mitigation. Even when a single-token named entity is removed from the loss, the student's held-out probability for that entity rises to 0.40 on average ($\sim 2500\times$), and a related semantic class transfers. In an untied-head model, orthogonalizing the trait's output row against entangled neighbours collapses leakage, while equal-size random-subspace edits do not. Thus removing a target string from distillation labels does not remove the corresponding preference: neighbouring tokens can carry it. Finally, conditional behaviours can route through the network body. For sycophancy, with agreement and correction markers masked from the loss, transfer reaches about 0.63 of the teacher's effect, localizes to body computation, and evades four audits across two model families. We scope this as masked transfer of a condition-present policy. Channel location is necessary for deciding which audits can be sound. It is not a deployment-ready screen: an audit used outside its carrier regime can give false assurance.
URL: https://openreview.net/forum?id=PmC0JcOKkF
---
Title: Stacked Feynman-Kac: A Generalised Method for Within-Timestep Sampling of Intermediate Distributions for Diffusion Models
Abstract: Generative diffusion models have proven highly effective for both unconditional generation and conditional tasks. Sequential Monte Carlo (SMC) methods provide a principled framework for conditional sampling from diffusion priors, with applications including inpainting, super-resolution, and de-blurring. However, existing SMC-based approaches typically only propagate particles across timesteps and does not sufficiently explore the intermediate target distributions. To address this, we introduce the Stacked Feynman-Kac (SFK) algorithm, which enables sampling of intermediate distribution estimates at each timestep. We instantiate this framework with a Hamiltonian Monte Carlo proposal, which exploits gradient information to efficiently explore the intermediate targets. We further develop a computationally efficient variant of this approach and demonstrate that it achieves superior performance across a range of benchmark inverse problems and datasets.
URL: https://openreview.net/forum?id=AVVFwOIfLY
---
Title: Unified Text-Image-to-Video Generation: A Training-Free Approach to Flexible Visual Conditioning
Abstract: Text-image-to-video (TI2V) generation is a critical problem for controllable video generation using both semantic and visual conditions. Most existing methods typically add visual conditions to text-to-video (T2V) foundation models by finetuning, which is costly in resources and only limited to a few pre-defined conditioning settings. To tackle these constraints, we introduce a unified formulation for TI2V generation with flexible visual conditioning. Furthermore, we propose an innovative training-free approach, dubbed FlexTI2V, that can condition T2V foundation models on an arbitrary amount of images at arbitrary positions. Specifically, we firstly invert the condition images to noisy representation in a latent space. Then, in the denoising process of T2V models, our method uses a novel random patch swapping strategy to incorporate visual features into video representations through local image patches. To balance creativity and fidelity, we use a dynamic control mechanism to adjust the strength of visual conditioning to each video frame. Extensive experiments validate that our method surpasses previous training-free image conditioning methods by a notable margin. Our method can also generalize to both UNet-based and transformer-based architectures across various image conditioning settings.
URL: https://openreview.net/forum?id=eCJvmZMhNb
---
Title: Calibration, Not Channel: A Controlled Study of What Governs Foundation-Model Open-Set Recognition
Abstract: Open-set recognition (OSR) classifies inputs from known categories and rejects categories unseen in training. Foundation models supply two kinds of evidence for this decision. Semantic evidence comes from vision-language text prompts. Geometric evidence comes from image-only features. We organize foundation-model OSR into five evidence-combination strategies: semantic-only, geometric-only, disagreement-hybrid, naive fusion, and learned light fusion. All five share one decision interface. We evaluate them under matched, multi-seed protocols on CIFAR-10/100 and real ImageNet-200, alongside classical and recent post-hoc OOD detectors. We frame the question as two hypotheses. The Channel Hypothesis says the evidence channel, and how elaborately it is combined, drive rejection. The Calibration Hypothesis says the scoring and calibration of the evidence drive it. Our verdict is conditional. The backbone dominates absolute reliability. Once the backbone is fixed, the remaining variation follows calibration, not the raw channel or more elaborate fusion. Three results support this. First, a rejector learned directly from raw features is the worst and least-calibrated detector on every backbone, while calibrated \emph{learned light fusion} is the best on every vision-language backbone. Second, calibrated disagreement is statistically equivalent to naive fusion (TOST, $\pm1$\,pt, $p<10^{-11}$); disagreement is thus a special case of fusion, not a distinct signal. Third, density scores help on DINOv2 but weaken on vision-language features. The ImageNet-200 study adds ReAct, ASH, ViM, GEN, and a matched NegLabel detector. None overturns the ordering, and learned light fusion still leads every vision-language cell. A legacy head-to-head only corroborates. The mechanism is calibration of the evidence on a fixed backbone, not learned capacity or any single channel.
URL: https://openreview.net/forum?id=VIWGu1UByE
---
Title: Event-Sourced Memory for Multi-Agent Systems: An Auditable, Replayable Substrate with a Deterministic Context Policy
Abstract: Large Language Model (LLM)--based multi-agent systems increasingly coordinate many agents over long horizons, yet most still share memory through a mutable workspace---a blackboard, buffer, or retrieval store---that scales poorly with interaction length and discards the provenance of each decision as state is overwritten. We introduce \textbf{Event-Sourced Memory (ESM)}, which represents multi-agent memory as an append-only log of immutable, typed events. Each agent reconstructs its working context by a deterministic projection over the log, and a deterministic, training-free \textbf{Context Policy} selects relevant events under a bounded token budget using semantic, temporal, and role-dependency signals. Because the log is never overwritten and projections are pure functions of it, ESM provides two properties by construction that mutable or summarized memories cannot: exactly reproducible context reconstruction, and full causal provenance for every decision. Its cost profile is likewise decoupled from history length---$O(1)$ appends and $O(\log T + k)$ indexed retrieval, against the $O(T)$ of replay-based memory---which we establish analytically. A multi-agent system built on ESM attains $72.1\%$ on Terminal-Bench~2.0, an externally verifiable leaderboard result, and ablations isolate the contributions of immutability, projection, and selective retrieval. ESM uses no learned components and no third-party runtime dependencies, and is positioned as a deterministic, auditable substrate complementary to long-context models and learned-memory systems.
URL: https://openreview.net/forum?id=KnfO7Rh8R4
---
Title: A Stationarity-and-Coupling Criterion for Training-Free Time-Lagged Spectral Embeddings of Multivariate Time Series
Abstract: We study training-free fixed-length descriptors for multivariate time series and ask a question the representation-learning literature usually leaves implicit: not merely whether such a descriptor performs well on a benchmark, but when it can be expected to work at all. Our object of study is $D(\tau)$, a descriptor built from the time-lagged correlation matrix originally developed for random-matrix monitoring of network traffic (Rojkova & Kantardzic, 2007; Rojkova, 2010), truncated at the Marchenko--Pastur (MP) edge so that only signal-bearing eigenvalues survive, and classified by cosine similarity to class centroids with zero learned parameters. The central contribution is not the descriptor but a falsifiable applicability criterion for it. Working from a stationary Gaussian VAR(1) generative model, we argue that $D(\tau)$ separates two classes when the signals are approximately stationary and the class information resides in their cross-channel temporal coupling rather than in marginal per-channel power. We derive, semi-formally, three consequences of this model: a condition under which two classes are distinguishable in the embedded space, why the static ($\tau=0$) covariance alone collapses to chance, and why a stationary but power-discriminated paradigm defeats the descriptor even though every sample is well-conditioned. The criterion is operational: a two-part pre-flight test, an augmented Dickey--Fuller stationarity check together with a power-baseline saturation check, predicts applicability before any training, and we summarize it as a decision rule. We validate both halves of the boundary on a deliberately mixed assortment. On four paradigms that satisfy the criterion --- Sleep-EDF sleep staging, BCI-IV-2a motor imagery, MIT-BIH arrhythmia, and ESC-50 environmental sound --- the descriptor is competitive with strong baselines at a fraction of their cost, reaching $88.5 \pm 4.5%$ under 20-subject leave-one-subject-out on Sleep-EDF in about thirteen minutes on a single CPU thread with no GPU. On three paradigms that violate it --- transient-response ERPs, which are non-stationary, and financial-volatility regimes together with wearable stress detection, both power-discriminated --- the descriptor fails the criterion, collapsing to chance on the non-stationary case and being outperformed by a simple power baseline on the power-discriminated ones, exactly as the pre-flight test anticipates; these negative results are the more informative half of the story. We are explicit throughout that $D(\tau)$ is not the most accurate representation available; its value is a compact, training-free embedding whose domain of validity is known in advance.
URL: https://openreview.net/forum?id=FLrlAKfX75
---
Title: MERC: Co-evolutionary Multi-Geometry Message Passing for Zero-Shot Knowledge Graph Reasoning
Abstract: Structural knowledge graph foundation models enable zero-shot reasoning over unseen entities and relations by learning transferable structural patterns. However, most existing models rely on uniform entity-level message operators and largely static relation conditioning, which can impose fixed algebraic biases and limit adaptation to heterogeneous graph topologies. We propose MERC, a deterministic co-evolutionary multi-geometry structural KG reasoning model that couples parallel message passing in Complex, Split-Complex and Dual spaces with layer-wise entity-relation recalibration. The geometric branches provide complementary structural biases, while co-evolution dynamically adapts relation states using evolving entity contexts. Query-guided fusion then combines branch-specific evidence for each query. We evaluate MERC on 54 zero-shot KG benchmarks and 31 biomedical graphs. Results show that MERC improves average ranking performance over deterministic structural baselines, with clear gains under relation-vocabulary shift, compositional and antisymmetric patterns, and biomedical domain transfer. Ablation and efficiency analyses indicate that MERC’s gains arise from the interaction between multi-geometry operators and relation recalibration, while MERC Flash offers a practical efficiency-oriented variant.
URL: https://openreview.net/forum?id=bs2z167yQ6
---
Title: Restoring Vision Under Transitional Weather: A Benchmark for Autonomous Driving Perception
Abstract: Adverse weather conditions substantially degrade autonomous vehicle (AV) perception, with rain, fog, and snow causing severe drops in detection and segmentation accuracy. The challenge is amplified under transitional weather, where conditions evolve dynamically (e.g., foggy to rainy, snowy to foggy), introducing non-stationary domain shifts and temporally varying degradations that hinder both restoration and high-level perception. Existing restoration methods are limited to single, discrete weather types, while recent multi-weather approaches rely on multi-encoder designs that incur high computational costs and struggle to generalize to transitional conditions, thereby limiting scalability and real-time deployment. To address these limitations, we propose TransWeatherNet, a modular two-stage framework for restoring transitional weather to enhance AV perception. At its core, we introduce TWRFormer, a transformer-based restoration network that jointly models global spatiotemporal dependencies and localized degradations through an adaptive encoder-decoder with learned weather embeddings and modified perceptual loss, while employing a lightweight depthwise separable projection head for image reconstruction. Using restored images with gains of +2.01 \textit{dB} PSNR and +0.03 SSIM, TransWeatherNet boosts downstream perception, achieving improvements of +3.1\% mAP, +3.5\% IoU, and +0.65\% BLEU across object detection, semantic and instance segmentation, image captioning, and visual question answering. Extensive experiments validate state-of-the-art restoration and perception performance on AIWD16, Foggy Cityscapes, RainCityscapes, ACDC, and BDD100K.
URL: https://openreview.net/forum?id=MoodUEtci4
---
Title: All Roads Lead to CLIP: Mapping the Landscape of CLIP-Based Image Forensics
Abstract: The increasing adoption of generative models for image generation, including Generative Adversarial Networks and Diffusion Models, has led to the dissemination of synthetic content online.
These techniques enable even non-expert users to create highly realistic content, raising concerns regarding authenticity and potential misuse.
In response, the multimedia forensics community has developed numerous techniques to determine whether an image is synthetically generated and to identify which specific generative model produced it.
Recent studies have shown that features extracted by large pre-trained models, most notably CLIP, provide a highly effective representation for deepfake detection when combined with simple classifiers.
While several different CLIP-based approaches have been proposed, the majority of them adhere to a largely standard pipeline and differ primarily in implementation choices, such as the selection of the CLIP backbone, classifier architecture, image preprocessing strategy, and evaluation protocol.
The impact of these choices is frequently overlooked, and the rationale behind them is rarely justified.
In this paper, we systematically analyze CLIP-based synthetic image detectors, studying how design choices affect detection and attribution performance.
Moreover, we investigate the scalability of the considered CLIP backbones across different training set sizes.
We identify the best configuration, referred to as DejaVu, and evaluate its generalization capabilities and robustness to common post-processing operations.
Our results show that larger and more expressive backbones produce more discriminative representations, particularly when paired with deeper MLP classifiers, achieving strong performance even with limited training data.
Finally, we show that DejaVu exhibits robustness to common real-world post-processing, especially for diffusion-based images, and that generative models cluster meaningfully, largely due to shared architectural characteristics, enabling near-optimal performance with fewer training generators.
URL: https://openreview.net/forum?id=G2ksWOJ4Qa
---
Title: GRASP: Grouped Activation Shared Parameterization for Parameter-Efficient Fine-Tuning and Robust Inference of Transformers
Abstract: Parameter-efficient fine-tuning (PEFT) provides a scalable alternative to full-model adaptation by updating only a small subset of parameters in large pre-trained models. We introduce GRASP — Grouped Activation Shared Parameterization — a lightweight PEFT framework that partitions the $D$-dimensional token representations of selected layers into $K \ll D$ groups and learns a shared scaling and shifting vector for each group. This grouped modulation reduces the number of trainable parameters significantly while preserving the ability of the model to learn task-specific features. Across GLUE (RoBERTa-base & RoBERTa-large) and E2E NLG (GPT-2 Medium), GRASP demonstrates competitive performance compared to established PEFT methods while achieving an order of magnitude reduction in trainable parameters compared to LoRA and BitFit. Analyzing the learned parameters, we observe that aggressive grouping (small $K$) transforms their layer-wise distribution from unimodal to multimodal, indicating that the method encodes distinct task-relevant structure under tight parameter budgets. Motivated by this, we further propose StochGRASP, which learns Gaussian distributions as perturbations to the pre-trained weights rather than deterministic values. This probabilistic parameterization along with a noise-aware loss function formulation enables modeling hardware-level variability in programmed weights and improves robustness under non-ideal inference conditions—an important requirement for deployment on edge-based emerging AI hardware. In preliminary experiments under simulated inference-time noise, StochGRASP consistently outperforms deterministic variants, demonstrating its suitability for energy-efficient and noise-prone hardware platforms.
URL: https://openreview.net/forum?id=rRh4VpDulm
---
Title: Correct Before You Clip: A Bias Correction Method for Differentially Private Stochastic Gradient Descent
Abstract: Differentially Private Stochastic Gradient Descent (DPSGD) is a foundational method for privacy-preserving machine learning, but its performance is compromised by the bias introduced by gradient clipping. While recent methods show progress in mitigating the utility-damaging bias of gradient clipping in DPSGD, they often compromise the computational efficiency and ease of tuning essential for deploying modern large models and datasets. We propose a novel algorithm called DP-MBC, which introduces a pre-clipping mechanism that constructs a corrected gradient estimate by adjusting the current gradient with a momentum-driven correction. We provide a rigorous convergence analysis for DP-MBC under non-convex settings and show that it satisfies differential privacy guarantees. Extensive experiments using large-scale image and sentence classification benchmarks show that DP-MBC achieves better privacy-utility trade-off than existing state-of-the-art methods, with simplified hyperparameter tuning and reduced memory and computational demands.
URL: https://openreview.net/forum?id=KCTC5wg1xK
---
Title: Repulsive Ensembles for Bayesian Inference in Physics-Informed Neural Networks
Abstract: Physics-informed neural networks (PINNs) have proven an effective tool for solving differential equations, in particular when considering non-standard or ill-posed settings. When inferring solutions and parameters of the differential equation from data, uncertainty estimates are preferable to point estimates, as they give an idea about the accuracy of the solution. In this work, we consider the inverse problem and employ repulsive ensembles of PINNs (RE-PINN) for obtaining such estimates. The repulsion is implemented by adding a particular repulsive term to the loss function, which has the property that the ensemble predictions correspond to the true Bayesian posterior in the limit of infinite ensemble members. Where possible, we compare the ensemble predictions to Monte Carlo baselines. Whereas the standard ensemble tends to collapse to maximum-a-posteriori solutions, the repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity.
URL: https://openreview.net/forum?id=qNeue4f0uW
---
Title: In-Context Reinforcement Learning under Non-Stationarity: A Survey
Abstract: The development of decision-pretrained transformers, algorithm distillation, long-context meta-RL, and retrieval-augmented agents has renewed interest in in-context reinforcement learning (ICRL): the ability of a pretrained or fine-tuned decision model to infer latent task rules and improve future behavior from interaction context, without test-time parameter updates. This line of work asks when trial-and-error evidence, rewards, transitions, demonstrations, feedback, or retrieved experience can make learning-like computation happen inside the context window. However, existing surveys of ICRL mainly organize the field around pretraining objectives, architectures, context formats, evaluation protocols, and theoretical mechanisms, while the non-stationary setting remains comparatively underexamined. In changing environments, accumulated context is not merely more evidence about a fixed task: the reward specification, transition kernel, observation channel, action interface, constraint model, or demonstration and memory distribution can fall out of alignment with the current regime. Previously useful context can therefore become stale, misleading, or useful again when an old regime returns. We survey non-stationary ICRL as the problem of adapting through context while deployed policy parameters remain fixed: the policy must infer both the current decision rule and which parts of its accumulated evidence still support that rule. We define non-stationary ICRL, relate it to meta-RL, decision sequence modeling, retrieval-augmented RL, value- and model-aware ICRL, and reward-feedback agents, and organize the literature along three questions: what changes, how the change unfolds, and how observable the change is to the agent. We further analyze methods through context-management operations and influence facets, including what is written, retrieved, compressed, trusted, forgotten, or isolated, and argue that average return and held-out task generalization are insufficient to establish adaptation under drift. The survey identifies a research agenda around lifecycle evaluation, stale-context stress tests, validity-aware retrieval, adaptive forgetting, context poisoning, and finite-context theory for non-stationary ICRL.
URL: https://openreview.net/forum?id=03nykfEKZ3
---
Title: Efficient High-Resolution Image Synthesis via Autoregressive Modeling in a Diffusion-Learned Continuous Latent Space
Abstract: Autoregressive image generators predict next-token probabilities by computing a softmax over a codebook of V ≥ 8,192 indices at every sequence position. This O(V) prediction-head cost dominates inference wall-clock and scales with codebook size, restricting deployment on resource-constrained hardware and limiting the practical sequence lengths required for high-resolution synthesis. We replace the discrete softmax with O(D) diagonal-Gaussian regression in a continuous latent space refined by a small diffusion denoiser, reducing per-token complexity by a factor of V/D = 32 at our default D = 256. On ImageNet 256×256 with classifier-free guidance, our 775M-parameter instance generates images in 1050 ms versus 1590 ms for the strongest autoregressive baseline VAR-d30 (1.51× faster) at 8.4 GB versus 11.8 GB peak memory (29% reduction), and our 343M-parameter instance generates in 780 ms versus 1820 ms for VQGAN+Transformer (2.33× faster) at 6.2 GB versus 12.4 GB (50% reduction). These efficiency gains come at a quality cost of 0.48 FID against VAR-d30 (2.45 versus 1.97) and 0.17 FID against LlamaGen-XL (2.45 versus 2.62) on ImageNet, while on face and scene benchmarks our 775M model outperforms all autoregressive baselines (FID 7.61 versus 7.84 on FFHQ, 6.45 versus 6.93 on CelebA-HQ, and 5.24 versus 5.47 on LSUN Churches). Diffusion encoding contributes only 12.6% of total generation time, and the speed advantage compounds at 1024×1024, where our 775M model runs 1.38× faster than VAR-d30. The continuous head trades 0.48 FID for 1.51× throughput against VAR-d30, making the approach a natural fit for batch generation and high-resolution synthesis.
URL: https://openreview.net/forum?id=wkNwPZ0g3G
---
Title: Does Asking for Help Mitigate Goal Misgeneralization?
Abstract: Reinforcement learning (RL) agents have achieved impressive results but can struggle to generalize, leading to unexpected and potentially unsafe behavior.
Prior work discovered that, under certain distribution shifts in the test environment, agents competently pursue incorrect goals, even when they behave near-optimally in the training environment.
This phenomenon, termed goal misgeneralization, occurs when RL agents find proxy goals that correlate with the intended goal during training but diverge at test time.
We investigate the extent to which goal misgeneralization can be mitigated by allowing agents to ask an expert for help.
We find that, in existing goal misgeneralization environments, simple uncertainty estimation methods and heuristics can mitigate apparent GMG failures. However, this success relies on a particular property of existing benchmarks: achieving the proxy goal has no negative consequences, making delayed failure detection viable.
We propose modified versions of existing goal misgeneralization benchmarks that do not allow recovery.
We find that this drastically reduces the ability of existing query strategies to mitigate goal misgeneralization, highlighting their limitations.
We encourage future work to study this more challenging class of goal misgeneralization failures with stricter failure modes.
Code and videos are available at https://asking-for-help-gmg.github.io/
URL: https://openreview.net/forum?id=vi3JeZr01w
---
Title: Long-Horizon GUI Agents: A Survey of Benchmarks, Mitigation Methods, and Failure Types
Abstract: Graphical user interface (GUI) agents are autonomous systems that control computing devices through their native user interfaces. They perceive the screen, decide an action, and issue clicks, keystrokes, or touch gestures, without requiring the target software to expose an API. Driven by large multimodal models, these systems are moving from short research demonstrations to commercial deployments that run for hours and span dozens of applications. Evaluating their reliability at long task horizons is an open engineering problem. As the base models matured and commercial deployments grew, the evaluation community responded with a wave of new benchmarks and mitigation methods between 2024 and 2026, most of which appeared in the last year. We catalog twenty-five long-horizon GUI agent benchmarks across desktop, mobile, and web platforms and compare how each defines horizon, constructs tasks, and measures success. We catalog twenty mitigation methods and compare their mechanisms, claimed targets, and reported gains. We normalize the failure terminology used across these papers into a vocabulary of ten failure types, each described by at least two independent surveyed papers, and cross-analyze which types have dedicated methods. We find that horizon definitions are inconsistent across benchmarks, that most methods are evaluated on only one or two benchmarks, and that three failure types that recur widely in the benchmark literature (UI navigation knowledge gaps, catastrophic forgetting of task constraints, and misinterpretation of the task instruction) have few or no methods explicitly designed to fix them. A secondary analysis of published results shows that benchmark rankings of agents agree only weakly and that reported method gains decrease as benchmark horizon increases. The largest gains are reported on the shortest and most established benchmarks, which are also the ones that have been available long enough for methods to be tested on them. We argue that current evaluation practice is dominated by outcome-level metrics and underrepresents process-level reliability, which is the property that matters most at long horizons. We discuss the limitations of this analysis and list open problems for the field.
URL: https://openreview.net/forum?id=0zDStYC6Mk
---
Title: Test-Time Harness for Strong-to-Weak Capability Transfer
Abstract: Recent work on distillation transfers the capabilities of large models to smaller ones often by updating the latter's parameters, through teacher forcing, on-policy distillation, and related training-time methods. In this paper, we ask whether such transfer can instead occur at test time. We study strong-to-weak scaffolding: whether a stronger builder model can construct inference-time harnesses that help a weaker target model solve tasks more reliably without any parameter updates. Using four representative Theory-of-Mind benchmarks, each builder model uses 5% of the data as a validation set to iteratively refine its harness over multiple rounds, after which the finalized harness is evaluated on the full test set. Empirically, this form of test-time capability transfer is highly effective, nearly doubling average target-model performance from 0.49 to 0.91. Our analysis shows that the gains come primarily from offloading unstable model reasoning into deterministic code, benchmark-specific routing, and strict answer-format enforcement, rather than from encouraging the target model to reason more extensively or sample more broadly. We further find that builder-model reasoning effort improves harness quality monotonically, platform effects are modest relative to the builder model's own capability, and weaker target models receive the largest gains. These results suggest that inference-time harness design is an important complement to conventional training-time distillation, enabling strong models to transfer cognitive structure to weaker models without retraining.
URL: https://openreview.net/forum?id=igwmpjj3gb
---
Title: Policy Gradient for Continuous-Time Robust Markov Decision Processes
Abstract: The framework of robust Markov decision processes (RMDPs) allows the design of reinforcement learning agents that satisfy performance guarantees under worst-case transition dynamics. Traditional RMDPs consider discrete-time dynamics and recently, sample-efficient policy gradient algorithms have been considered in this context. This paper investigates policy gradient algorithms within a continuous-time RMDP framework. Policy gradients and adversarial gradients are derived using pathwise and adjoint-based formulas for stochastic and ordinary differential equations. We propose double-loop optimisers to obtain linear convergence in the oracle-based setting and an $\tilde{\mathcal{O}}(\frac{1}{\epsilon^2})$ sample complexity in the sample-based setting in an analysis which also derives novel tools for the framework of undiscounted total cost MDPs. Additionally, we propose mean-field optimisers as distributional optimisers with an $\tilde{\mathcal{O}}(\frac{1}{K})$ oracle-based convergence rate and an $\tilde{\mathcal{O}}(\frac{N^2}{\epsilon})$ sample complexity under $N$-particle approximation. The effectiveness of continuous-time policy gradient algorithms is confirmed for both optimisers on continuous-time RMDPs with neural ordinary differential equation dynamics.
URL: https://openreview.net/forum?id=IJaYVRDov8
---
Title: Multiple Treatments Causal Effects Estimation with Task Embeddings and Balanced Representation Learning
Abstract: The simultaneous application of multiple treatments is increasingly common in many fields, such as healthcare and marketing.
In such scenarios, it is important to estimate the single treatment effects and the interaction treatment effects that arise from treatment combinations.
Previous studies have proposed using independent outcome networks with subnetworks for interactions, or combining task embedding networks that capture treatment similarity with variational autoencoders.
However, these methods suffer from the lack of parameter sharing among related treatments, or the estimation of unnecessary latent variables reduces the accuracy of causal effect estimation.
To address these issues, we propose a novel deep learning framework that incorporates a task embedding network and a representation learning network with the balancing penalty.
The task embedding network enables parameter sharing across related treatment patterns because it encodes elements common to single effects and contributions specific to interaction effects.
The representation learning network with the balancing penalty learns representations nonparametrically from observed covariates while reducing discrepancies in representation distributions across different treatment patterns.
This process reduces the estimation variance caused by selection bias and avoids model misspecification.
Simulation studies demonstrate that the proposed method outperforms existing baselines, and application to real-world marketing datasets confirms the practical implications and utility of our framework.
URL: https://openreview.net/forum?id=xG2cjYSVjk
---
Title: Unimodal, Calibrated, and Order-Aware: A Deep Architecture for Reliable Ordinal Regression
Abstract: Ordinal regression is a supervised machine learning technique aimed at predicting the value of a discrete dependent variable with an ordered set of possible outcomes. It is often hindered by a trade-off between respecting the inherent order of classes and producing calibrated probability estimates. While methods based on maximum likelihood generally perform well in terms of calibration, they disregard the ordinal nature of the labels. Conversely, Optimal Transport explicitly models this order but generates poorly calibrated outputs. To resolve this conflict, we propose UnicornNet, the first approach to ensure all three critical properties: it leverages an \textbf{optimal transport (OT) loss} to model the ordinal structure, employs a novel accuracy-preserving procedure to guarantee \textbf{well-calibrated probabilities}, and incorporates an architectural design that enforces \textbf{unimodality} of the output distribution - a key inductive bias for many real-world ordinal regression tasks. On six real-world image benchmarks and a tabular dataset, UnicornNet matches or improves upon recent deep ordinal regression models on both accuracy and calibration metrics, while enforcing unimodality.
URL: https://openreview.net/forum?id=jtCnSIJ3r5
---
Title: Addressing the Representation Gap: A Study on Synthetic Face Generation for Underrepresented Global South Demographics
Abstract: Existing benchmark face image datasets are primarily scraped from the web and exhibit limited representation of the Global South. Text-to-Image (T2I) diffusion models are increasingly used to augment these datasets with synthetic data. Due to the inherent subjectivity in human-designed prompts and poor model alignment, it is challenging to generate realistic, non-stereotypical, high-fidelity images; consequently, these images often have limited utility in practice, particularly for auditing or debiasing face recognition systems. In this work, we introduce a novel framework -- SemCLoRA, that first aligns VLMs across multiple curated face image datasets to obtain more relevant attribute-rich prompts, followed by a novel "trajectory-sensitive" contrastive learning approach on text-to-image diffusion models to ensure greater prompt-image alignment. We evaluate the approach on two curated seed datasets representing different forms of visibility and underrepresentation: sportspersons from around the world (CheSS) and individuals from rural India (PARI). On both datasets, we observe lower FID and face-matching accuracy, indicating the efficacy of our framework in generating diverse, realistic images that are difficult to match to the training seed set. We further assess the utility of the synthetic images in an auditing & debiasing setting by fine-tuning a face recognition system for binary gender classification. We observe improved accuracy across eight datasets compared to a zero-shot scenario, with gains of up to ~22%, while simultaneously reducing the disparity in accuracy between the two binary gender groups by 51%. A user study with 57 participants reveals that humans have difficulty identifying synthetic images with an uncertainty rate of ~15%. We conclude with a discussion of responsible-use guidelines and the policy and governance implications of using synthetic data for auditing face recognition systems.
URL: https://openreview.net/forum?id=tCLem18ZO8
---
Title: Spectral and Conformal: Neural Network Methods for Calibrated Trajectory Prediction and Control-Point Intervals
Abstract: Autonomous systems that navigate their environment need methods to quickly assess the feasibility and safety of desired trajectories. Traditional solutions that rely on numerical integration can be computationally heavy, while fast data-driven surrogates lack the formal guarantees that numerical approaches provide. This work demonstrates a framework that uses (1) a neural network with a Koopman-like spectral backbone to learn trajectories and quantile bands with built-in constraints (initial conditions and stability); (2) a modified conformal quantile regression to provide calibrated \emph{joint}, trajectory-level conformal coverage; and (3) interval arithmetic to extend coverage guarantees to control-points. For a quadrotor under a closed-loop linear quadratic regulator, we show that by refactoring the network for inference, the trajectory inference time is 3.5~ms on a Raspberry Pi Zero 2 W, within a total pipeline time of 9.81~ms, enabling operation at approximately 100~Hz, within the range of typical control rates. The framework shows that learned surrogates can compactly express meaningful statistical guarantees at control-relevant speeds on constrained hardware.
URL: https://openreview.net/forum?id=zKLMev1mnQ
---
Title: GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
Abstract: In-Context Learning (ICL) has become a powerful mechanism for adapting Large Language Models (LLMs) to new tasks without fine-tuning. Extending this concept to Large Multimodal Models (LMMs), Multimodal In-Context Learning (M-ICL) relies on retrieving relevant examples, such as images, captions, or question-answer pairs, to guide predictions across tasks like classification, captioning, and visual question answering (VQA). Most existing approaches select in-context examples based on feature-space similarity, assuming that semantically similar samples provide the most useful context. However, our systematic analysis reveals that this assumption does not always hold: visually similar examples are not necessarily those that most effectively enhance in-context learning performance.
To address this, we propose the Guided Retrieval of In-context Prompts (GRIP), a learnable vision-only retrieval framework that leverages feedback from LMMs to identify examples that truly improve model predictions. GRIP learns to distinguish beneficial from detrimental in-context examples through contrastive training, refining retrieval beyond pure similarity. Across three multimodal tasks, namely classification, captioning, and VQA, GRIP improves consistently over similarity-based retrieval on Qwen2.5-VL-7B, with its strongest gains in classification on Idefics2-8B. Moreover, we demonstrate that retrievers trained with feedback from one open LMM can be transferred to other models without retraining, including closed-source GPT-4o and Gemini, enabling scalable and cost-efficient deployment of M-ICL. Code will be published upon acceptance.
URL: https://openreview.net/forum?id=sEraextJ8C
---
Title: Less is More: Compressed Reasoning with Large Language Models via Structured Prompting
Abstract: Chain-of-thought (CoT) prompting significantly improves LLM performance on reasoning and mathematical tasks by enabling transformers to perform complex computations across multiple inference steps. Modern pre- and post-training pipelines have amplified this: LLMs are now post-trained with explicit CoT supervision during SFT and further reinforced via RL to produce longer, more structured reasoning chains, achieving state-of-the-art results on benchmarks such as MATH-500, GSM8K, and ARC-Challenge, but at the cost of significantly increased token generation. In this paper, we theoretically analyse the reason CoT improves the ability of LLMs to perform reasoning tasks, and identify two properties - step-complexity and step-expressiveness - that any compressed reasoning format must preserve to maintain CoT-level performance. We show that existing concise prompting strategies fail because they violate one or both of these properties. As an instantiation of our analysis, we introduce TeleMathLang (TML), a minimal structured syntax for reasoning that enables LLMs to generate complete reasoning chains while reducing response length by 25–69% across MATH-500, GSM8K, and AI2-ARC, without requiring further post-training which could be expensive and time-consuming. TML works as a prompting strategy with any instruction-tuned model, and we further show via conditional generation that models already trained via SFT and RL to produce long reasoning chains can be efficiently fine-tuned to generate TML output without accuracy loss. TML outperforms existing concise prompting baselines in accuracy across all three benchmarks.
URL: https://openreview.net/forum?id=VcpaalTs87
---
Title: Training Language Models for Bilateral Trade with Private Information
Abstract: Bilateral bargaining under incomplete information provides a controlled testbed for evaluating large language model (LLM) agent capabilities, as it demands individual rationality, strategic surplus maximization, and cooperation to realize gains from trade. We develop a structured bargaining environment in which LLMs negotiate via tool calls within an event-driven simulator, separating binding offers from natural-language messages to enable automated evaluation. The environment serves two purposes: as a benchmark for frontier models and as a training environment for open-weight models via reinforcement learning.
In benchmark experiments, a round-robin tournament among five frontier models (15,000 negotiations) reveals that effective strategies implement price discrimination through sequential offers. Aggressive anchoring, calibrated concession, and temporal patience are associated with both the highest surplus share and the highest deal rate. Accommodating strategies that concede quickly disable price discrimination in the buyer role, yielding the lowest surplus capture and deal completion. Strategically competent models scale their behavior proportionally to item value, maintaining consistent performance across price tiers; weaker models perform well only when wide zones of possible agreement compensate for suboptimal strategies.
In training experiments, we fine-tune Qwen3 (8B, 14B) via supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) against a fixed frontier opponent. The two stages optimize competing objectives: SFT approximately doubles surplus share but reduces deal rates, while RL recovers deal rates but erodes surplus gains---a tension traceable to the reward structure. SFT also compresses surplus variation across price tiers, and this compression generalizes to opponents unseen during training, suggesting that behavioral cloning instills proportional strategies rather than memorized price points.
URL: https://openreview.net/forum?id=HEziIcixje
---
Title: Selective Growing Q-Networks (SGQN): Value-Guided Adaptive Discretisation for Continuous Control
Abstract: One way to tackle reinforcement learning for continuous control is to discretise the action space and apply value-based Q-learning, where the resolution of that discretisation fundamentally shapes the exploration--exploitation trade-off. Coarse discretisations facilitate early exploration but limit final precision; fine discretisations enable precise control but slow convergence. Existing adaptive approaches such as Growing Q-Networks (GQN) address this by growing uniformly from coarse to fine resolution, agnostic to which regions of the action space are actually promising. We introduce \textbf{Selective Growing Q-Networks (SGQN)}, an algorithm that replaces uniform growth with value-guided selective expansion: bins are refined only in action regions where Q-values are high, while persistently low-value bins are lazily pruned. During action selection, a temperature-annealed softmax over active bins concentrates probability mass on high-value regions as learning matures. SGQN maintains the scalability of decoupled Q-learning while focusing representational capacity where it matters most. On six tasks from the DeepMind Control Suite spanning 2D to 12D action spaces, SGQN achieves competitive or superior performance to strong baselines. These results suggest that value-guided selective discretisation is a competitive approach to continuous control, though per-component attribution of the performance gains requires further ablation.
URL: https://openreview.net/forum?id=zNjdjWaN6H
---
Title: Controlling Foundation Models via Internal Interventions: A Survey of Inference-Time Methods
Abstract: Large generative foundation models exhibit broad but often implicit behaviors that researchers have begun to steer by intervening in model computation at inference time. Rather than updating model parameters or relying solely on explicit prompts, inference- time interventions modify hidden activations, sparse features, attention and routing patterns, or token-selection scores during the forward pass and decoding. These methods offer a potentially flexible, request-time framework for post-training control, but they differ in
required model access, computational overhead, robustness, and side effects. This survey reviews such methods under a common intervention lens. We categorize the literature into representation-level, attention-level, and decoding-level interventions, and analyze them
across large language models, vision-language models, and diffusion models. We make the scope boundary explicit by separating core fixed-weight, inference-only methods from adjacent paradigms such as prompting, retrieval, fine-tuning, weight editing, and newly trained
controllers. To support comparison, we describe transformer, multimodal, and diffusion generation using computational-graph notation; summarize reported performance deltas, access requirements, and risk profiles; and connect empirical steering methods to evidence
for approximately linear representations and sparse feature decompositions. We close by outlining practical gaps: how to compare steering methods on shared benchmarks, decide where an intervention should be applied, measure unintended side effects, and combine
multiple intervention types without interference.
URL: https://openreview.net/forum?id=NIwL22qGYy
---
Title: Prover Agent: An Agent-Based Framework for Formal Mathematical Proofs
Abstract: We present Prover Agent, a novel AI agent for automated theorem proving that integrates large language models (LLMs) with a formal proof assistant, Lean. Prover Agent coordinates an informal reasoning LLM, a formal prover model, and feedback from Lean while also generating auxiliary lemmas. These auxiliary lemmas are not limited to subgoals in the formal proof but can also include special cases or potentially useful facts derived from the assumptions, which help in discovering a viable proof strategy. It achieves an 88.1% success rate on MiniF2F and solves 25 problems on the PutnamBench with a smaller sample budget than previous approaches, establishing a new state-of-the-art on both benchmarks among methods using small language models (SLMs). We also present theoretical analyses and case studies that illustrate how these generated lemmas contribute to solving challenging problems. Our code is publicly available in the supplementary material.
URL: https://openreview.net/forum?id=05islLXVt6
---
Title: Evaluating the Effect of Numerical Solvers on Learning in Neural ODEs
Abstract: Continuous-depth neural networks, or Neural ODEs, are usually evaluated based on ac
curacy, speed, or memory use, and the numerical solver is often treated as a technical
implementation choice. In this paper, we provide evidence that this assumption can fail
under controlled training conditions: solver choice is associated with measurable changes in
the learned representation dynamics. We first examine this using a frozen-weight interven
tion: a model is trained with one solver and then evaluated with another, without changing
the weights. This replacement can lead to a large drop in performance, indicating that the
solver is not always functionally interchangeable after training under the tested setup. To
investigate this behavior, we introduce a third-order Neural ODE formulation that enables
direct measurement of higher-order trajectory dynamics and representation evolution with
out requiring nested differentiation through the solver. Across CIFAR-10, CIFAR-100, and
Tiny-ImageNet, we observe that different solvers produce distinct representation dynam
ics, even when final clean accuracy is closely matched. We also use adversarial robustness
as a downstream diagnostic of these learned representations. In some regimes, models with
larger higher-order trajectory-norm quantities tend to exhibit improved robustness under it
erative attacks. However, at larger integration horizons, robustness appears to depend more
strongly on how the learned dynamics interact with the solver stability envelope, making
the relationship regime-dependent. Finally, we provide evidence that even solvers with the
same external stability polynomial can lead to different learned behaviors due to differences
in internal stage realization. Overall, our results suggest that numerical integration can act
as an important factor in Neural ODE training, with measurable associations to learned
representation geometry and robustness beyond clean predictive accuracy
URL: https://openreview.net/forum?id=Mopa70gMnp
---
Title: GLOPaint: History-Normalized Loss Balancing for Resource- Efficient Graph-Based High-Resolution Image Inpainting
Abstract: High-resolution image inpainting requires restoring missing regions with high reconstruction fidelity, perceptual consistency, and structural coherence while remaining practical in model size, memory usage, and inference latency. We propose GLOPaint, a lightweight graph-based framework for efficient high-resolution image inpainting. GLOPaint combines edge-aware convolutional feature extraction with graph attention reasoning at the bottle- neck, allowing compact appearance features and predicted boundary cues to interact through a feature-space graph rather than fixed spatial proximity. This design supports structure- aware information exchange between semantically related regions while avoiding dense full- resolution graph construction. To stabilize training with heterogeneous objectives, we further introduce Historical Moving Average (HMA) weighting, a history-normalized loss balancing mechanism that adjusts each objective’s dynamic weight according to its own optimization trajectory rather than relying on fixed coefficients or direct cross-objective normalization. Experiments on Places2 and CelebA-HQ at 1024 × 1024 resolution show that GLOPaint achieves the lowest LPIPS on both datasets and competitive PSNR among the compared high-resolution baselines. With 13.6M parameters and around 2.9 ms inference time per image, GLOPaint offers a favorable quality–efficiency trade-off for deployment- oriented high-resolution inpainting, particularly in settings where training data and GPU resources are constrained. These results suggest that compact feature-space graph reasoning, combined with history-normalized objective balancing, provides an effective and computationally practical direction for structure-aware image completion at high resolution
URL: https://openreview.net/forum?id=P4ROtojDRc
---
Title: CHIMERA-Bench: A Benchmark Dataset for Epitope-Specific Antibody Design
Abstract: Computational antibody design has seen rapid methodological progress, with dozens of deep generative methods proposed in the past three years, yet the field lacks a standardized benchmark for fair comparison and model development. These methods are evaluated on different SAbDab snapshots, non-overlapping test sets, and incompatible metrics, and the literature fragments the design problem into numerous sub-tasks with no common definition. We introduce CHIMERA-Bench: (CDR Modeling with Epitope-guided Redesign), a unified benchmark built around a single canonical task: epitope-conditioned CDR sequence-structure co-design. CHIMERA-Bench provides three components. The first is a curated, deduplicated dataset of 2,922 antibody-antigen complexes with epitope and paratope annotations. The second is a set of three biologically motivated splits that test generalization to unseen epitopes, unseen antigen folds, and prospective temporal targets. The third is a comprehensive evaluation protocol with five metric groups, including novel epitope-specificity measures. We benchmark eleven methods spanning six generative paradigms and report results across all splits. CHIMERA-Bench is the largest dataset of its kind for the antibody design problem, allowing the community to develop and test novel methods and evaluate their generalizability.
URL: https://openreview.net/forum?id=3m8uaq7luq
---
Title: Coarse-to-Fine: A Reproducibility Study of Hierarchical Sparse Representations for Multimodal Interpretation
Abstract: Sparse Autoencoders (SAEs) aim to improve interpretability through disentangled features
in its latent code. Zaigrajew et al. (2025) build upon SAEs by proposing Matryoshka SAEs
(MSAEs), which learn hierarchical features at increasing granularities and show improved
performance over existing SAE models. Furthermore, the authors show that MSAEs can
be applied to concrete problems by interpreting CLIP representations. Though our work
validates their claims, our results slightly differ from those of the original work. We also
identify a population of always-on neurons specific to MSAE and show that a targeted L1
penalty removes them at a small cost to reconstruction. We further introduce a concept
saliency method that localizes MSAE features within an image, complementing the model’s
image-level concept decomposition with a spatial, pixel-level account.
URL: https://openreview.net/forum?id=IKlxz4pB2s
---
Title: Enhancing Quantum Machine Learning: The Power of Non-Linear Optical Reproducing Kernels
Abstract: Amidst the array of quantum machine learning algorithms, the quantum kernel method has emerged as a focal point, primarily owing to its compatibility with noisy intermediate-scale quantum devices and its promise to achieve quantum advantage. This method operates by nonlinearly transforming data into feature space constructed with quantum states, enabling classification and regression tasks. In this study, we present a novel feature space constructed using Kerr coherent states, which generalize, coherent states, and squeezed states. Notably, the feature space exhibits constant curvature, comprising both spherical and hyperbolic geometries, depending on the sign of the Kerr parameter. Remarkably, the physical parameters associated with the coherent states, enable control over the curvature of the feature space.
Our study employs Kerr kernels derived from encoding data into the phase and amplitude of Kerr coherent states. We analyze various datasets ranging from Moon to breast cancer diagnostics. Our findings demonstrate the robustness of Kerr coherent states, attributed to their flexibility in accommodating different hyperparameters, thereby offering superior performance across noisy datasets and hardware setups.
URL: https://openreview.net/forum?id=nWLsKNTGMU
---
Title: Towards Autoregressive Neural Network Parameter Generation
Abstract: Large Language Models (LLMs) have recently established a new paradigm in machine learning, demonstrating exceptional zero-shot generalization and complex reasoning capabilities. Concurrently, neural network parameter generation has emerged as a promising avenue for rapid task adaptation and model synthesis. However, current weight generation techniques struggle with out-of-distribution generalization. Training a unified model on a diverse corpus of pretrained weights remains highly challenging due to the continuous, high-dimensional nature of parameter spaces and the difficulty of conditioning the generation on novel tasks. To mitigate these limitations, we propose a novel, LLM-inspired approach that formulates network parameter generation as an autoregressive modeling problem. By leveraging a decoder-only transformer architecture, our method treats sequential parameter spaces analogously to linguistic tokens, enabling scalable and conditionally accurate weight synthesis.
Extensive experiments on multiple vision datasets demonstrate that our method consolidates diverse pretrained models into a single flexible generative framework. The synthesized parameters achieve competitive or superior performance compared with state-of-the-art baselines, with particular advantages in scalability and efficiency when applied to larger architectures.
URL: https://openreview.net/forum?id=8Ieryq0C7c
---
Title: Uncertainty Distillation: Teaching Language Models to Express Semantic Confidence
Abstract: As large language models (LLMs) are increasingly used for factual question-answering, it becomes more important for LLMs to have the capability to communicate the likelihood that their answer is correct. For these verbalized expressions of uncertainty to be meaningful, they should reflect the error rates at the expressed level of confidence. However, when prompted to express confidence, the error rates of current LLMs are inconsistent with their communicated confidences, highlighting the need for uncertainty quantification methods. Many prior methods calculate lexical uncertainty, estimating a model's confidence in the specific string it generated. In some cases, however, it may be more useful to estimate semantic uncertainty, or the model's confidence in the answer regardless of how it is verbalized. We propose a simple procedure, uncertainty distillation, to teach an LLM to verbalize calibrated semantic confidences. Using held-out data to map initial uncertainty estimates to meaningful probabilities, we create examples annotated with verbalized probabilities for supervised fine-tuning. We find that our method yields verbalized confidences that correlate well with observed error rates, even when compared to strong baselines, some of which are more than twenty times slower at inference time. Additionally, we demonstrate that our method can be applied to black-box models that allow API-based fine-tuning, resulting in estimates of uncertainty that are both more effective and more efficient than any of our baselines.
URL: https://openreview.net/forum?id=Wq4zDn3HTl
---
Title: Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale
Abstract: Contemporary studies in mechanistic interpretability have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models, such as induction heads, function vectors, and the Hydra effect. Some of these individual phenomena have been independently tied to different data distributional properties, while some have been loosely associated with model architecture and how Transformers process information. However, a unified understanding of the relationship between data, model architecture, and optimization remains lacking, failing to answer the fundamental question: why do these three phenomena appear universally across different model families and scales, despite their seeming disconnect? In this work, we answer this question by unifying these three phenomena as consequences of hierarchical latent structures in the data generation process, coupled with decorrelated gradients across additive model components and directional concavity in the representation geometry. We validate our theoretical results in a toy model regime and in a large-scale synthetic data regime, comparing them with language models trained on natural language data.
URL: https://openreview.net/forum?id=OwrBUO1YZj
---
Title: Mind's Eye in 4D: 4D Scene Generation conditioned on fMRI Representations
Abstract: Brain-computer interfaces (BCIs) have enabled breakthroughs like translating fMRI signals into images or videos. However, human perception operates in a dynamic 3D world, processing information across both spatial and temporal dimensions. In this work, we introduce Brain-to-4D, a novel fMRI-conditioned 4D scene generation that synthesizes dynamic 3D scenes conditioned on decoded fMRI signals. Since the subjects observe only 2D videos, the target is not to directly reconstruct physically exact 4D geometry from brain activity. Instead, our goal is to generate plausible 4D scenes whose high-level semantics are guided by fMRI representations, while unobserved multi-view geometry and high-frequency dynamics are completed by generative priors. To address the absence of paired fMRI–4D data, we propose Mind4D, an innovative fMRI-conditioned 4D generation framework capable of learning asymmetric hierarchical representations from fMRI signals in a weakly supervised manner. Our approach captures both high-level and low-level representations, along with the decomposition of scene backgrounds and object foregrounds. By conditioning and integrating multiple generative priors for the foreground and background, Mind4D produces prior-guided semantic 4D visuals. Extensive experiments show that Mind4D generates multi-view consistent 4D visuals semantically aligned with brain activity.
URL: https://openreview.net/forum?id=b0m1QyjlgF
---
Title: Tight Bounds for Online Learning with Adversarially Corrupted Experts
Abstract: We study learning from expert advice with corrupted feedback, where the learner observes only *noisy* losses but is evaluated on the underlying *true* losses. Unlike prior work on stochastic perturbations, we consider a fully adversarial model in which perturbations are arbitrary but bounded by a per-expert budget $C$. We establish the first minimax-optimal risk of order $\tilde{\Theta}(C \log K)$ for the corrupted-feedback setting, even when the corruption budget $C$ is *unknown*. Our achievability result builds on a novel potential-based analysis of the Exponentially Weighted Average (EWA) algorithm together with an epoch-based scheduling scheme inspired by the doubling trick. By contrast, we prove that the FTRL-style self-confident EWA (Auer et al., 2002a), and more generally a broad class of FTRL-style adaptive schedules, suffers risk at least $\Omega(C^{\beta})$ for any $1<\beta<\frac{3}{2}$, highlighting a fundamental separation between risk evaluations on noisy and true losses. Finally, we extend our guarantees beyond the realizable case by establishing sublinear regret bounds. Together, these results establish a complete minimax theory for the corrupted-expert model, capturing both the optimal rates and the algorithmic principles.
URL: https://openreview.net/forum?id=m1lOe2TZ48
---
Title: On the Reproducibility and Robustness of Differentially Private Fine-Tuning of Diffusion Models
Abstract: Differentially private (DP) fine-tuning of generative models has emerged as a promising direction for adapting powerful pretrained systems to sensitive domains without compromising individual privacy. In this work, we investigate the robustness of DP-LoRA (Tsai et al. 2024) under backbone substitution and practical compute constraints through an independent empirical reproduction and extension study for high-resolution conditional image synthesis, using CelebA-HQ and a publicly pretrained latent diffusion backbone. We find that qualitative DP-LoRA trends remain stable across backbone substitutions, while absolute utility metrics are substantially affected by the choice of pretrained backbone. Our results confirm the core behavioral trends: image quality improves as the privacy budget relaxes, moderate LoRA ranks and thoughtful adapter placement matter, and parameter-efficient fine-tuning stays below 1% of total model parameters across all configurations. We assess reproducibility in terms of qualitative trends rather than exact numerical replication. Beyond reproduction, we extend the study with additional ablations on DP-SGD hyperparameters, static rank allocation schedules, alternating LoRA schedules and a localized DP feasibility analysis. While our absolute metrics differ from the original, the qualitative behavior of DP-LoRA remains consistent, lending credibility to the original findings. Our code is available at: https://github.com/anonymous2026-5-1/dp-lora-reproducibility
URL: https://openreview.net/forum?id=EDLNBsY7vw
---
Title: Zero Generalization Error Theorem for Random Interpolators via Algebraic Geometry
Abstract: We theoretically demonstrate that the generalization error of random interpolators in teacher-student regression problems achieves zero if the number of training samples exceeds a certain threshold. Understanding why highly over-parameterized models generalize well remains a central problem in machine learning theory. While recent studies have often attributed this phenomenon to the implicit bias of optimization algorithms such as stochastic gradient descent (SGD), empirical evidence indicates that this phenomenon primarily stems from properties of the model itself. Specifically, even randomly sampled interpolators, namely parameters that achieve zero training error, have been observed to generalize effectively. In this paper, we study the generalization behavior of random interpolators in noiseless teacher-student regression problems. We prove that the generalization error of a randomly sampled interpolator becomes exactly zero once the number of training samples exceeds a threshold determined by the geometric structure of the interpolator set in parameter space.
Technically, we leverage tools from algebraic geometry to characterize this geometric structure.
URL: https://openreview.net/forum?id=MGM6R1C7pR
---
Title: Show, Don’t Just Tell: A Comprehensive Survey of Visual Prompting for Multimodal Large Language Models
Abstract: Multimodal large language models (MLLMs) equip pre-trained large language models (LLMs) with visual capabilities. While textual prompting in LLMs has been widely studied, visual prompting has emerged for more fine-grained and free-form visual instructions. This paper presents the first comprehensive survey on visual prompting methods in MLLMs, focusing on visual prompting, prompt generation, compositional reasoning, and prompt learning. We categorize existing visual prompts and discuss generative methods for automatic prompt annotations on the images. We also examine visual prompts that enable better alignment between visual encoders and backbone LLMs, concerning MLLM's visual grounding, referring, and compositional reasoning abilities. This paper examines visual prompting methods developed in MLLMs and provides a vision of the future of these methods.
URL: https://openreview.net/forum?id=sYtytA6dLe
---
Title: On the Wasserstein Gradient Flow Interpretation of Drifting Models
Abstract: Recently, Deng et al. (2026) proposed Generative Modeling via Drifting (GMD), a novel framework for generative tasks. This note presents an analysis of GMD through the lens of Wasserstein Gradient Flows (WGF), i.e., the path of steepest descent for a functional in the space of probability measures, equipped with the geometry of optimal transport. Unlike previous WGF-based contributions, GMD can be thought of as directly targeting a fixed point of a specific WGF flow. We demonstrate three main results: first, that one algorithm proposed by Deng et al. can be viewed as targeting the fixed point of the KL Wasserstein gradient flow after replacing the intractable scores by Parzen-smoothed score estimates. Second, that the algorithm actually implemented by Deng et al. corresponds to a different procedure, which bears some resemblance to the fixed point of a WGF on the Sinkhorn divergence, but lacks certain desirable properties of the latter. Third, the same idea can be extended to the limiting point of other WGFs, including the Maximum Mean Discrepancy (MMD), the sliced Wasserstein distance, and GAN critic functions.
URL: https://openreview.net/forum?id=Gr887O3nLD
---
Title: Towards Resolving Positional Control Failures in Diffusion Transformers
Abstract: Text-to-image (T2I) generation has advanced rapidly, and the historically difficult Position task now appears largely solved in top models. However, our analysis challenges this view, showing that even state-of-the-art (SOTA) models struggle with more complex position-related tasks. We introduce PosEval, a benchmark comprising five new challenging position-based tasks, designed for in-depth evaluation of existing T2I models and to support the development of next-generation layout guidance methods. Leveraging PosEval for evaluation, we propose a new layout-guidance method compatible with modern architectures. Prior work shows that transferring layout guidance from U-Net to transformer models introduces blending artifacts, a challenge addressed in existing methods by averaging sub-predictions to enforce coherence. We introduce Stitch, which instead uses segmentation to extract and compose foreground objects, yielding a latent the base model can directly transform into a coherent image. Tested on Qwen-Image, FLUX, and SD3.5, Stitch consistently enhances base models, even improving FLUX by 218% on GenEval's Position task and by 206% on PosEval. Stitch achieves SOTA results with Qwen-Image on PosEval, improving over previous models by 54%, all while integrating position control into leading models training-free.
URL: https://openreview.net/forum?id=nVdbpPdZtx
---