Daily TMLR digest for Mar 19, 2025

[TMLR]

Accepted papers
===============


Title: Which Backbone to Use: A Resource-efficient Domain Specific Comparison for Computer Vision

Authors: Pranav Jeevan P, Amit Sethi

Abstract: For computer vision applications on small, niche, and proprietary datasets, fine-tuning a neural network (NN) backbone that is pre-trained on a large dataset, such as the ImageNet, is a common practice. However, it is unknown whether the backbones that perform well on large datasets, such as vision transformers, are also the right choice for fine-tuning on smaller custom datasets. The present comprehensive analysis aims to aid machine learning practitioners in selecting the most suitable backbone for their specific problem. We systematically evaluated multiple lightweight, pre-trained backbones under consistent training settings across a variety of domains spanning natural, medical, deep space, and remote sensing images. We found that even though attention-based architectures are gaining popularity, they tend to perform poorly compared to CNNs when fine-tuned on small amounts of domain-specific data. We also observed that certain CNN architectures consistently perform better than others when controlled for network size. Our findings provide actionable insights into the performance trade-offs and effectiveness of different backbones for a broad spectrum of computer vision domains.

URL: https://openreview.net/forum?id=XVSQnnf7QT

---


New submissions
===============


Title: Diverse Condensed Data Generation via Class Preserving Distribution Matching

Abstract: Large-scale datasets for training many real-world machine learning models pose significant computational resource challenges. One approach to mitigate this is via data condensation, which aims at learning a small dataset but still sufficiently capturing the rich information in the original one. Most of existing approaches learn the condensed dataset and task-related model parameters (e.g., classifier) in a bi-level meta-learning way. The recently proposed distribution matching (DM), however, avoids the expensive bi-level optimization but ignores task-related models. This work proposes a novel class preserving DM framework consisting of two key components. The first one is responsible for capturing the original data distribution of each class based on energy distance, which can encourage the diversity in the generated synthetic data. The other is classifier-critic constraint, which forces the learned synthetic samples to fit pre-trained task-related models, such as an off-the-shelf classifier. Designing the optimization loss in this way, we can generate more diverse and class preserving distilled data without the bi-level optimization. Extensive experiments reveal that our method can produce more effective condensed data for downstream tasks with less training cost and can also be successfully applied to de-biased dataset condensation.

URL: https://openreview.net/forum?id=QOrzmDQYou

---

Title: Out-of-Distribution Detection with Overlap Index

Abstract: Out-of-distribution (OOD) detection is crucial for the deployment of machine learning models in the open world. While existing OOD detectors are effective in identifying OOD samples that deviate significantly from in-distribution (ID) data, they often come with trade-offs. For instance, deep OOD detectors usually suffer from high computational costs, require tuning hyperparameters, and have limited interpretability, whereas traditional OOD detectors may have a low accuracy on large high-dimensional datasets. To address these limitations, we propose a novel effective OOD detection approach that employs an overlap index (OI)-based confidence score function to evaluate the likelihood of a given input belonging to the same distribution as the available ID samples. The proposed OI-based confidence score function is non-parametric, lightweight, and easy to interpret, hence providing strong flexibility and generality. Extensive empirical evaluations indicate that our OI-based OOD detector is competitive with state-of-the-art OOD detectors in terms of detection accuracy on a wide range of datasets while requiring less computation and memory costs. Lastly, we show that the proposed OI-based confidence score function inherits nice properties from OI (e.g., insensitivity to small distributional variations and robustness against Huber $\epsilon$-contamination) and is a versatile tool for estimating OI and model accuracy in specific contexts.

URL: https://openreview.net/forum?id=bDHJhFEgcA

---

Title: Modeling Human Beliefs about AI Behavior for Scalable Oversight

Abstract: Contemporary work in AI alignment often relies on human feedback to teach AI systems human preferences and values. Yet as AI systems grow more capable, human feedback becomes increasingly unreliable. This raises the problem of scalable oversight: How can we supervise AI systems that exceed human capabilities? In this work, we propose to model the human evaluator's beliefs about the AI system's behavior to better interpret the human's feedback. We formalize human belief models and theoretically analyze their role in inferring human values. We then characterize the remaining ambiguity in this inference and conditions for which the ambiguity disappears. To mitigate reliance on exact belief models, we then introduce the relaxation of human belief model covering. Finally, we propose using foundation models to construct covering belief models, providing a new potential approach to scalable oversight.

URL: https://openreview.net/forum?id=gSJfsdQnex

---

Title: Fully Automatic Neural Network Reduction for Formal Verification

Abstract: Formal verification of neural networks is essential before their deployment in safety-critical applications.
However, existing methods for formally verifying neural networks are not yet scalable enough to handle practical problems under strict time constraints.
We address this challenge by introducing a fully automatic and sound reduction of neural networks using reachability analysis.
The soundness ensures that the verification of the reduced network entails the verification of the original network.
Our sound reduction approach is applicable to neural networks with any type of element-wise activation function, such as ReLU, sigmoid, and tanh.
The network reduction is computed on the fly while simultaneously verifying the original network and its specifications.
All parameters are automatically tuned to minimize the network size without compromising verifiability.
We further show the applicability of our approach to convolutional neural networks by explicitly exploiting similar neighboring pixels.
Our evaluation shows that our approach reduces large neural networks to a fraction of the original number of neurons
and thus shortens the verification time to a similar degree.

URL: https://openreview.net/forum?id=gmflcWlVMl

---

Title: Efficient Representations for Whole Slide Image Classification

Abstract: The advent of digital pathology has transformed diagnostic and research capabilities, offering unprecedented insights through the analysis of high-resolution whole slide images (WSIs). However, the gigapixel size and complexity of WSIs present significant computational challenges. To address this, we propose a scalable and efficient pipeline for WSI classification that integrates patch-based feature extraction, clustering, and advanced representation techniques. Our methodology begins by extracting features from patches identified based on their pathological significance using deep feature embeddings from a pre-trained convolutional neural network (CNN) fine tuned on a histology dataset under noisy labels. This approach ensures that the extracted features are robust and tailored to histopathological patterns despite the inherent noise in the training data. These embeddings are then clustered using K-means clustering to group semantically similar regions. To represent these clusters effectively, we experimented with two strategies: first, using the cluster mean to summarize each cluster; and second, employing Fisher vector (FV) encoding to model the distribution of patch embeddings within clusters using a parametric Gaussian mixture model (GMM). The resulting high-dimensional feature vector encapsulates both local and global tissue structures, enabling robust classification of WSIs. This approach significantly reduces computational overhead while maintaining high accuracy, as validated across multiple datasets. Our innovative framework combines the precision of Fisher vectors with the scalability of clustering, establishing an efficient and precise solution for WSI analysis that advances the practical application of digital pathology in medical diagnostics and research.

URL: https://openreview.net/forum?id=vKLH4PDN7V

---