Daily TMLR digest for Jun 30, 2023
TMLR
Accepted papers
===============
Title: Learning Symbolic Rules for Reasoning in Quasi-Natural Language
Authors: Kaiyu Yang, Jia Deng
Abstract: Symbolic reasoning, rule-based symbol manipulation, is a hallmark of human intelligence. However, rule-based systems have had limited success competing with learning-based systems outside formalized domains such as automated theorem proving. We hypothesize that this is due to the manual construction of rules in past attempts. In this work, we take initial steps towards rule-based systems that can reason with natural language but without manually constructing rules. We propose MetaQNL, a "Quasi-Natural Language" that can express both formal logic and natural language sentences, and MetaInduce, a learning algorithm that induces MetaQNL rules from training data consisting of questions and answers, with or without intermediate reasoning steps. In addition, we introduce soft matching—a flexible mechanism for applying rules without rigid matching, overcoming a typical source of brittleness in symbolic reasoning. Our approach achieves state-of-the-art accuracies on multiple reasoning benchmarks; it learns compact models with much less data and produces not only answers but also checkable proofs. Further, experiments on two simple real-world datasets demonstrate the possibility for our method to handle noise and ambiguity.
URL: https://openreview.net/forum?id=gwRwHUZUgz
---
New submissions
===============
Title: Unlocking Unlabeled Data: Ensemble Learning with the Hui-Walter Paradigm for Performance Estimation in Online and Static Settings
Abstract: n the realm of machine learning and statistical modeling, practitioners often work under
the assumption of accessible, static, labeled data for evaluation and training. However,
this assumption often deviates from reality where data may be private, encrypted, difficult-
to-measure, or unlabeled. In this paper, we bridge this gap by adapting the Hui-Walter
paradigm, a method traditionally applied in epidemiology and medicine, to the field of
machine learning. This approach enables us to estimate key performance metrics such as false
positive rate, false negative rate, and prior in scenarios where no ground truth is available.
We further extend this paradigm for handling online data, opening up new possibilities for
dynamic data environments. Our methodology, applied to two diverse datasets, the Wisconsin
Breast Cancer and the Adult dataset, involves partitioning each into latent classes to simulate
multiple data populations and independently training models to replicate multiple tests. By
cross-tabulating binary outcomes across ensemble categorizers and multiple populations, we
are able to estimate unknown parameters through Gibbs sampling, eliminating the need
for ground-truth or labeled data. This paper showcases the potential of our methodology
to transform machine learning practices by allowing for accurate model assessment under
dynamic and uncertain data conditions.
URL: https://openreview.net/forum?id=s8TsIPTgVG
---
Title: Representation Ensembling for Synergistic Lifelong Learning with Quasilinear Complexity
Abstract: In lifelong learning, data are used to improve performance not only on the current task, but also on previously encountered, and as yet unencountered tasks. In contrast, classical machine learning which starts from a blank slate, or tabula rasa, uses data only for the single task at hand. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain performance on old tasks given new tasks. But striving to avoid forgetting sets the goal unnecessarily low. The goal of lifelong learning should be to not only improve performance on future tasks (forward transfer) but also to improve performance on past tasks (backward transfer) with any new data. Our key insight is that we can synergistically ensemble representations that were learned independently on disparate tasks to enable both forward and backward transfer. This generalizes ensembling independently learned representations (like in decision forests) and complements ensembling dependent representations (like in gradient boosted trees). Moreover, we ensemble representations in quasilinear space and time. We demonstrate this insight with two algorithms: representation ensembles of (1) trees and (2) networks. Both algorithms demonstrate forward and backward transfer in a variety of simulated and benchmark data scenarios, including tabular, image, spoken, and adversarial tasks, including CIFAR-100, Five-Dataset, Split Mini-Imagenet, and Food1k, as well as the spoken digit dataset. This is in stark contrast to the reference algorithms we compared to, most of which failed to transfer either forward or backward, or both, despite that many of them require quadratic space or time complexity.
URL: https://openreview.net/forum?id=DPtLNKzkg0
---
Title: No Free Lunch in Self Supervised Representation Learning
Abstract: Self-supervised representation learning in computer vision relies heavily on hand-crafted image transformations to learn meaningful and invariant features. However few extensive explorations of the impact of transformation design have been conducted in the literature. In particular, although the dependence of representation quality to transformation design has been established, it has not been thoroughly studied. In this work, we explore this relationship and its impact on a domain other than natural images. We demonstrate that designing transformations can be viewed as a form of beneficial supervision. Firstly, we not only show that transformations have an effect on the features in representations and the relevance of clustering, but also that each category in a supervised dataset can be impacted differently in a controllable manner. Furthermore, we explore the impact of transformation design on a domain such as microscopy images where differences between classes are more subtle than in natural images. In this case, we observe a more significant impact on the features encoded into the resulting representations. Finally, we demonstrate that transformation design can be leveraged as a form of supervision, as careful selection of these transformation, based on the desired features, can lead to a drastic increase in performance by improving the resulting representation.
URL: https://openreview.net/forum?id=xXgxlrQ6uo
---
Title: Pairwise Learning with Adaptive Online Gradient Descent
Abstract: In this paper, we propose an adaptive online gradient descent method with momentum for pairwise learning, in which the stepsize is determined by historical information. Due to the structure of pairwise learning, the sample pairs are dependent on the parameters, causing difficulties in the convergence analysis. To this end, we develop novel techniques for the convergence analysis of the proposed algorithm. We show that the proposed algorithm can output the desired solution in strongly convex, convex, and nonconvex cases. Furthermore, we present theoretical explanations for why our proposed algorithm can accelerate previous workhorses for online pairwise learning. All assumptions used in the theoretical analysis are mild and common, making our results applicable to various pairwise learning problems. To demonstrate the efficiency of our algorithm, we compare the proposed adaptive method with the non-adaptive counterpart on the benchmark online AUC maximization problem.
URL: https://openreview.net/forum?id=rq1SaHQg2k
---