OxCSML seminar this week: Laurence Aitchison

[Hai Dang Dau]

Dear all,

This week we welcome Laurence Aitchison from University of Bristol to give a talk in our OxCSML seminar. Please find the details below. 

Looking forward to seeing you there,

Saif & Hai-Dang.

==============

Speaker: Laurence Aitchison (University of Bristol)
Time and date: 2pm-3pm, Friday 16 Feb
Place: Small Lecture Theatre (LG.03), Department of Statistics

Zoom: https://zoom.us/j/96780624019?pwd=WktHN1lFN2VnK1hjK2hxcmJiMUhiZz09

Title: Deep kernel processes and machines.

Abstract: The successes of modern deep neural networks (DNNs) are founded on their ability to transform inputs across multiple layers to build good high-level representations. It is therefore critical to understand this process of representation learning. However, standard theoretical approaches involving infinite width limits give very limited insights into representation learning. For instance, the NNGP infinite-width limit entirely eliminates representation learning. Alternatively, mu-P just tells us whether or not representation learning is possible, without telling us anything about the representations that are actually learned. We therefore develop a new infinite width limit, the Bayesian representation learning limit, that exhibits representation learning mirroring that in finite-width networks, yet at the same time, remains extremely tractable. This limit gives rise to an elegant objective that describes how learning shapes representations at every layer. Using this objective, we develop a new, scalable family of "deep kernel methods", which are based on an infinite-width limit of deep Gaussian processes.  In practice, deep kernel methods just use kernels without ever using any features or weights. We develop a convolutional variant, known as Convolutional Deep Kernel Machines, and push their performance to 94.1% on CIFAR-10 (the previous SOTA for kernel methods was 91.2%, from Adlam et al. 2023)