OxCSML seminar Friday 16 June 2023: Sanjeev Arora, Jakiw Pidstrigach and Brian Trippe
19 views
Skip to first unread message
Hai Dang Dau
Jun 12, 2023, 3:33:01 AM6/12/23
to oxcsml...@googlegroups.com, oxc...@googlegroups.com
Dear all,
This Friday we welcome three speakers to our OxCSML seminar: Sanjeev Arora, Jakiw Pidstrigach and Brian Trippe. Please find details of the talks below. Looking forward to seeing you there.
Best,
Hai-Dang.
============
Date and place: Friday 16 June 2023. Department of Statistics, University of Oxford. Large Lecture Theatre (LG.01).
Time: 13:30 to 18:00. Timeline as follows.
13:30 - 14:30 Jakiw Pidstrigach (University of Potsdam)
Title: Infinite-Dimensional Diffusion Models for Function Spaces
Abstract: We define diffusion-based generative models in infinite dimensions, and apply them to the generative modeling of functions and Bayesian inverse problems. By first formulating such models in the infinite-dimensional limit and only then discretizing, we are able to obtain a sampling algorithm with dimension-free bounds on the distance from the sample measure to the target measure.
15:00 - 16:00 Brian Trippe (Columbia University)
Title: Twisted diffusion sampling for accurate conditional generation with application to protein design
Abstract: Diffusion generative models have enabled recent progress in molecular design and text-to-image generation. However, these achievements have primarily depended on expensive, task-specific conditional training or error-prone heuristic approximations. Ideally, a conditional generation method should provide exact samples for a broad range of conditional distributions without requiring task-specific training. To this end, we introduce the Twisted Diffusion Sampler (TDS). TDS is a sequential Monte Carlo (SMC) algorithm that targets the conditional distributions of diffusion models. The main idea is to use twisting, an SMC technique that improves computational efficiency, to incorporate heuristic approximations without compromising asymptotic exactness. When applied to the motif-scaffolding problem, a core problem in protein design, TDS enables more flexible conditioning criteria than conditionally trained models, and outperforms the previous state of the art on 9/12 problems in a benchmark set of short proteins.
16:30 - 17:30 Sanjeev Arora (Google Deepmind and Princeton University)
Title: A Theory for Emergence of Complex Skills in Large Language Models
Abstract: A driver of current AI research is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems forbiddingly difficult. We give an analyses using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this slingshot generalization since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving k-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves. This competence at a k-tuple of skills could emerge despite not having seen the particular combination at training time. (Joint work with Anirudh Goyal of Google Deepmind)
Zoom registration link:
https://www.eventbrite.co.uk/e/oxcsml-seminar-friday-16-june-2023-tickets-654783002587
Time: 13:30 to 18:00. Timeline as follows.
13:30 - 14:30 Jakiw Pidstrigach (University of Potsdam)
Title: Infinite-Dimensional Diffusion Models for Function Spaces
Abstract: We define diffusion-based generative models in infinite dimensions, and apply them to the generative modeling of functions and Bayesian inverse problems. By first formulating such models in the infinite-dimensional limit and only then discretizing, we are able to obtain a sampling algorithm with dimension-free bounds on the distance from the sample measure to the target measure.
15:00 - 16:00 Brian Trippe (Columbia University)
Title: Twisted diffusion sampling for accurate conditional generation with application to protein design
Abstract: Diffusion generative models have enabled recent progress in molecular design and text-to-image generation. However, these achievements have primarily depended on expensive, task-specific conditional training or error-prone heuristic approximations. Ideally, a conditional generation method should provide exact samples for a broad range of conditional distributions without requiring task-specific training. To this end, we introduce the Twisted Diffusion Sampler (TDS). TDS is a sequential Monte Carlo (SMC) algorithm that targets the conditional distributions of diffusion models. The main idea is to use twisting, an SMC technique that improves computational efficiency, to incorporate heuristic approximations without compromising asymptotic exactness. When applied to the motif-scaffolding problem, a core problem in protein design, TDS enables more flexible conditioning criteria than conditionally trained models, and outperforms the previous state of the art on 9/12 problems in a benchmark set of short proteins.
16:30 - 17:30 Sanjeev Arora (Google Deepmind and Princeton University)
Title: A Theory for Emergence of Complex Skills in Large Language Models
Abstract: A driver of current AI research is the fact that new skills emerge in language models when their parameter set and training corpora are scaled up. This phenomenon is poorly understood, and a mechanistic explanation via mathematical analysis of gradient-based training seems forbiddingly difficult. We give an analyses using the famous (and empirical) Scaling Laws of LLMs and a simple statistical framework. Contributions include: (a) A statistical framework that relates cross-entropy loss of LLMs to competence on the basic skills that underlie language tasks. (b) Mathematical analysis showing that the Scaling Laws imply a strong form of inductive bias that allows the pre-trained model to learn very efficiently. We informally call this slingshot generalization since naively viewed it appears to give competence levels at skills that violate usual generalization theory. (c) A key example of slingshot generalization, that competence at executing tasks involving k-tuples of skills emerges essentially at the same scaling and same rate as competence on the elementary skills themselves. This competence at a k-tuple of skills could emerge despite not having seen the particular combination at training time. (Joint work with Anirudh Goyal of Google Deepmind)
Zoom registration link:
https://www.eventbrite.co.uk/e/oxcsml-seminar-friday-16-june-2023-tickets-654783002587
Reply all
Reply to author
Forward
0 new messages