OxCSML seminar this week: Nicolas Flammarion
28 views
Skip to first unread message
Hai Dang Dau
Nov 6, 2023, 6:40:04 AM11/6/23
to oxcsml...@googlegroups.com, oxc...@googlegroups.com
Dear all,
This week, we welcome Nicolas Flammarion from EPFL to give a talk in our OxCSML seminar. Please find the details below.
Looking forward to seeing you there,
Saif & Hai-Dang.
============
Speaker: Nicolas Flammarion, Assistant Professor in Computer Science at EPFL
Time and date: 14.00 - 15.00 Friday 11 Nov
Place: Room LG.03 (Small Lecture Theatre), Department of Statistics
Title: Saddle-to-saddle dynamics in diagonal linear networks
Abstract: When training neural networks with gradient methods and small weight initialisation, peculiar learning curves are observed: the training initially shows minimal progress, which is then followed by a sudden transition where a new "feature" is rapidly learned. This pattern is commonly known as incremental learning. In this talk, I will demonstrate that we can comprehensively understand this phenomenon within the context of a simplified network architecture. In this setting, we can establish that the gradient flow trajectory transitions from one saddle point of the training loss to another. The specific saddle points visited, as well as the timing of these transitions, can be determined using a recursive algorithm that is reminiscent of the LARS method used in computing the Lasso path.
Zoom: https://zoom.us/j/92081686303?pwd=b3Nqa3RydzNkVEZwYjNJU3owWEt4Zz09
Time and date: 14.00 - 15.00 Friday 11 Nov
Place: Room LG.03 (Small Lecture Theatre), Department of Statistics
Title: Saddle-to-saddle dynamics in diagonal linear networks
Abstract: When training neural networks with gradient methods and small weight initialisation, peculiar learning curves are observed: the training initially shows minimal progress, which is then followed by a sudden transition where a new "feature" is rapidly learned. This pattern is commonly known as incremental learning. In this talk, I will demonstrate that we can comprehensively understand this phenomenon within the context of a simplified network architecture. In this setting, we can establish that the gradient flow trajectory transitions from one saddle point of the training loss to another. The specific saddle points visited, as well as the timing of these transitions, can be determined using a recursive algorithm that is reminiscent of the LARS method used in computing the Lasso path.
Zoom: https://zoom.us/j/92081686303?pwd=b3Nqa3RydzNkVEZwYjNJU3owWEt4Zz09
Reply all
Reply to author
Forward
0 new messages