[ContinualAI Seminars]: "Anytime Progressive Pruning"
3 views
Skip to first unread message
Keiland Cooper
May 23, 2022, 11:03:17 PM5/23/22
to Continual Learning & AI News
Hi All,
This Thursday 26-05-2022, 15:30 AM UTC, for the ContinualAI Seminar, Diganta Misra (MILA and Morgan Stanley Research) will present the paper:
Title: “Anytime Progressive Pruning”
Paper link: https://arxiv.org/abs/2204.01640
Abstract: With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings. Although many methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time, sparse networks training in such settings have often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as Anytime Progressive Pruning (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of ≈7% and a reduction in the generalization gap by ≈22%, while being ≈1/3rd the size of the dense baseline model in few-shot restricted imagenet training. We further observe interesting nonmonotonic transitions in the generalization gap in the high number of megabatches-based ALMA.
---------------
- YouTube link:
https://www.youtube.com/watch?v=EZS2PzfyfXY
- Microsoft Teams:
https://bit.ly/clai-seminars
- YouTube recordings of the previous sessions:
https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
This Thursday 26-05-2022, 15:30 AM UTC, for the ContinualAI Seminar, Diganta Misra (MILA and Morgan Stanley Research) will present the paper:
Title: “Anytime Progressive Pruning”
Paper link: https://arxiv.org/abs/2204.01640
Abstract: With the latest advances in deep learning, there has been a lot of focus on the online learning paradigm due to its relevance in practical settings. Although many methods have been investigated for optimal learning settings in scenarios where the data stream is continuous over time, sparse networks training in such settings have often been overlooked. In this paper, we explore the problem of training a neural network with a target sparsity in a particular case of online learning: the anytime learning at macroscale paradigm (ALMA). We propose a novel way of progressive pruning, referred to as Anytime Progressive Pruning (APP); the proposed approach significantly outperforms the baseline dense and Anytime OSP models across multiple architectures and datasets under short, moderate, and long-sequence training. Our method, for example, shows an improvement in accuracy of ≈7% and a reduction in the generalization gap by ≈22%, while being ≈1/3rd the size of the dense baseline model in few-shot restricted imagenet training. We further observe interesting nonmonotonic transitions in the generalization gap in the high number of megabatches-based ALMA.
---------------
- YouTube link:
https://www.youtube.com/watch?v=EZS2PzfyfXY
- Microsoft Teams:
https://bit.ly/clai-seminars
- YouTube recordings of the previous sessions:
https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
Keiland Cooper
May 23, 2022, 11:04:17 PM5/23/22
to Continual Learning & AI News
Hi All,
This Thursday 26-05-2022, 15:30 PM UTC, for the ContinualAI Seminar, Diganta Misra (MILA and Morgan Stanley Research) will present the paper:
This Thursday 26-05-2022, 15:30 PM UTC, for the ContinualAI Seminar, Diganta Misra (MILA and Morgan Stanley Research) will present the paper:
Reply all
Reply to author
Forward
0 new messages