[ContinualAI Seminars]: "SS-IL: Separated Softmax for Incremental Learning"
5 views
Skip to first unread message
Keiland Cooper
Mar 8, 2022, 5:26:10 AM3/8/22
to Continual Learning & AI News
Hi All,
This Thursday 10-03-2022, at the temporary new time of 14.00 PM CET, for the ContinualAI Seminar, Hongjoon Ahn (Sungkyunkwan University) will present the paper:
Title: “SS-IL: Separated Softmax for Incremental Learning”
Abstract: We consider class incremental learning (CIL) problem, in which a learning agent continuously learns new classes from incrementally arriving training data batches and aims to predict well on all the classes learned so far. The main challenge of the problem is the catastrophic forgetting, and for the exemplar-memory based CIL methods, it is generally known that the forgetting is commonly caused by the classification score bias that is injected due to the data im- balance between the new classes and the old classes (in the exemplar-memory). While several methods have been pro- posed to correct such score bias by some additional post- processing, e.g., score re-scaling or balanced fine-tuning, no systematic analysis on the root cause of such bias has been done. To that end, we analyze that computing the softmax probabilities by combining the output scores for all old and new classes could be the main cause of the bias. Then, we propose a new method, dubbed as Separated Softmax for Incremental Learning (SS-IL), that consists of separated softmax (SS) output layer combined with task-wise knowledge distillation (TKD) to resolve such bias. Throughout our extensive experimental results on several large-scale CIL benchmark datasets, we show our SS-IL achieves strong state-of-the-art accuracy through attaining much more balanced prediction scores across old and new classes, without any additional post-processing.
---------------
- YouTube link: [click here]
- Microsoft Teams: [ click here]
- YouTube recordings of the previous sessions: https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
This Thursday 10-03-2022, at the temporary new time of 14.00 PM CET, for the ContinualAI Seminar, Hongjoon Ahn (Sungkyunkwan University) will present the paper:
Title: “SS-IL: Separated Softmax for Incremental Learning”
Abstract: We consider class incremental learning (CIL) problem, in which a learning agent continuously learns new classes from incrementally arriving training data batches and aims to predict well on all the classes learned so far. The main challenge of the problem is the catastrophic forgetting, and for the exemplar-memory based CIL methods, it is generally known that the forgetting is commonly caused by the classification score bias that is injected due to the data im- balance between the new classes and the old classes (in the exemplar-memory). While several methods have been pro- posed to correct such score bias by some additional post- processing, e.g., score re-scaling or balanced fine-tuning, no systematic analysis on the root cause of such bias has been done. To that end, we analyze that computing the softmax probabilities by combining the output scores for all old and new classes could be the main cause of the bias. Then, we propose a new method, dubbed as Separated Softmax for Incremental Learning (SS-IL), that consists of separated softmax (SS) output layer combined with task-wise knowledge distillation (TKD) to resolve such bias. Throughout our extensive experimental results on several large-scale CIL benchmark datasets, we show our SS-IL achieves strong state-of-the-art accuracy through attaining much more balanced prediction scores across old and new classes, without any additional post-processing.
---------------
- YouTube link: [click here]
- Microsoft Teams: [ click here]
- YouTube recordings of the previous sessions: https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
Reply all
Reply to author
Forward
0 new messages