[ContinualAI Seminars]: "DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion"
10 views
Skip to first unread message
Keiland Cooper
Apr 6, 2022, 2:08:48 AM4/6/22
to Continual Learning & AI News
Hi All,
This Thursday 07-04-2022, 17.30 PM CEST, for the ContinualAI Seminar, Arthur Douillard (Sorbonne Universite, Heuritech) will present the paper:
Title: “DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion”
Abstract: Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn’t need any hyperparameter tuning to control the network’s expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.
---------------
- YouTube link: [click here]
- Microsoft Teams: [ click here]
- YouTube recordings of the previous sessions: https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
This Thursday 07-04-2022, 17.30 PM CEST, for the ContinualAI Seminar, Arthur Douillard (Sorbonne Universite, Heuritech) will present the paper:
Title: “DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion”
Abstract: Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an expansion of the parameters can reduce catastrophic forgetting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having negligible memory and time overheads due to strict control of the parameters expansion. Moreover, this efficient strategy doesn’t need any hyperparameter tuning to control the network’s expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet1000 while having less parameters than concurrent dynamic frameworks.
---------------
- YouTube link: [click here]
- Microsoft Teams: [ click here]
- YouTube recordings of the previous sessions: https://www.youtube.com/c/ContinualAI
---------------
Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai
Please also contact me if you want to speak at one of the next sessions!
Looking forward to seeing you all there!
All the best,
Keiland Cooper
University of California
ContinualAI Co-founder
http://kwcooper.xyz
Reply all
Reply to author
Forward
0 new messages