[ContinualAI Seminars]: "Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline"

[Keiland Cooper]

Hi All,

Join us this Thursday 02-06-2022, 15:30 PM UTC, for the ContinualAI Seminar, where Massimo Caccia (MILA and AWS) will present the paper:

Title: “Task-Agnostic Continual Reinforcement Learning: In Praise of a Simple Baseline”
Link: https://arxiv.org/abs/2205.14495

Abstract: We study task-agnostic continual reinforcement learning (TACRL) in which standard RL challenges are compounded with partial observability stemming from task agnosticism, as well as additional difficulties of continual learning (CL), i.e., learning on a non-stationary sequence of tasks. Here we compare TACRL methods with their soft upper bounds prescribed by previous literature: multi-task learning (MTL) methods which do not have to deal with non-stationary data distributions, as well as task-aware methods, which are allowed to operate under full observability. We consider a previously unexplored and straightforward baseline for TACRL, replay-based recurrent RL (3RL), in which we augment an RL algorithm with  recurrent mechanisms to address partial observability and experience replay mechanisms to address catastrophic forgetting in CL. Studying empirical performance in a sequence of RL tasks, we find surprising occurrences of 3RL matching and overcoming the MTL and task-aware soft upper bounds. We lay out hypotheses that could explain this inflection point of continual and task-agnostic learning research. Our hypotheses are empirically tested in continuous control tasks via a large-scale study of the popular multi-task and continual learning benchmark Meta-World. By analyzing different training statistics including gradient conflict, we find evidence that 3RL's outperformance stems from its ability to quickly infer how new tasks relate with the previous ones, enabling forward transfer.


---------------
- YouTube link:
https://youtu.be/ldJW6tMO-zw

- Microsoft Teams:
https://bit.ly/clai-seminars

- YouTube recordings of the previous sessions:
https://www.youtube.com/c/ContinualAI
---------------

Feel free to share this email to anyone interested and invite them to subscribe this mailing-list here: https://groups.google.com/g/continualai

Please also contact me if you want to speak at one of the next sessions!

Looking forward to seeing you all there!

All the best,
Keiland Cooper

University of California
ContinualAI Co-founder
http://kwcooper.xyz