Stanford MLSys Seminar Episode 59: Zhuohan Li [Today, 1.35-2.30pm PT]

[Karan Goel]

Hi everyone,

We're back in the new quarter with the fifty-ninth episode of the MLSys Seminar today from 1.35-2.30pm PT. 

We'll be joined by Zhuohan Li, who will talk about automating model parallel training. The format is a 30 minute talk followed by a 30 minute podcast-style discussion, where the live audience can ask questions.

Livestream Link: https://www.youtube.com/watch?v=y1NXHjcl6V0

Guests: Zhuohan Li

Title: Alpa: Automated Model-Parallel Deep Learning

Abstract: Alpa (https://github.com/alpa-projects/alpa) automates model-parallel training of large deep learning models by generating execution plans that unify data, operator, and pipeline parallelism. Alpa distributes the training of large deep learning models by viewing parallelisms as two hierarchical levels: inter-operator and intra-operator parallelisms. Based on it, Alpa constructs a new hierarchical space for massive model-parallel execution plans. Alpa designs a number of compilation passes to automatically derive the optimal parallel execution plan in each independent parallelism level and implements an efficient runtime to orchestrate the two-level parallel execution on distributed compute devices. Alpa generates parallelization plans that match or outperform hand-tuned model-parallel training systems even on models they are designed for. Unlike specialized systems, Alpa also generalizes to models with heterogeneous architectures and models without manually-designed plans.
Bio: Zhuohan Li is a PhD student in Computer Science at UC Berkeley, advised by Prof. Ion Stoica. His interest lies in the intersection of machine learning and distributed systems in general. His recent research focuses on distributed model parallel training and inference. He completed his BS at Peking University and has interned at Microsoft Research, Anyscale, and Google Brain.

See you all there!

Best,

Karan