MLSys Seminars Episodes 70 + 71: Stella Biderman, Ludwig Schmidt [Mon, Wd, 3:30-4:20pm PT]
75 views
Skip to first unread message
Dan Fu
Jan 27, 2023, 8:58:56 PM1/27/23
to stanford-ml...@googlegroups.com, cs-se...@lists.stanford.edu, ai-...@cs.stanford.edu, stanf...@googlegroups.com, dawn-i...@lists.stanford.edu
Hi everyone,
Bio: Stella Biderman is the head of research at EleutherAI, an online research lab that has revolutionized open access to large language models. She is best known for her work on democratizing LLMs, especially the GPT-Neo-2.7B, GPT-NeoX-20B, and BLOOM-176B models, all of which where the largest publicly available GPT-3-style LLMs in the world at time of release. Her work on publicly available datasets and evaluation frameworks has become an integral part of training foundation models in NLP. Her interest in open sourcing NLP models is primarily driven by her passion for interpretability research, a topic she has increasingly focused on as access to LLMs has increased. She proudly does not possess a PhD.
First, we survey the current robustness landscape based on a large-scale experimental study involving more than 200 different models and test conditions. The CLIP models stand out with unprecedented robustness gains on multiple challenging distribution shifts. To further improve CLIP, we then introduce new methods for reliably fine-tuning models by interpolating the weights of multiple models. Finally, we investigate the cause of CLIP’s robustness via controlled experiments to disentangle the influence of language supervision and training distribution. While CLIP leveraged large scale language supervision for the first time, its robustness actually comes from the pre-training dataset.
Based on our findings, we will conclude with initial experiments to improve the pre-training datasets for image-text models.
Next week we'll have two episodes of the MLSys Seminar -- Monday 3:30-4:20pm PT, and Wednesday 3:30-4:20pm PT.
Monday will be Stella Biderman, who will be talking about mechanistic interpretability in Transformers, and Wednesday will be Ludwig Schmidt, who will be talking about...
Livestream links:
Talk details below!
Stella Biderman
Title: Mechanistic Interpretability – Reverse Engineering Learned Algorithms from Transformers
Abstract: Transformers are exceptionally powerful technologies that have quickly gone from smashing NLP benchmarks to being one of, if not the premier ML technology in a wide array of fields. Given their growing role in technological pipelines and society writ large, understanding how and why they work is a pressing issue. In this talk I give an overview of research on Mechanistic Interpretability, a field of work that has had substantial success picking apart transformers and understanding the algorithms that trained models use to reason. Topics covered include: the algorithm that toy LLMs can use to perform arithmetic accurately; how real-world LLMs do object identification; and how AlphaFold learns 2D projections of structures and then inflates them over time. Time permitting, I hope to discuss recent discoveries at EleutherAI currently under review for publication.Bio: Stella Biderman is the head of research at EleutherAI, an online research lab that has revolutionized open access to large language models. She is best known for her work on democratizing LLMs, especially the GPT-Neo-2.7B, GPT-NeoX-20B, and BLOOM-176B models, all of which where the largest publicly available GPT-3-style LLMs in the world at time of release. Her work on publicly available datasets and evaluation frameworks has become an integral part of training foundation models in NLP. Her interest in open sourcing NLP models is primarily driven by her passion for interpretability research, a topic she has increasingly focused on as access to LLMs has increased. She proudly does not possess a PhD.
Ludwig Schmidt
Title: A data-centric view on reliable generalization
Abstract: Researchers have proposed many methods to make neural networks more reliable under distribution shift, yet there is still large room for improvement. Are better training algorithms or training data the more promising way forward? In this talk, we study this question in the context of computer vision and OpenAI’s CLIP model for learning from image-text data.
First, we survey the current robustness landscape based on a large-scale experimental study involving more than 200 different models and test conditions. The CLIP models stand out with unprecedented robustness gains on multiple challenging distribution shifts. To further improve CLIP, we then introduce new methods for reliably fine-tuning models by interpolating the weights of multiple models. Finally, we investigate the cause of CLIP’s robustness via controlled experiments to disentangle the influence of language supervision and training distribution. While CLIP leveraged large scale language supervision for the first time, its robustness actually comes from the pre-training dataset.
Based on our findings, we will conclude with initial experiments to improve the pre-training datasets for image-text models.
Bio: Ludwig Schmidt is an assistant professor in the Paul G. Allen School of Computer Science & Engineering at the University of Washington. Ludwig’s research interests revolve around the empirical foundations of machine learning, often with a focus on datasets, evaluation, reliable methods, and large models. Ludwig completed his PhD at MIT under the supervision of Piotr Indyk and was a postdoc at UC Berkeley hosted by Benjamin Recht and Moritz Hardt. Recently, Ludwig's research group contributed to multimodal language & vision models by creating OpenCLIP and the LAION-5B dataset. Ludwig’s research received a new horizons award at EAAMO, best paper awards at ICML & NeurIPS, a best paper finalist at CVPR, and the Sprowls dissertation award from MIT.
See you all there!
Best, Dan
Dan Fu
Jan 30, 2023, 6:20:50 PM1/30/23
to stanford-ml...@googlegroups.com, cs-se...@lists.stanford.edu, ai-...@cs.stanford.edu, stanf...@googlegroups.com, dawn-i...@lists.stanford.edu
We're live with Stella in 10!
Dan
Dan Fu
Feb 1, 2023, 6:21:59 PM2/1/23
to stanford-ml...@googlegroups.com, cs-se...@lists.stanford.edu, ai-...@cs.stanford.edu, stanf...@googlegroups.com, dawn-i...@lists.stanford.edu
We're live with Ludwig in 10!
Dan
On Fri, Jan 27, 2023 at 5:58 PM Dan Fu <da...@cs.stanford.edu> wrote:
Reply all
Reply to author
Forward
0 new messages