|WING-NUS NLP Seminar 2021 - Talk 6|
|Title:||Implicit Representations of Meaning in Neural Language Models|
EECS and CSAIL at Massachusetts Institute of Technology (MIT)
|Date/Time:||01 July 2021, Thursday, 10:00 AM to 11:00 AM|
|Venue:||Join Zoom Meeting
ZOOM Room ID: 770 447 8736, PIN: 3244
|Chaired by:||A/P Min-Yen Kan, School of Computing
Neural language models, which place probability distributions over sequences of words, produce vector representations of words and sentences that are useful for language processing tasks as diverse as machine translation, question answering, and image captioning. These models’ usefulness is partially explained by the fact that their representations robustly encode lexical and syntactic information. But the extent to which language model training also induces representations of meaning remains a topic of ongoing debate. I will describe recent work showing that language models—trained on text alone, without any kind of grounded supervision—build structured meaning representations that are used to simulate entities and situations as they evolve over the course of a discourse. These representations can be linearly decoded into logical representations of world state (e.g. discourse representation structures). They can also be directly manipulated to produce predictable changes in generated output. Together, these results suggest that (some) highly structured aspects of meaning can be recovered by relatively unstructured models trained on corpus data.
Jacob Andreas is the X Consortium Assistant Professor at MIT. His research focuses on building intelligent systems that can communicate effectively using language and learn from human guidance. Jacob earned his Ph.D. from UC Berkeley, his M.Phil. from Cambridge (where he studied as a Churchill scholar) and his B.S. from Columbia. He has been the recipient of a Sony Faculty Innovation Award, an MIT Kolokotrones teaching award, and paper awards at NAACL and ICML.
Please contact me if you have any questions. Contact info:
Thanks very much. Looking forward to your participation.