news about word embeddings and Google

[Ted Pedersen]

Google just announced a change in how they are doing web search that includes the use of a model called BERT - this is a more advanced embedding method than word2vec, but the fundamentals are certainly very related. 

We won't talk about BERT this semester in CS [45]242 (other than mentioning it briefly like this) but it will be included in the Advanced NLP class (CS 5642) next semester. 

https://www.blog.google/products/search/search-language-understanding-bert  

I'll talk a little bit more about CS 5642 as we get closer to registration time. In general it will be a continuation of CS [45]242 but with a focus more so on Deep Learning. We are laying the foundation for that this semester with some attention paid to traditional machine learning (Naive Bayes, Decision Lists) and soon to Neural Networks. 

The class will however be somewhat different - we will not be writing programs "from scratch" but instead relying on PyTorch, a new and very good tool for Deep Learning. https://pytorch.org/ While we will continue to use the Jurafsky and Martin book where we can, we will also rely on blog posts and other readings to fill in some background. 

The overall goal of the course will be to come away with a good understanding of sequence to sequence models (especially as applied to Machine Translation) and also contextual embeddings (like we see described with BERT above). The combination of seq2seq models and contextual embeddings underlies a lot of modern NLP, and folks like Google, Facebook, Microsoft etc. are intensely interested (and it's certainly all the rage in graduate programs in NLP, Machine Learning, and AI). 

This will be the first time I have taught this course so it will be quite a bit less formal than this semester, and will evolve in ways I can't really anticipate until it starts. I'm also expecting a fairly small number of students, which tends to make the course less formal and more adaptable. 

In any case, more about this in the coming weeks, and please let me know if you have any questions. 

---

Ted Pedersen
http://www.d.umn.edu/~tpederse