What's new in TensorFlow.js: 🆕🤖 Give your next ML project superpowers at scale - July 2020

[Chloe Chao]

TensorFlow.js Newsletter - H1 2020

TensorFlow.js enables developers to deploy ML solutions to the web - bringing GPU-accelerated ML to the browser, JS-based mobile apps, and Node.js applications. TF.js includes a variety of pre-trained models for image classification, pose detection, body part segmentation, text classification etc., as well as a Keras-like API to create, train and deploy ML models in JavaScript.  Check out the site and keep reading to see some great use cases and updates from the team and community below.

1.9M	13K	120	165	7000
npm downloads	Github stars	Contributors	Dependent npm packages	Dependent Github repos

Here are key highlights from this year so far: TensorFlow.js 2.0 launched, enabling smaller package size and modular builds. Facemesh and Handpose models for web are now available.  A new Web Assembly backend now supports faster CPU execution (SIMD kernel support coming soon)  Users like Modiface, HuggingFace, and IBM, several Google internal teams, as well as many community developers are building amazing applications showing the power of ML on the web.  Read on to learn more!

Product News

Cool #MadeWithTFJS Applications

Coming Soon ...

Events / Other Updates

Project Links

Product News

Smaller package size and modular builds with TensorFlow.js 2.0 TensorFlow.js 2.0 moves CPU and WebGL backends out of the core library and into their own packages. This enables users to optionally drop any backend they don’t want to use from their bundles and enables further work on making the library more modular. See the release notes for details.

	New Models Released TF.js users can now easily utilize face landmark detection and handpose estimation in their web apps with Facemesh and Handpose models (released in partnership with the Mediapipe team). These models enable  use cases for AR-based product tryon, gesture-based control for accessibility, and many more.
	New Web Assembly Backend A new WebAssembly backend for both the browser and for Node.js is now available. This backend is an alternative to the WebGL backend, bringing fast CPU execution with minimal code changes. For smaller models such as blazeface, the WASM backend is up to 1.7x faster than the WebGL backend. This backend helps improve performance on a broader set of devices, especially lower-end mobile devices that lack WebGL support or have a slow GPU. It uses the XNNPACK library for further acceleration.
	Hugging Face achieves 2x performance boost See how Hugging Face achieved a 2x performance boost with tfjs-node based server-side inference service compared to Python, for Question Answering with DistilBERT in this fresh guest blog post.
	Performance Benchmarking Tooling Users can now benchmark their own models by simply providing a model link in the new benchmark tool (https://tensorflow.github.io/tfjs/e2e/benchmarks/). The tool can automatically generate random input data and show various benchmark metrics and charts.
	New Codelab launched: Make your very own smart camera! Users can make their very own version of a Nest smart cam using TensorFlow.js in the browser. Now you can see if your doggo is being a good boy or sneaking into the kitchen for treats whilst you're out.
	TF.js models now on TensorFlow Hub  TF.js models are now integrated with TensorFlow Hub, and all models are available for easy, one-line use in your web applications. Developers can even try-out many models directly in the browser, powered by TensorFlow.js - see SPICE model demo for audio pitch recognition.

Cool #MadeWithTFJS Applications

Here are just a few highlights since our last newsletter for some of our favourite creations that were #MadeWithTFJS - search the hashtag on Twitter / LinkedIn for even more submissions if you are feeling curious, or use the tag yourself to be featured in future newsletters.

	Real time person removal (4,400 Github stars) Privacy is important to us all, and with this creation by Jason Mayes, we can now remove ourselves from the webcam feed in real time using our BodyPix model. Notice how the bed still deforms as Jason walks on it. As JavaScript has such rich support for multimedia and rendering this prototype was created in under 1 day. Check out the code and try it yourself!
	Pose Animator (6,900 Github stars) What if you could become any 2D character of your choosing? Well now you can! This amazing creation by Shan Huang allows you to use FaceMesh and PoseNet to control the rigging of any 2D SVG character in real time. Try it out for yourself here.
	Delivery Doodles - turn children's drawings into real food Grab's new DeliveryDoodles feature magically turns children’s drawings into food orders using TensorFlowJS. Pretty sure kids will not be drawing vegetables any time soon but still a cool creative use case in production.
	Lip Sync to your favourite song A web experience created by the partner innovation team in APAC that lets you lip sync to music live in the web browser. LipSync was created as a playful way to demonstrate the FaceMesh model for TensorFlow.js. We partnered with Australian singer Tones and I to let you lip sync to Dance Monkey in this demonstration that is now live. You can also read more on our blog about how this was made.
	Teleportation Using some of the latest web technologies available we even managed to teleport ourselves this quarter - no apps required. In the new WFH situation Jason was looking for a more meaningful way to communicate with friends and family and created this prototype using TensorFlow.js that allows him to project himself anywhere in the world using mixed reality and real time communication to make it feel like you are talking to someone right in front of you and is powered by TensorFlow.js of course for segmentation. Learn more here.
	Google Sheets - machine learning in Google Apps Script This article shows how to set up, train, and predict spreadsheet data with deep-learning using Tensorflow.js. You don't need to call REST APIs or use other 3rd party storage. All your data stays in your Google Sheet and runs on the client machine.
	YogAI - your AI Yoga Companion It’s time to get your WFH workout on with this TensorFlow.js powered Yoga teacher using our posenet model. Try it out yourself here. Made by our community member Cristina Maillo, Spain.
	Semantic segmentation - go beyond the world of BodyPix in JS Hugo Zanini has created a brilliant write up showing JS devs how they can take current ML research from Python land and port it to TFJS world to go beyond the models created by our team. This is an important step for the JS community to start contributing to the models available in JS. We are super excited to see where this goes.
	WashOS - detect the sound of running water to time your cleaning habits The famous JS developer DevDevCharlie is embracing TensorFlow.js for her latest creation to help ensure she is washing her hands thoroughly in these current times. Using our sound detection model Charlie is able to detect when water is running and initiate a countdown. On the flipside it would not be too hard to also estimate water usage using this method too. Lots of potential.
	Mortal Kombat - learn how to play your favourite games using your body This great write up explains how one of our community members managed to use TensorFlow.js to play his favourite game. In fact we have seen quite a few gaming HCI examples recently such as this one too.
	Upcycle old devices to make them smart! People throw away devices just because they have a crack in the screen, but often these devices can be upcycled to do more useful things. Using the power of the web, in just 1 click, TensorFlow.js transforms your old device into a smart one that can be useful to you around the home. We can now make something akin to a Nest Cam or Smart Doorbell just by visiting a website as Jason demonstrated here. Kudos to the Symphony team at Google for the inspiration of this prototype.
	Chrome Extensions Join a Google Meet (or any online meeting), emulate a webcam, and apply TensorFlow.js model results to the live cam imagery to produce useful effects or tools. Here, TensorFlow.js works in a background thread in a Chrome Extension so the ML model is loaded only once for efficiency. We encourage users to contribute their own effects to this once open sourced so stay tuned for more details. Combining this with examples above can in real time remove people from the live stream to avoid interruptions when WFH, blur backgrounds, add effects and more. Video here. Contact Jason if you have an effect / idea you would like to try out.
	Clothing Size Prediction in 15 seconds We don’t know about you, but it can be super hard to know what size we are for clothing on different brand websites. Using TensorFlow.js, this prototype allows you to get cm level accuracy which is good enough to predict if you should click on S/M/L on the website at checkout. Finally, no more returns! Watch the video here.

Coming Soon ...

Here are some of the exciting things the team is working on to help grow TensorFlow.js with support for more platforms and use-cases:  Modular TensorFlow.js: Making TensorFlow.js more modular and amenable to tree-shaking. This will allow production users to create bundles of minimal size that just contain the functionality needed to run their specific models and code. New TF.js Converter: Designing a new model converter, which will add support for functional control flow (control flow v2) and support better error handling and user friendly error messages. This converter will leverage MLIR, Google's cutting edge compiler technology for ML, which makes it easier to extend to accommodate feature requests. WASM SIMD Support: With TF.js v2.0.1, the WebAssembly backend supports SIMD. This brings a performance boost of up to 3x for users of our WASM backend on SIMD-compatible platforms.  Support for TF.js in TFX: To make it easier to deploy models with TF.js with existing ML pipelines, TF.js is adding support in TFX pipelines for automatic model conversion and evaluation. This will make it easier to build end-to-end systems.  Model benchmarking toolkit: A new model benchmark toolkit that provides detailed insight into inference time, memory consumption and op performance of our pretrain models and user’s custom model. Initially this benchmark tool aims to help users to measure local performance, the long term goal is to provide services that automatically collect benchmarks for desktop/laptop and various mobile devices. More pretrained models: In the coming quarter, we will be actively working with the researchers to launch several state of art models, including several NLP (Intent, Smart Reply, USE QnA) and AR models. We will also continue to improve the performance and documentation of existing models. Forming a new TensorFlow.js SIG for helping community members drive the platform for specific workstreams.

Events / Other Updates

Past Event Highlights TensorFlow Dev Summit 2020 TFJS  latest features and applications were shown at TF Dev Summit 2020. In the talk (9K views, recording here) TFJS users got to see the new backend WASM and performance, direct saved model execution in Node backend, three newly released TFJS models and the latest TFJS use cases. Web.dev Live 2020 TensorFlow.js kicked off day 2 at Web.dev Live 2020 with Dion Almaer who heads up the web dev ecosystem here at Google and was viewed by over 16,000 people. The talk covered some of the creative ways you can use machine learning in the browser for some delightful results - from invisibility and privacy to teleportation. Check the recording to learn more. OpenJS World 2020 TFJS had a 101 session on both Machine Learning along with how developers should take their first steps with TensorFlow.js at the popular OpenJS World yearly conference. Over 130 people joined the talk even though this was the first time the event had been held digitally. 2nd TensorFlow.js Show & Tell Back by popular demand we hosted the 2nd ever TensorFlow.js show-and-tell livestream (this time at an APAC friendly time) from our new WFH recording studio :-) We had 8 wonderful presenters talking about how they have used TensorFlow.js in their own projects covering making music with tesla coils (seriously cool) from some folk at IBM, doodling on your face in 3D in real time using our face mesh model, to bringing SVG characters to life using pose net and much more. Check out the recording right now! TFjs presented at several other talks this quarter too including: CHI HCI Conference (thanks to Na Li for setting this one up) Women Who Code Los Angeles Bangkit Indonesia (to over 200 students) IEEE Chandigarh (over 800 engineers registered to join the live talk) Our first ever podcast on DevChat.tv for “Adventures in Machine Learning” discussion. University Outreach: Catolica in Bolivia, and Embedded & Distributed AI group at Jonkuping. Numerous GDG group meetups (Mumbai, London, Mysore, Ahmedabad, and more reaching over 150 attendees per talk average) Upcoming events W3C: The future of Machine Learning on the web workshop W3C is convening a virtual workshop to build a shared understanding of how the Machine Learning ecosystem intersects with the Web browsers and help chart its standardization roadmap. TensorFlow.js is presenting two sessions sharing perspectives, lessons learned, and opportunities for web ML.  3rd TensorFlow.js Show & Tell TensorFlow.js Show and Tells are moving to the official TensorFlow YouTube Channel for our 3rd Show & Tell. We have 8 new presenters lined up with some amazing demos, so do look out for announcements hopefully around August for this new format show & tell. See you there. Going forward we are aiming for a session every 2 - 3 months. Have something to show for a future session? Just get in touch.

Project Links

TFJS Site  Get inspired: Search #MadeWithTFJS on Twitter / LinkedIn for great projects using TensorFlow.js	tf...@tensorflow.org (public group)
Please feel free to reach out to the team (tensorflowjs-team@) if you have specific questions, are curious, or would like to see any additional content in the newsletter!