Web Machine Learning (ML) API POC

[ningxin hu]

Hi Folks,

As you know, there are emerging JavaScript machine learning libs, such as deeplearn.js, keras.js, WebDNN and OpenCV.js. These JS ML libs allow web apps to inference pre-trained ML model on client side. The usages include classifying images, detecting objects and recognizing faces etc.,.

These JS ML libs usually employ WebAssembly (WASM) and WebGL to execute ML workload. However, there is still a big performance gap (up to ~10X scale) between web and native implementation. By using WASM optimization as example, it is possible to optimize the neural network inference by 128-bit SIMD of WebAssembly. However, a native implementation, like MKL-DNN, is able to leverage wider SIMD instructions, e.g. AVX-512, if it is available on device’s CPU. Optimization by WebGL has similar situation comparing to GPU optimization in native/OS ML libs.

Besides the CPU/GPU, the hardware industry moves fast to innovate AI accelerators. Those AI accelerators, from DSP, FPGA to dedicated ASIC, accelerate the performance as well as reduce the power consumption. Especially, that makes efficient ML model inference on edge devices possible. Today's Web platform is disconnected from these AI accelerators.

We know Accelerated Shape Detection API already allows web apps to utilize the hardware capabilities for computer vision. So can we follow this approach and introduce API (WebML) to accelerate machine learning on Web? Client OS are adding ML APIs, for example Google’s Tensoflow-lite/NNAPI, Apple’s CoreML/BNNS/MPS, and Microsoft's WinML/DirectML.

I've been following up with Miguel on this idea and come up with a WebML polyfill and browser POC. You may checkout the MobileNet examples by WebML:

https://huningxin.github.io/webml-examples/

The WebML polyfill implements both WASM and WebGL2 backends. The browser POC is based on Chromium M65. The Android POC implements with NNAPI. The MacOS POC implements with MPSCNN. We observed 6-7X speedup for MobileNet inference by WebML POC comparing to WASM/WebGL2.

Miguel suggest me to check with device-dev folks here. I am attaching a deck about the idea and POC. Your feedbacks are highly welcome. :)

Thanks,

-ningxin

[Cos Thee]

An online function reference listing for OpenCV.js would be nice