Intent to Prototype: Background Mask/Segmentation

72 views
Skip to first unread message

Bhaumik, Rijubrata

unread,
Jun 26, 2024, 5:22:26 PM (3 days ago) Jun 26
to blin...@chromium.org, Hakkinen, Eero, elad...@google.com, Harald Alvestrand, mark a. foltz, Guido Urdaneta, François Beaufort 🇫🇷, CJ Huang

Contact emails

rijubrat...@intel.com


Explainer

https://github.com/riju/backgroundBlur/blob/main/explainer.md#background-segmentation-mask-api


Specification

https://github.com/w3c/mediacapture-extensions/pull/142


Summary

The Background Segmentation/Mask API allows web developers to use the native platform's API for camera, leveraging on-device AI models for background segmentation. Today, on Windows/Apple, this can leverage NPU (Neural Processing Units) and bring power and performance gains. This extends our previously proposed Background Blur API [1] with some implementation difference [2] –

  • BG Blur preprocesses video frames and replaces the original video frames with ones with background blurred thus the original video frames become unavailable from a web application point of view (until BG Blur is disabled)
  • BG MASK retains the original frames intact, does segmentation and provides mask frames in addition to the original video frames thus web applications receive both the original frames and mask frames in the same video frame stream.

 

In many cases, it might be important to have access to the original camera feed [3]. Green Screen, Background Replacement (with an Image) and many other use cases can be enabled [4] using this feature via Canvas and WebGPU/GL pipeline. There is still an ongoing discussion about how Mask data is best provided [5].

[1] https://developer.chrome.com/blog/background-blur/
[2] https://github.com/riju/backgroundBlur/blob/main/explainer.md#blur-vs-mask
[3] https://github.com/w3c/mediacapture-extensions/issues/118#issuecomment-1772204182
[4] https://github.com/riju/backgroundBlur/blob/main/explainer.md#demo

[5] https://github.com/riju/backgroundBlur/blob/main/explainer.md#mask-data----videoframe--imagebitmap--imagedata

 

Blink component

Blink>ImageCapture

Motivation

A vast majority of communication these days happens on our client devices and features like Blur, Background Replacement, Green Screen have become table stakes. This Background Segmentation-Mask API gives developers a choice to use the native platform's API and leverage the on-device AI models already present in the platform.

 

Initial public proposal

https://github.com/w3c/mediacapture-extensions/pull/142

TAG review

None

TAG review status

Pending

Risks

 

Interoperability and Compatibility

None


Gecko: No signal. https://github.com/mozilla/standards-positions/issues/1041

WebKit: No signal. https://github.com/WebKit/standards-positions/issues/364

Web developers: No signals

Other signals:


WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

 

Debuggability

None

 

Is this feature fully tested by web-platform-tests?

No

Flag name on chrome://flags

None

Finch feature name

None

Non-finch justification

None

Requires code in //chrome?

False

Estimated milestones

No milestones specified

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5187283610501120

This intent message was generated by Chrome Platform Status.

 

Reply all
Reply to author
Forward
0 new messages