Intent to Prototype : Background Blur API.

362 views
Skip to first unread message

Bhaumik, Rijubrata

unread,
Jun 22, 2022, 5:46:20 PMJun 22
to blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla

Contact emails

rijubrat...@intel.com, eero.h...@intel.com

Explainer

https://github.com/riju/backgroundBlur/blob/main/explainer.md

Specification

https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support

Summary

Background Blur API gives web developers a choice to use the native platform's API for background segmentation. Background Blur has become one of the most used features on Video conferencing apps like Teams, Meet, Zoom, Webex, etc. We want give Web apps similar capabilities as their native counterparts, leveraging the same platform APIs and to delight users without completely relying on ML frameworks like TensorFlow.js, Mediapipe, WASM libraries or cloud based solutions.

 

Blink component

Blink>ImageCapture

Motivation

A vast majority of communication these days happens on our client devices. During video meetings, participants are usually aware of how they look and what their environment (usually their home) is revealing to the audience. Most folks, especially ones without a dedicated office space would be inclined to hide messy rooms with pets and kids. Video meetings like face to face meetings are important for non-verbal communication but participants would rather focus on the important subject by removing the distractions in the background and prevent any accidental snafus. Microsoft says in a 38 minute conference call, 13 minutes are wasted dealing with distractions and interruptions. Background Blur goes a long way to cutting down those disruptions. Zoom says- "When a custom virtual background is unavailable or not suiting your needs, but you still want to maintain some privacy with regards to your surroundings, the blur background option can be a great alternative. This option simply blurs the background of your video, obscuring exactly who or what is behind you. It's great for hiding a cluttered dorm room, taking a meeting in a coffee shop, or just keeping things professional." . In fact, NCSC (National Cyber Security Centre UK) suggests using background Blur or a background image for staff meetings to add a degree of personal privacy.

On the Web, due to a lack of a standardized API for Background Blur and widespread demand, developers have no options but to use ML frameworks like Tensorflow.js and other WASM libraries to satisfy their customers. This Background Blur API gives developers a choice to use the native platform's API.

Initial public proposal

https://github.com/w3c/mediacapture-extensions/pull/49
https://github.com/w3c/mediacapture-extensions/pull/61

Search tags

background blurwebrtc

TAG review

TAG review status

Pending

Risks

 

Interoperability and Compatibility



Gecko: No signal, but spec LGTM ed by Jan-Ivar (Mozilla). https://github.com/mozilla/standards-positions/issues/658

WebKit: No signal but Apple co-author of explainer / PR. https://lists.webkit.org/pipermail/webkit-dev/2022-June/032311.html

Web developers: No signals

Other signals: Almost every video-conference app has background blur feature.


WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?



Debuggability



Is this feature fully tested by web-platform-tests?

Yes ( as part of CL )

Flag name

MediaCaptureBackgroundBlur


Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1338665

 

Estimated milestones

No milestones specified



Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5077577782263808

This intent message was generated by Chrome Platform Status.

 

Yoav Weiss

unread,
Jun 23, 2022, 3:26:43 AMJun 23
to Bhaumik, Rijubrata, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla
Exciting work!! 

On Wed, Jun 22, 2022 at 11:46 PM Bhaumik, Rijubrata <rijubrat...@intel.com> wrote:

Contact emails

rijubrat...@intel.com, eero.h...@intel.com

Explainer

https://github.com/riju/backgroundBlur/blob/main/explainer.md

Specification

https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support

Summary

Background Blur API gives web developers a choice to use the native platform's API for background segmentation. Background Blur has become one of the most used features on Video conferencing apps like Teams, Meet, Zoom, Webex, etc. We want give Web apps similar capabilities as their native counterparts, leveraging the same platform APIs and to delight users without completely relying on ML frameworks like TensorFlow.js, Mediapipe, WASM libraries or cloud based solutions.

 

Blink component

Blink>ImageCapture

Motivation

A vast majority of communication these days happens on our client devices. During video meetings, participants are usually aware of how they look and what their environment (usually their home) is revealing to the audience. Most folks, especially ones without a dedicated office space would be inclined to hide messy rooms with pets and kids. Video meetings like face to face meetings are important for non-verbal communication but participants would rather focus on the important subject by removing the distractions in the background and prevent any accidental snafus. Microsoft says in a 38 minute conference call, 13 minutes are wasted dealing with distractions and interruptions. Background Blur goes a long way to cutting down those disruptions. Zoom says- "When a custom virtual background is unavailable or not suiting your needs, but you still want to maintain some privacy with regards to your surroundings, the blur background option can be a great alternative. This option simply blurs the background of your video, obscuring exactly who or what is behind you. It's great for hiding a cluttered dorm room, taking a meeting in a coffee shop, or just keeping things professional." . In fact, NCSC (National Cyber Security Centre UK) suggests using background Blur or a background image for staff meetings to add a degree of personal privacy.

On the Web, due to a lack of a standardized API for Background Blur and widespread demand, developers have no options but to use ML frameworks like Tensorflow.js and other WASM libraries to satisfy their customers. This Background Blur API gives developers a choice to use the native platform's API.

Initial public proposal

https://github.com/w3c/mediacapture-extensions/pull/49
https://github.com/w3c/mediacapture-extensions/pull/61

Search tags

background blurwebrtc

TAG review


May be a good idea to file for a TAG review early.
 


TAG review status

Pending

Risks

 

Interoperability and Compatibility



Gecko: No signal, but spec LGTM ed by Jan-Ivar (Mozilla). https://github.com/mozilla/standards-positions/issues/658

WebKit: No signal but Apple co-author of explainer / PR. https://lists.webkit.org/pipermail/webkit-dev/2022-June/032311.html

Web developers: No signals

Other signals: Almost every video-conference app has background blur feature.


WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?



Debuggability



Is this feature fully tested by web-platform-tests?

Yes ( as part of CL )

Flag name

MediaCaptureBackgroundBlur


Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1338665

 

Estimated milestones

No milestones specified



Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5077577782263808

This intent message was generated by Chrome Platform Status.

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/PH0PR11MB5191E86DD684B0D86A12B0EEF0B29%40PH0PR11MB5191.namprd11.prod.outlook.com.

François Beaufort 🇫🇷

unread,
Jun 23, 2022, 3:51:08 AMJun 23
to Yoav Weiss, Bhaumik, Rijubrata, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla

Thank you for starting this effort!

Quick question: Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Not just exposing backgroundBlur properties but also it "real" support.

Bhaumik, Rijubrata

unread,
Jun 23, 2022, 4:34:20 AMJun 23
to François Beaufort 🇫🇷, Yoav Weiss, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla

We are going to first implement this on Windows. “Real” support is dependent on driver support. If on your client machine, the camera app can do it, then this API can also do it.
I don’t have a Mac, but there’s platform support, so we can collaborate once the initial Blink CLs lands.

Ken Russell

unread,
Jun 24, 2022, 6:45:52 PMJun 24
to Bhaumik, Rijubrata, François Beaufort 🇫🇷, Yoav Weiss, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla
May I please ask two questions?

1) Has any thought been given to exposing the segmentation result - a grayscale mask defining the foreground (person) vs. the background?

Building this functionality on top of technologies like MediaPipe, Tensorflow.js, Wasm, etc. uses the web platform's existing low-level, composable primitives to provide a range of capabiilties. On the web today, in addition to background blur, MediaPipe and Meet offer many background replacement options, which are arguably more privacy-preserving than just blurring. If the segmentation result were provided, the web application could implement these effects on top of it.

It's understood that this API enables the use of custom IP cores which can deliver a result with lower power. At the same time, it's less flexible than what can be done on the web today.

2) Have the MediaPipe or Meet teams been asked for input? MediaPipe and Meet pioneered background effects in videoconferencing on the web. They're conspicuously absent from the "Stakeholder Feedback/Opposition" section of https://github.com/riju/backgroundBlur/blob/main/explainer.md .

Thanks,

-Ken



Bhaumik, Rijubrata

unread,
Jun 29, 2022, 4:16:55 AMJun 29
to Ken Russell, François Beaufort 🇫🇷, Yoav Weiss, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla

Thanks Ken,

 

  1. segmentation result - a grayscale mask : Yes, Harald (Google) did comment on this, something that MediaPipe uses. We discussed that MASK would be more important in Background Replacement scenarios. Since no platform APIs support Background Replacement right now, we decided to move it to future work.
  2. Meet for input : Sorry, fixed that in the explainer. I am reaching out to Meet again. In general, most of the stakeholders were supportive and thought it’s definitely worth prototyping. I will try to add more quotes/ feedback from others.

Arthur Hemery

unread,
Jun 29, 2022, 6:16:46 AMJun 29
to blink-dev, rijubrat...@intel.com, François Beaufort, yoav...@chromium.org, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla, Kenneth Russell
Hi there!

We discussed this yesterday in the OWP Security&Privacy review and had a couple of questions/remarks:
- Using underlying platform APIs is fine as long as all providers are aware that they are going to receive random content from the internet. It's important because they are often more privileged than Chrome. We should strongly fuzz these API to verify that they meet the bar.
- From reading the spec it seems possible to obtain a MediaStreamTrack that contains arbitrary data and to apply background blur to it. Is that correct? If that's the case, then it makes fuzzing even more important.
- With this change we know via constraints when the system has already blurred the image right? I imagine you get this information only after you've queried a MediaStreamTrack? 

Thanks!
Arthur

Bhaumik, Rijubrata

unread,
Jun 30, 2022, 9:32:33 AMJun 30
to Arthur Hemery, blink-dev, François Beaufort, yoav...@chromium.org, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla, Kenneth Russell

Hi Arthur,

Thanks for the questions, I will add more info in the Explainer regarding Privacy in Security.

Replies inline -

 

From: Arthur Hemery <ahe...@google.com>
Sent: Wednesday, June 29, 2022 1:17 PM
To: blink-dev <blin...@chromium.org>
Cc: Bhaumik, Rijubrata <rijubrat...@intel.com>; François Beaufort <fbea...@google.com>; yoav...@chromium.org <yoav...@chromium.org>; blin...@chromium.org <blin...@chromium.org>; Guido Urdaneta <gui...@chromium.org>; Harald Alvestrand <h...@google.com>; Youenn Fablet <you...@apple.com>; Bernard Aboba <Bernar...@microsoft.com>; Jan-Ivar Bruaroey @Mozilla <j...@mozilla.com>; Kenneth Russell <k...@chromium.org>
Subject: Re: [blink-dev] Intent to Prototype : Background Blur API.

 

Hi there!



We discussed this yesterday in the OWP Security&Privacy review and had a couple of questions/remarks:
- Using underlying platform APIs is fine as long as all providers are aware that they are going to receive random content from the internet. It's important because they are often more privileged than Chrome. We should strongly fuzz these API to verify that they meet the bar.

[Bhaumik, Rijubrata] The only source for Background Blur API is client-side camera via getUserMedia() and the native API is going to do the entire processing on the client. The native APIs does in-stream correction. Only then is the "blurred" stream sent to the internet. 


- From reading the spec it seems possible to obtain a MediaStreamTrack that contains arbitrary data and to apply background blur to it. Is that correct? If that's the case, then it makes fuzzing even more important.

[Bhaumik, Rijubrata] Actually, the implementation [CL] does not itself apply background blur but it instead requests camera (drivers) to apply background blur so that the browser gets video frames whose background is blurred. I am not sure if it’s possible for the browser to retrieve unblurred video frames and try to apply background blur constraint to (frames originating from a canvas) those video frames, but this implementation does not do that.  


- With this change we know via constraints when the system has already blurred the image right? I imagine you get this information only after you've queried a MediaStreamTrack? 

[Bhaumik, Rijubrata] Yes, it is possible to know via the MediaStreamTrack’s settings the current background blurring is in effect (whether it is due to a system wide default or due to a constraints applied by a web application). The MediaStreamTrack’s settings are only available after a web application has requested to get a MediaStream and the user has granted that.

Arthur Hemery

unread,
Jul 4, 2022, 5:48:37 AMJul 4
to Bhaumik, Rijubrata, François Beaufort, yoav...@chromium.org, blin...@chromium.org, Guido Urdaneta, Harald Alvestrand, Youenn Fablet, Bernard Aboba, Jan-Ivar Bruaroey @Mozilla, Kenneth Russell
Hi Bhaumik,

Thanks for the precisions! Let me rephrase to verify my understanding:
"Background blur is applied by request or constraint. In the first implementation only stream sources like cameras obtained by GetUserMedia support blurring. This is completely unidirectional, from the device to the platform. Getting a stream from another source such as CanvasCaptureMediaStreamTrack will simply not support the capability. Therefore there is no way to send custom crafted bits into the native APIs, and in general no content from the internet flows into the native APIs."

Is that accurate?

Cheers,
Arthur


Reply all
Reply to author
Forward
0 new messages