Intent to Experiment: Background Blur API.

[Bhaumik, Rijubrata]

Contact emails

rijubrat...@intel.com, eero.h...@intel.com, che...@google.com, fbea...@google.com

 

Explainer

https://github.com/riju/backgroundBlur/blob/main/explainer.md

 

Specification

https://w3c.github.io/mediacapture-extensions/#exposing-mediastreamtrack-source-background-blur-support

https://w3c.github.io/mediacapture-extensions/#exposing-change-of-mediastreamtrack-configuration

 

Summary

The Background Blur API allows web developers to use the native platform's API for camera background segmentation. As Background Blur has become one of the most used features on video conferencing apps like Teams, Meet, Zoom, and Webex, we want web apps to leverage the same platform APIs without completely relying on ML frameworks like TensorFlow.js, Mediapipe, WASM libraries or cloud-based solutions.

 

Blink component

Blink>ImageCapture

 

TAG review

https://github.com/w3ctag/design-reviews/issues/826

 

TAG review status

Pending

 

Risks

 

External signals

Gecko: Positive (https://github.com/mozilla/standards-positions/issues/658)

 

WebKit: Positive (https://github.com/WebKit/standards-positions/issues/5)

 

Web developers: Positive (https://twitter.com/quicksave2k/status/1638908125385220096)

 

Debuggability

No specific DevTools changes are required. This feature is treated like any other JS method.

 

Ongoing technical constraints

None.

 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No - Supported on ChromeOS, Windows, and macOS, due to the feature’s reliance on platform APIs.

 

Is this feature fully tested by web-platform-tests?

Yes. https://wpt.fyi/results/mediacapture-extensions/GUM-backgroundBlur.https.html?label=experimental&label=master&aligned

 

Flag name

--enable-blink-features=MediaCaptureBackgroundBlur

 

Requires code in //chrome?

No.

 

Tracking bug

https://crbug.com/1338665

 

Estimated milestone

We hope to start the Origin Trial sometime during M114 beta. We plan to continue the Origin Trial until at least M117 to give developers time to test the API and provide feedback. 

 

Goals for experimentation

Large customers like Meet can evaluate this API on a big number of users. We can learn if the API in its present form satisfies the needs of the users, or do we need something more to wow the users. One concrete example would be boolean vs enumeration. Right now, BG Blur only uses a bool which is an ON/OFF switch, but ChromeOS and Windows 11 offer shallow focus also where the blur intensity is more than the present implementation. We wanted to start off with something simple, get feedback and iterate on building something which the majority wants.

 

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5147589575442432

 

Links to previous Intent discussions

https://groups.google.com/a/chromium.org/g/blink-dev/c/3ke9hgaREwM

 

 

[Mike Taylor]

LGTM to experiment from M114 to M117 inclusive.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CO1PR11MB5186BD2AC21518C03A5211FDF08B9%40CO1PR11MB5186.namprd11.prod.outlook.com.

[François Beaufort]

Sorry for the typo. Here's the correct URL: https://chromestatus.com/feature/5077577782263808 

 

Links to previous Intent discussions

https://groups.google.com/a/chromium.org/g/blink-dev/c/3ke9hgaREwM

 

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CO1PR11MB5186BD2AC21518C03A5211FDF08B9%40CO1PR11MB5186.namprd11.prod.outlook.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/713c866a-a1ef-39d1-bbdc-820ac06ee928%40chromium.org.

[Arthur Hemery]

Hi Bhaumik,
There was one question left unanswered in the I2P thread from the security review, so I'm reposting it here:

"This is my understanding, let me know if that's correct:
Background blur is applied by request or constraint. In the first implementation only stream sources like cameras obtained by GetUserMedia support blurring. This is completely unidirectional, from the device to the platform. Getting a stream from another source such as CanvasCaptureMediaStreamTrack will simply not support the capability. Therefore there is no way to send custom crafted bits into the native APIs, and in general no content from the internet flows into the native APIs."

Is that accurate?

Cheers,
Arthur

[eero.h...@intel.com]

Hi Arthur,

On Tuesday, March 28, 2023 at 11:13:45 AM UTC+3 ahe...@google.com wrote:

Hi Bhaumik,
There was one question left unanswered in the I2P thread from the security review, so I'm reposting it here:

"This is my understanding, let me know if that's correct:
Background blur is applied by request or constraint. In the first implementation only stream sources like cameras obtained by GetUserMedia support blurring. This is completely unidirectional, from the device to the platform. Getting a stream from another source such as CanvasCaptureMediaStreamTrack will simply not support the capability. Therefore there is no way to send custom crafted bits into the native APIs, and in general no content from the internet flows into the native APIs."

Is that accurate?

Yes, that's accurate.

I also updated  the Explainer Security considerations sections to describe this.

BR,
 Eero

[Kaustubha Govind]

Hi all,

Thanks for your work on this feature!

Would you be able to expand your Privacy Considerations section to address whether the API is likely to expose any new fingerprinting surfaces? Particularly; I am interested in understanding:

1. Whether there is additional information about the platform (e.g. OS version) that can be gleaned by querying whether the platform supports background blurring. Note that with the User Agent Reduction work, we are attempting to limit default access to some of this platform identifying information.
2. How stable do we expect the blur to be across hardware configurations? It would be unfortunate if this turned into a problem similar to Canvas Fingerprinting.

Perhaps one mitigation here is that the capability is gated behind the getUserMedia() permission; which limits drive-by fingerprinting?

I would appreciate a brief analysis of whether or not any fingerprinting risks exist; and if yes, how those risks can be mitigated by the implementation.

Thank you!

Kaustubha

[eero.h...@intel.com]

Hi,

On Thurday, March 30, 2023 at 3:43:06 PM UTC+3Kaustubha Govind wrote:

Would you be able to expand your Privacy Considerations section to address whether the API is likely to expose any new fingerprinting surfaces?

I have added Fingerprinting subsection to our  Privacy Considerations section.

Particularly; I am interested in understanding:

1. Whether there is additional information about the platform (e.g. OS version) that can be gleaned by querying whether the platform supports background blurring. Note that with the User Agent Reduction work, we are attempting to limit default access to some of this platform identifying information.

In short, there isn't but see the linked Fingerprinting subsection for detailed analysis.

2. How stable do we expect the blur to be across hardware configurations? It would be unfortunate if this turned into a problem similar to Canvas Fingerprinting.

In Canvas Fingerprinting, sites can draw to a canvas (the source) as they like and fingerprint the result which may be a bit different on different platforms and platform versions. I assume that you refer to this (lack of) stable result here.

This is really not an issue with background blur. In the case of the getUserMedia tracks, the sites has access to tracks which may have background blur in effect and may allow background blur to be disabled/enabled and to the resulting frames. However, sites have no control on the source (what's in the field-of-view of a camera) so it is not possible to reprocess the same frames on different platforms and platform versions and to compare the results.

 

Perhaps one mitigation here is that the capability is gated behind the getUserMedia() permission; which limits drive-by fingerprinting?

That is the main gate but see the linked Fingerprinting subsection for detailed analysis.

BR,
 Eero

[Kaustubha Govind]

Thanks for explaining the fingerprinting considerations, Eero!

One quick correction: the reduced User Agent string does not reveal OS/platform version as your recently added text seems to suggest. Other than that, I am satisfied that the API does not elevate fingerprinting risk.