Ready for Trial: Multi-surface capture with auto-accept (for managed sessions only)

Simon Hangl

unread,

May 19, 2023, 9:56:27 AM5/19/23

to blin...@chromium.org

Summary

Web apps will be able to capture multiple surfaces at once.

This feature introduces a new API "getAllScreensMedia()" that allows developers to request several surfaces at once (instead of only one with "getDisplayMedia()"). This API will auto-accept capture requests (for managed sessions only), guarded by policies that have to be explicitly set by the device owners and with clear usage indicators so that users are aware of capturing at all times.

Blink component

Blink

TAG review

None

TAG review status

Pending

Risks

Interoperability and Compatibility

This API rejects requests from pages that are not allow-listed through an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

WebKit: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics

No

Security

Malicious sites exploiting the API and stealing sensitive information shown on the users device: Prevented by enabling the API only for pages allowlisted by the device’s admin.
Users exposing private information on managed devices: Prevented by notifying the user that 1) capture *may* happen in the session, 2) showing a notification when capturing *is* happening and 3) showing a system tray icon when capturing *is* happening (c.f. Privacy comments and approval on https://bugs.chromium.org/p/chromium/issues/detail?id=1300881).

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Goals for experimentation

Learn about the experience of web developers and how this API fulfills their needs.

Ongoing technical constraints

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

Right now this API is only supported on ChromeOS. The privacy measures (e.g. the right way to notify users on login / API users) will have to be developed for other platforms first. Data from the ChromeOS platform will help to design this on other platforms.

Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).

DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

Flag name

enable-get-all-screens-media

Requires code in //chrome?

True. To support this API, embedders need to implement the ContentBrowserClient::ContentCreateScreenEnumerator interface.

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1300883

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1300881

Estimated milestones

DevTrial on desktop

116

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Jeffrey Yasskin

unread,

May 22, 2023, 2:27:23 PM5/22/23

to Simon Hangl, blin...@chromium.org

On Fri, May 19, 2023 at 6:56 AM Simon Hangl <sim...@chromium.org> wrote:

Contact emails
sim...@chromium.org

Explainer
https://github.com/screen-share/capture-all-screens/blob/main/explainer.md

Specification
https://screen-share.github.io/capture-all-screens/

Design docs

https://screen-share.github.io/capture-all-screens
https://github.com/screen-share/capture-all-screens/blob/main/explainer.md
https://docs.google.com/document/d/1p8hhW8cp1PbhEClMTWzYGjfTkBxaNcD34170F60FRpg/edit?resourcekey=0-gLQD4Q6bPVJlZ3gEyZ4_mA#heading=h.qxjlhbc2utcv

Summary
Web apps will be able to capture multiple surfaces at once.

This feature introduces a new API "getAllScreensMedia()" that allows developers to request several surfaces at once (instead of only one with "getDisplayMedia()"). This API will auto-accept capture requests (for managed sessions only), guarded by policies that have to be explicitly set by the device owners and with clear usage indicators so that users are aware of capturing at all times.

Do you have an Alternatives Considered section somewhere that explains why the bikeshed is painted this color? e.g. this could be `getMultipleDisplayMedia()` and be identical to `getDisplayMedia()` except that it returns a sequence<> and lets the user pick more than one screen to present. The UI could have a checkbox for "all screens" if that's a common option. There could be an enterprise policy to auto-accept requests from a particular origin, but such an API could also be useful on the open web, as Elad argued in https://github.com/w3c/mediacapture-screen-share/issues/204.

Blink component
Blink

TAG review
None

TAG review status
Pending

Risks

Interoperability and Compatibility
This API rejects requests from pages that are not allow-listed through an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed
WebKit: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

Have we ever asked their position on a managed-only API? Firefox and Safari do have support for enterprise policies, so it's not obvious that they'd have no opinion about this API.

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics
No

Security
Malicious sites exploiting the API and stealing sensitive information shown on the users device: Prevented by enabling the API only for pages allowlisted by the device’s admin.
Users exposing private information on managed devices: Prevented by notifying the user that 1) capture *may* happen in the session, 2) showing a notification when capturing *is* happening and 3) showing a system tray icon when capturing *is* happening (c.f. Privacy comments and approval on https://bugs.chromium.org/p/chromium/issues/detail?id=1300881).

FWIW, my framework for this is at https://w3ctag.github.io/privacy-principles/#device-administrators, and I think you've done a reasonably good job of handling the tradeoff. My one concern is whether complete screen capture is actually reasonable for device owners' legitimate concerns. Certainly they _want_ to completely surveil their employees or students, and they'll say they need to, but do they really? Do you want to work with your screen always being recorded?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo91PLDMpuKuUBbqTtWvA9KRBAAJW6MxozCKcaipaGOTbQ%40mail.gmail.com.

Elad Alon

unread,

May 23, 2023, 8:10:18 AM5/23/23

to blink-dev, jyas...@chromium.org, blin...@chromium.org, Simon Hangl

Do you want to work with your screen always being recorded?

There are many factors that go into choosing an employer, and some drawbacks can be offset by other benefits. Depending on the opportunities available to me, I could definitely see myself accepting employment at a company which recorded *their* screen while I was using it - provided that I was properly informed! And this is what Simon's API offers over alternatives - whereas native apps running on Windows, Mac or Linux can record a user without their knowledge, an application using the API proposed here would be forced to announce itself to the user.

Would I have preferred a world where no employer ever recorded their employees? Probably. But I don't think we can build this world. So I prefer this lesser evil.

That said, one thing that appears to be missing from the current spec, is a guarantee that users would be informed *in advance* of recording. I would advocate adding this requirement before proceeding.

Simon Hangl

unread,

May 23, 2023, 1:49:20 PM5/23/23

to Elad Alon, blink-dev, jyas...@chromium.org

Thanks for the input @Jeffrey Yasskin .

Do you have an Alternatives Considered section somewhere that explains why the bikeshed is painted this color? e.g. this could be `getMultipleDisplayMedia()` and be identical to `getDisplayMedia()` except that it returns a sequence<> and lets the user pick more than one screen to present. The UI could have a checkbox for "all screens" if that's a common option. There could be an enterprise policy to auto-accept requests from a particular origin, but such an API could also be useful on the open web, as Elad argued in https://github.com/w3c/mediacapture-screen-share/issues/204.

I added a paragraph with the considered alternative in the explainer here.

+1 to what @Elad Alon said. I added a draft for the improvement suggested by @Elad Alon here .

Best,

Simon

Simon Hangl

Software Engineer

sim...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

Reply all

Reply to author

Forward