Intent to Extend Experiment 2: Capture all screens

[Simon Hangl]

Contact emails

sim...@google.com, swetha...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Specification

https://screen-share.github.io/capture-all-screens

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/1XB8rQRnY5N8G2PeEcNJpVO0q22CutvwW8GGKCZ1z_vc/edit?usp=sharing

Summary

Capture all the screens currently connected to the device using getAllScreensMedia().

Calling getDisplayMedia() multiple times requires multiple user gestures, with the user manually selecting the next screen each time, and without a guarantee to the app that all screens were selected. getAllScreensMedia() improves on all of these fronts.

(As this feature has extreme privacy ramifications, it is only exposed behind an enterprise policy, and users are warned before recording even starts, that recording *could* start at some point.)

Blink component

Blink>Media>GetAllScreensMedia

TAG review

https://github.com/w3ctag/design-reviews/issues/856

TAG review status

TAG has expressed concerns about exposing such a powerful capability on the web. We mitigate their concerns by moving the API to Isolated Web Apps and only exposing it to apps that are explicitly allowlisted by the device owner.

Chromium Trial Name

GetAllScreensMedia

Link to origin trial feedback summary

https://github.com/screen-share/capture-all-screens/issues

Origin Trial documentation link

https://github.com/screen-share/capture-all-screens

Risks

Interoperability and Compatibility

This API is only available to origins allowlisted by administrators through a policy. The policy itself is non-standard, limiting even theoretical interoperability. This API rejects requests from pages that are not allowlisted by an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

WebKit: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics

No

Activation

The challenge for developers is the limitation of the API to origins allowlisted by an enterprise policy.

Security

1. Risk of malicious sites exploiting the API and gaining access to sensitive information on users' devices. This risk is mitigated by the API only being accessible to origins allowlisted by an enterprise policy.

2. Risk of an allowlisted site being compromised to gain access to sensitive information on users’ devices. This risk it mitigated by the API only being accessible to Isolated Web Apps. 

3. Risk of users loading private information that gets recorded and made available to apps affiliated with their device's admin. This risk is mitigated by informing users that recording might start at any moment before the API becomes accessible. (In CrOS, this warning is delivered in the log-in screen, and when users log-in despite the warning, this is tantamount to assent.)

4. Risk of users forgetting that their screens are being recorded. This risk is mitigated through a persistent notification.

WebView application risks

N/A (No change in behavior for existing APIs).

Debuggability

Reason this experiment is being extended

This is needed to provide developers with enough time to pivot their drive-by web solutions to isolated web app solutions. 

We have made significant progress on the launch of `getAllScreensMedia` for isolated web apps and as discussed here, this is a requirement to extend the origin trial.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

This API is initially implemented on CrOS, where demand for it is greatest, and where we have the most flexibility in offering users early warning that their screens may be recorded if they proceed past the log-in screen. Lessons learned from shipping this API on CrOS will be used when deciding how to correctly implement such warnings on other platforms.

Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).

DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

Flag name on chrome://flags

chrome://flags#enable-get-all-screens-media

Finch feature name

GetAllScreensMedia

Non-finch justification

This feature is only available through active enabling by admin policy and can be disabled the same way at any time.

Requires code in //chrome?

True

Tracking bug

https://issues.chromium.org/issues/40216442

Launch bug

https://launch.corp.google.com/launch/4276771

Measurement

As this is a managed feature, monthly active users can be measured and are displayed at go/contact-center-dashboard (Googlers only).

Availability expectation

Feature is available only on ChromeOS for the foreseeable future.

Adoption expectation

We anticipate this feature being used by partners in the contact center space (or other areas that have to comply with regulation or established usage patterns that require screen capture).

Adoption plan

There is already a significant number of developers that are working on integrating this feature in their products (beyond the developers that expressed public interest here).

Non-OSS dependencies

At this time, this feature is only enabled through the Chrome admin panel.

Sample links

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Estimated milestones

Shipping on desktop	137
Origin trial desktop first	118
Origin trial desktop last	128
Origin trial extension 1 end milestone	131
Origin trial extension 2 end milestone	136
DevTrial on desktop	116

Note that we already have Blink owners approval to launch the API in M137 (intent to ship). As discussed in the first intent to extend experiment, we request this extension to (including) M136 in order to give developers sufficient time to move from their drive-by web solutions to isolated web app solutions.

Anticipated spec changes

No open issues and no anticipated changes.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120?gate=5610053803966464

Links to previous Intent discussions

Intent to Prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/6TRT0XsVOE4/m/NOm-YEQCAgAJ

Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/HErdlr3e_V0

[Alex Russell]

LGTM

[Alex Russell]

Hey, going back through the timelines here, it's worrying that this feature isn't going to ship until 137. The OT isn't particularly problematic as you're going to ship the same API shape, but the duration of this whole thing is worrying. Why isn't this going to launch in 131?

[Simon Hangl]

Hi, this is entirely for process reasons. The origin trial needs to be alive until (including) M136 to allow developers to move to isolated web apps. Can I formally keep the drive-by web OT alive while launching the IWA version now? If so, I am happy to launch this earlier.

[Simon Hangl]

As a quick clarification: The version of getAllScreensMedia that is exposed in isolated contexts will launch the latest by M132 (I am targeting to make the branch date for M131). The only work left is to land a CL that exposes the API in isolated contexts while still keeping the OT on the drive-by web open. This way, developers have at least 4 milestones to pivot.

I will update the Chrome status entry.