Intent to Ship: Capture all screens

[Simon Hangl]

Contact emails

sim...@google.com, swetha...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Specification

https://screen-share.github.io/capture-all-screens

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/1XB8rQRnY5N8G2PeEcNJpVO0q22CutvwW8GGKCZ1z_vc/edit?usp=sharing

Summary

Capture all the screens currently connected to the device using getAllScreensMedia().

Calling getDisplayMedia() multiple times requires multiple user gestures, with the user manually selecting the next screen each time, and without a guarantee to the app that all screens were selected. getAllScreensMedia() improves on all of these fronts.

(As this feature has extreme privacy ramifications, it is only exposed behind an enterprise policy, and users are warned before recording even starts, that recording *could* start at some point.)

Blink component

Blink>Media>GetAllScreensMedia

TAG review

https://github.com/w3ctag/design-reviews/issues/856

TAG review status

TAG has expressed concerns about exposing such a powerful capability on the web. We mitigate their concerns by moving the API to Isolated Web Apps and only exposing it to apps that are explicitly allowlisted by the device owner.

Chromium Trial Name

GetAllScreensMedia

Link to origin trial feedback summary

https://github.com/screen-share/capture-all-screens/issues

Origin Trial documentation link

https://github.com/screen-share/capture-all-screens

Risks

Interoperability and Compatibility

This API is only available to origins allowlisted by administrators through a policy. The policy itself is non-standard, limiting even theoretical interoperability. This API rejects requests from pages that are not allowlisted by an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

WebKit: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics

No

Activation

The challenge for developers is the limitation of the API to origins allowlisted by an enterprise policy.

Security

1. Risk of malicious sites exploiting the API and gaining access to sensitive information on users' devices. This risk is mitigated by the API only being accessible to origins allowlisted by an enterprise policy.

2. Risk of an allowlisted site being compromised to gain access to sensitive information on users’ devices. This risk is mitigated by the API only being accessible to Isolated Web Apps. 

3. Risk of users loading private information that gets recorded and made available to apps affiliated with their device's admin. This risk is mitigated by informing users that recording might start at any moment before the API becomes accessible. (In CrOS, this warning is delivered in the log-in screen, and when users log-in despite the warning, this is tantamount to assent.)

4. Risk of users forgetting that their screens are being recorded. This risk is mitigated through a persistent notification.

WebView application risks

N/A (No change in behavior for existing APIs).

Debuggability

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

This API is initially implemented on CrOS, where demand for it is greatest, and where we have the most flexibility in offering users early warning that their screens may be recorded if they proceed past the log-in screen. Lessons learned from shipping this API on CrOS will be used when deciding how to correctly implement such warnings on other platforms.

Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).

DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

Flag name on chrome://flags

chrome://flags#enable-get-all-screens-media

Finch feature name

GetAllScreensMedia

Non-finch justification

This feature is only available through active enabling by admin policy and can be disabled the same way at any time.

Requires code in //chrome?

True

Tracking bug

https://issues.chromium.org/issues/40216442

Launch bug

https://launch.corp.google.com/launch/4276771

Measurement

As this is a managed feature, monthly active users can be measured and are displayed at go/contact-center-dashboard (Googlers only).

Availability expectation

Feature is available only on ChromeOS for the foreseeable future.

Adoption expectation

We anticipate this feature being used by partners in the contact center space (or other areas that have to comply with regulation or established usage patterns that require screen capture).

Adoption plan

There is already a significant number of developers that are working on integrating this feature in their products (beyond the developers that expressed public interest here).

Non-OSS dependencies

At this time, this feature is only enabled through the Chrome admin panel.

Sample links

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Estimated milestones

Shipping on desktop	137
Origin trial desktop first	118
Origin trial desktop last	128
Origin trial extension 1 end milestone	131
DevTrial on desktop	116

Note there is a gap between the end of the origin trial (M131) and the launch of this API (M137). Developers are currently using this API in PWAs via OT and we agreed with Blink owners (assuming substantial progress on the launch of this API in Isolated Web Apps) to extend the OT until (including) M136 to enable developers to move from PWAs to IWAs. Please refer to this thread for further information.

Anticipated spec changes

No open issues and no anticipated changes.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120?gate=5610053803966464

Links to previous Intent discussions

Intent to Prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/6TRT0XsVOE4/m/NOm-YEQCAgAJ

Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/HErdlr3e_V0

[Reilly Grant]

LGTM as an IWA OWNER (3x LGTM from Blink API OWNERS are still required according to the IWA-specific API launch process).

This feature has been an interesting case study for when to restrict a capability to IWAs because the two underlying components of it, capturing a screen and enterprise policies to control a permission, exist in other features without requiring such drastic security measures. In consultation with the Security reviewers however we found that the combination of a requirement to capture all screens (which is more dangerous than normal screen capture because it doesn't allow the user to differentiate between shared and unshared content) and an administrator control (which removes user agency to decide whether their screen is displaying sensitive information) makes this feature a particularly attractive target for an attacker and necessitates the code integrity protections provided by Isolated Web Apps.

Reilly Grant | Software Engineer | rei...@chromium.org | Google Chrome

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP0TkgEM43oxOSdADK5upZauT9HgGnse4AfS5r403kKs9uoi8Q%40mail.gmail.com.

[Alex Russell]

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Vladimir Levin]

LGTM2

LGTM1

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP0TkgEM43oxOSdADK5upZauT9HgGnse4AfS5r403kKs9uoi8Q%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/aecf44e6-c464-4cc9-bb8d-07b284ad2f61n%40chromium.org.

[Yoav Weiss (@Shopify)]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADsXd2NVJ469vLsTo1tmjtAM%2Bnu5mnELkkCeo7gFdHpSviLoBA%40mail.gmail.com.