Intent to Experiment: Capture all screens

[Simon Hangl]

Contact emails

sim...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/explainer.md

Specification

https://screen-share.github.io/capture-all-screens/

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/explainer.md

https://docs.google.com/document/d/1p8hhW8cp1PbhEClMTWzYGjfTkBxaNcD34170F60FRpg/edit?resourcekey=0-gLQD4Q6bPVJlZ3gEyZ4_mA#heading=h.qxjlhbc2utcv

Summary

Capture all the screens currently connected to the device using getAllScreensMedia().

Calling getDisplayMedia() multiple times requires multiple user gestures, burdens the user with choosing the next screen each time, and does not guarantee to the app that all the screens were selected. getAllScreensMedia() improves on all of these fronts.

(As this feature has extreme privacy ramifications, it is only exposed behind an enterprise policy, and users are warned before recording even starts, that recording *could* start at some point.)

Blink component

Blink>GetDisplayMediaSet

TAG review

None

TAG review status

Pending

Risks

Interoperability and Compatibility

This API is only available to origins allowlisted by administrators through a policy. The policy itself is non-standard, limiting even theoretical interoperability.This API rejects requests from pages that are not allow-listed through an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

WebKit: N/A - given that the API is limited to managed configurations, it's not clear that requesting a position is needed

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics

No

Security

1. Risk of malicious sites exploiting the API and gaining access to sensitive information on users' devices. This risk is mitigated by the API only being accessible to origins allowlisted by an enterprise policy.

2. Risk of users loading private information that gets recorded and made available to apps affiliated with their device's admin. This risk is mitigated by informing users that recording might start at any moment before the API becomes accessible. (In CrOS, this warning is delivered in the log-in screen, and when users log-in despite the warning, this is tantamount to assent.)

3. Risk of users forgetting that their screens are being recorded. This risk is mitigated through a persistent notification.

WebView application risks

None

Goals for experimentation

Learn about the experience of web developers and how this API fulfills their needs.

Ongoing technical constraints

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

This API is initially implemented on CrOS, where demand for it is greatest, and where we have the most flexibility in offering users early warning that their screens may be recorded if they proceed past the log-in screen. Lessons learned from shipping this API on CrOS will be used when deciding how to correctly implement such warnings on other platforms.

Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).

DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

Flag name

enable-get-all-screens-media

Requires code in //chrome?

True. To support this API, embedders need to implement the ContentBrowserClient::ContentCreateScreenEnumerator interface.

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1300883

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1300881

Estimated milestones

OriginTrial desktop last	119
OriginTrial desktop first	117
DevTrial on desktop	116

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Ready for trial:

https://groups.google.com/a/chromium.org/g/blink-dev/c/e_cUVJEvzuY?pli=1

[Chris Harrelson]

LGTM

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo8ijX%3DMG_bE_U0%2B_HMOd1VSBj8GM5fv13HZ69gtoeHgdw%40mail.gmail.com.

[Mike Taylor]

Also, it would be good to file a TAG review sooner than later to prevent any delays at a possible future I2S time.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw8UCe1jw3D%2BKd_%2Be%2BGJSthaMPy6fU3pDf%3DARRk_pYry3w%40mail.gmail.com.

[Artur Janc]

Hey folks,

I've taken a look at the explainer as part of our security/privacy triage process (note: it looks like the link above is broken, it should be https://github.com/screen-share/capture-all-screens) and wanted to chime in with some security feedback.

Concretely, this API gives any origin that has the ability to call it (anything allowlisted in the Enterprise Policy) access to ~all of the user's browsing data -- not just what's displayed on the screen. This is because the caller can also redirect the user (or open popups, etc.) to arbitrary cross-site endpoints which might contain authentication tokens (e.g. JSON responses with a JWT or similar secret); these credentials can later be used to gain access to accounts to which the user is logged in, even if the user hasn't intentionally interacted with the affected websites while screen capture was happening.

This makes any allowlisted origin privileged to access almost arbitrary cross-origin information; any XSS bug in such an origin will be capable of doing the same. This introduces a somewhat unprecedented attack surface where finding a common web vulnerability in a single origin/site will give attackers the capability to persistently compromise the user's accounts on other sites. 

My guess is that the proposal should include a security section to discuss this risk and implement restrictions on the API to prevent its abuse. Some possibilities include ensuring that JS can't get access to the MediaStream (e.g. the MediaStream can be sent to an allowlisted origin but not accessed by scripts) and requiring the user to click through a non-dismissible All-Screen Capture interstitial before starting capture (to prevent the API from being called when the user is away from the device). I'd suggest having a fairly in-depth discussion of the options to prevent this API from introducing a substantial risk for its users.

Cheers,

-Artur

[Elad Alon]

Hi Arthur!

Using the pre-existing getDisplayMedia(), an application can already gain access to the user's screen and capture content in pop-ups, detect browsing history, etc. - provided the user chooses to capture their (often only) screen. All of the concerns above apply for gDM. Further, any website can call gDM.

In contrast, getAllScreensMedia() can only be called by websites expressly allowlisted by the device's owner. This configuration would typically be carried out by an admin who is more tech-savvy than the average user, and it would usually be the admin's job to apply some good judgement with respect to security risks their organization might face.

It is expected that a reasonable company would only allowlist:
1. Its own applications.
2. Applications from providers who were contracted specifically for this purpose, and who are trusted by the contracting company to such an extent, that long-term full-screen recording is permitted.

Lastly, this API is currently only shipping for CrOS, where the user is already prompted to consent to this capture before they log into the system[*]. Shipping for non-CrOS platforms will naturally be harder, and that work is not currently on anyone's OKRs.

Wdyt?

---
[*] As Matthew and Simon tell me. I have not checked.

[Artur Janc]

On Thu, Jun 15, 2023 at 3:52 PM Elad Alon <elad...@google.com> wrote:

Hi Arthur!

Using the pre-existing getDisplayMedia(), an application can already gain access to the user's screen and capture content in pop-ups, detect browsing history, etc. - provided the user chooses to capture their (often only) screen. All of the concerns above apply for gDM. Further, any website can call gDM.

The "provided the user chooses to capture their screen" part is what does the heavy lifting here. Each time gDM is called the user has to explicitly consent to sharing their screen and decide what to share; as such, even an XSS in an application which calls gDM generally isn't enough to read the contents of the user's screen because the user has to interact with the UI that will enable screen sharing.

This feature doesn't have an equivalent protection and permits the allowlisted origin to call the API at any time without obtaining consent from the user, making it possible to abuse the privileges of the allowlisted origin if an attacker finds an XSS bug in it.

 

In contrast, getAllScreensMedia() can only be called by websites expressly allowlisted by the device's owner. This configuration would typically be carried out by an admin who is more tech-savvy than the average user, and it would usually be the admin's job to apply some good judgement with respect to security risks their organization might face.

It is expected that a reasonable company would only allowlist:
1. Its own applications.
2. Applications from providers who were contracted specifically for this purpose, and who are trusted by the contracting company to such an extent, that long-term full-screen recording is permitted.

It's not at all clear that domain admins who want to monitor their users' activity are familiar with the web security posture of all the origins they want to allowlist. Even if the scope of the allowlist is just "the company's own applications", there is a substantial risk that an XSS bug in any of the applications (which could previously only access the user's data in that single application) will now be granted the privileges to access the user's data on all other websites. 

Lastly, this API is currently only shipping for CrOS, where the user is already prompted to consent to this capture before they log into the system[*]. Shipping for non-CrOS platforms will naturally be harder, and that work is not currently on anyone's OKRs.

In this case the entity calling the API is an attacker with a vulnerability in any of the origins allowlisted in the Enterprise Policy. I don't think the fact that a user has clicked through a dialog when logging into the device serves as any protection for the user in this case because it's the attacker that is abusing the persisted privileges of the allowlisted origins to read the user's data, rather than the application that the domain admin has added to their allowlist.

[Simon Hangl]

| Lastly, this API is currently only shipping for CrOS, where the user is already prompted to consent to this capture before they log into the system[*].

One correction - due to technical limitations, the user is NOT warned on the login screen that if they proceed they might be recorded. This notification is only shown (immediately) after login. As mentioned, it is a notification that shows up even before recording starts, letting the user know that they might be recorded at any moment.

[Simon Hangl]

Hi everybody,

I'd like to update the OT milestones mentioned above to an OT of 6 months duration [1]:

OriginTrial desktop last: 124

OriginTrial desktop first: 118

[1] https://www.chromium.org/blink/launching-features/#origin-trials

[Mike West]

6 milestones is consistent with our OT process' recommendations. Still LGTM. Good luck with the experimentation.

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0e6f204e-dfda-4f98-9df5-2531364f79aen%40chromium.org.