Intent to Extend Experiment: Capture all screens

639 views
Skip to first unread message

Simon Hangl

unread,
Mar 18, 2024, 12:28:07 PMMar 18
to blin...@chromium.org

Hello blink-dev,


We’d like to ask for an extension to our Origin Trial, from M124 to M130. This is due to the dependency on isolated web apps.


Contact emails

sim...@google.com


Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md


Specification

https://screen-share.github.io/capture-all-screens


Design docs


https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/13el0NriAUpAzLUw96V7zQiMSjgH9zVaTXUHtuaq8-HI/edit?resourcekey=0-jRPpeLth1odq6M5iFLswig


Summary

Capture all the screens currently connected to the device using getAllScreensMedia(). Calling getDisplayMedia() multiple times requires multiple user gestures, burdens the user with choosing the next screen each time, and does not guarantee to the app that all the screens were selected. getAllScreensMedia() improves on all of these fronts. (As this feature has extreme privacy ramifications, it is only exposed behind an enterprise policy, and users are warned before recording even starts, that recording *could* start at some point.)



Blink component

Blink>GetDisplayMediaSet


TAG review

https://github.com/w3ctag/design-reviews/issues/856


TAG review status

Complete


Chromium Trial Name

GetAllScreensMedia


Link to origin trial feedback summary

https://github.com/screen-share/capture-all-screens/issues


Origin Trial documentation link

https://github.com/screen-share/capture-all-screens


Risks



Interoperability and Compatibility

This API is only available to origins allowlisted by administrators through a policy. The policy itself is non-standard, limiting even theoretical interoperability.This API rejects requests from pages that are not allow-listed through an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.



Gecko: N/A


WebKit: N/A


Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)


Other signals:


Ergonomics

No



Activation

The challenge for developers is the limitation of the API to origins allowlisted by an enterprise policy.



Security

1. Risk of malicious sites exploiting the API and gaining access to sensitive information on users' devices. This risk is mitigated by the API only being accessible to origins allowlisted by an enterprise policy.


2. Risk of users loading private information that gets recorded and made available to apps affiliated with their device's admin. This risk is mitigated by informing users that recording might start at any moment before the API becomes accessible. (In CrOS, this warning is delivered in the log-in screen, and when users log-in despite the warning, this is tantamount to assent.)

3. Risk of users forgetting that their screens are being recorded. This risk is mitigated through a persistent notification.



Goals for experimentation

Learn about the experience of web developers and how this API fulfills their needs.


Reason this experiment is being extended

This API will eventually be released for isolated contexts, which are delayed. Hence, we are asking for an extension of the origin trial.


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

This API is initially implemented on CrOS, where demand for it is greatest, and where we have the most flexibility in offering users early warning that their screens may be recorded if they proceed past the log-in screen. Lessons learned from shipping this API on CrOS will be used when deciding how to correctly implement such warnings on other platforms.


Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).


DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md


Flag name on chrome://flags

enable-get-all-screens-media


Finch feature name

None


Non-finch justification

None


Requires code in //chrome?

True


Tracking bug

https://issues.chromium.org/issues/40216442


Launch bug

https://launch.corp.google.com/launch/4201060


Estimated milestones

Origin trial desktop first

118

Origin trial desktop last

124

Origin trial extension 1 end milestone

130

DevTrial on desktop

116



Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120


Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/6TRT0XsVOE4/m/NOm-YEQCAgAJ


Simon Hangl

Software Engineer

sim...@google.com


Google Germany GmbH

Erika-Mann-Straße 33

80636 München


Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg


Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.


Yoav Weiss (@Shopify)

unread,
Mar 19, 2024, 12:30:36 PMMar 19
to Simon Hangl, blin...@chromium.org
That's a Google-only doc. Is there a public variant?
 
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP0TkgE%2B58Z59SF%3Dpbvrbx8ovaUFC8V0uki7bovVSKg6GAbxOg%40mail.gmail.com.

Vladimir Levin

unread,
Mar 21, 2024, 11:38:49 AMMar 21
to Simon Hangl, blin...@chromium.org
On Mon, Mar 18, 2024 at 11:17 AM 'Simon Hangl' via blink-dev <blin...@chromium.org> wrote:

Hello blink-dev,


We’d like to ask for an extension to our Origin Trial, from M124 to M130. This is due to a dependency on isolated web apps, which are delayed.


The intent process only allows extensions of 3 milestones at a time. It also requires evidence of substantial progress on the feature. It sounds like here, the original experiment did not go as planned due to a dependency. Do you know if the isolated web apps feature is ready now? In other words, is this dependency satisfied? 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Simon Hangl

unread,
Jun 19, 2024, 10:42:14 AMJun 19
to blink-dev, Vladimir Levin, blin...@chromium.org, Simon Hangl
Apologies for the delay. We'd like to ask for an extension of the origin trial from M129 to M132.

@Yoav, I made the design doc available for all chromium accounts here.

@Vladimir, we are on track with isolated web apps and an intent to ship will be submitted in the next milestones.

Daniel Bratell

unread,
Jun 25, 2024, 1:48:07 PMJun 25
to Simon Hangl, blink-dev, Vladimir Levin

Any reason to not make it available for everyone? Asking for a friend.

Another thing, when extending experiments we want to see evidence of substantial progress on the feature so that it doesn't just roll along until it's burned in by pure inertia. Could you please tell us about the progress since the last extension?

/Daniel

Simon Hangl

unread,
Jun 26, 2024, 4:43:35 AMJun 26
to blink-dev, Daniel Bratell, Vladimir Levin, Simon Hangl
@Daniel, thanks for your questions / comments. We intend to make getAllScreensMedia available for everybody once isolated web apps launch (we are asking to extend the origin trial to already gain insights on the API before isolated web apps launch - see also the "Short term solution until IWAs are available" section in the design doc). This also brings me to the 2nd part of your question: we made significant progress towards isolated web apps (we are mostly code complete and the intent to launch will be submitted within the next 1-3 milestones).

Simon Hangl

unread,
Jun 26, 2024, 12:18:45 PMJun 26
to blink-dev, Simon Hangl, Daniel Bratell, Vladimir Levin
Oops, upon friendly clarification from a colleague I realized that your comment was probably about making the doc visible to everyone :) . I updated the doc permissions now.

Yoav Weiss (@Shopify)

unread,
Jul 10, 2024, 12:58:46 PM (6 days ago) Jul 10
to blink-dev, Simon Hangl, Daniel Bratell, vmp...@google.com
A few things trouble me here.
  • Dependency injection
    • The initial intent indicated dependency on Enterprise Policy, rather than IWAs.
    • I see some reasoning for the new dependency in the design doc's security considerations, but it seems incomplete
      • e.g. why couldn't you enforce CSP and TrustedTypes as a requirement for this regardless of IWA? How does bundling help when allowing one app to leak information from others? Wasn't there controls in place limiting the origins that can do that as part of the Enterprise Policy?
      •  I may be missing context as a lot of the links in that doc are still Google-only
  • Timelines
    • The initial trial went from 118 to 124.
    • On this thread I see you started by asking for an extension from 124 to 130, and then switched to asking for 129 to 132.
    • At the same time, I don't believe the OT was put on hold when 124 was released. 
    • What happened between M124 and M128?
  • Progress towards shipping
    • On top of that, no evidence of substantial progress towards shipping was demonstrated. Again, the design doc still contains many Google-only links, so I may be missing context here, but this section feels very much like a soft launch. The Origin Trial risks we are trying to avoid don't seem to have been carefully considered.

Putting all this together, I don't think we should renew the current trial.



Reilly Grant

unread,
Jul 11, 2024, 1:45:24 PM (5 days ago) Jul 11
to Yoav Weiss (@Shopify), blink-dev, Simon Hangl, Daniel Bratell, vmp...@google.com
CSP and Trusted Types give you protections against XSS but only the bundling provided by IWAs provides the protection against server compromise that Chrome Security is asking for for this API.

Shipping this API in its final form has been blocked on IWAs being ready to launch (which is imminent).
Reilly Grant | Software Engineer | rei...@chromium.org | Google Chrome


Simon Hangl

unread,
Jul 12, 2024, 8:08:12 AM (4 days ago) Jul 12
to blink-dev, Reilly Grant, blink-dev, Simon Hangl, Daniel Bratell, vmp...@google.com, Yoav Weiss (@Shopify)

Thanks for your response, Yoav. Please find my answers to your questions below:


ad 1) “Why not CSP + trusted types instead of IWAs”: We discussed this with Artur, who initially flagged the vulnerability here. We do enforce these requirements. The API is only exposed in contexts that meet certain requirements on client-side XSS mitigation. These are necessary but not sufficient, as server-side XSS remains a meaningful risk in the absence of the packaging/signing guarantees of IWAs. We're managing that risk during this experimental period via Enterprise policy requirements on the one hand, and OT registration on the other.


ad 2) “What happened between M124 and M128”: We did clarify with the Blink owners whether we could extend the origin trial until M136, to ensure partners can already work with the API followed by the transition to IWAs. The origin trial accidentally was created longer than the formal 6 milestones (see discussions here), which I realized after I applied for extension on this thread. While we did clarify whether we can extend the origin trial with the timelines above, I sincerely apologize for not following the formal process.


ad 3) “Progress towards shipping”: We acknowledge that with our approach we went beyond the intent of an origin trial. We did however check in advance with the Blink owners whether we could follow this approach due to exceptional circumstances in order to

  • get this API into the hands of selected web developers and

  • timebox the temporary solution through origin trial to make sure this API does not remain on the drive-by web.

The origin trial is essential to keep current developer momentum and grant enough time for the selected developers to prepare the API launch in context of IWA. Good evidence for progress towards shipping can be seen by the multitude of IWA related work to prepare the upcoming launch (IWA Launch).


I hope to have answered your questions sufficiently. Please let me know if you have any further concerns or follow-up questions.

Reply all
Reply to author
Forward
0 new messages