Intent to Extend Experiment: Capture all screens

[Simon Hangl]

Hello blink-dev,

We’d like to ask for an extension to our Origin Trial, from M124 to M130. This is due to the dependency on isolated web apps.

Contact emails

sim...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Specification

https://screen-share.github.io/capture-all-screens

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/13el0NriAUpAzLUw96V7zQiMSjgH9zVaTXUHtuaq8-HI/edit?resourcekey=0-jRPpeLth1odq6M5iFLswig

Summary

Capture all the screens currently connected to the device using getAllScreensMedia(). Calling getDisplayMedia() multiple times requires multiple user gestures, burdens the user with choosing the next screen each time, and does not guarantee to the app that all the screens were selected. getAllScreensMedia() improves on all of these fronts. (As this feature has extreme privacy ramifications, it is only exposed behind an enterprise policy, and users are warned before recording even starts, that recording *could* start at some point.)

Blink component

Blink>GetDisplayMediaSet

TAG review

https://github.com/w3ctag/design-reviews/issues/856

TAG review status

Complete

Chromium Trial Name

GetAllScreensMedia

Link to origin trial feedback summary

https://github.com/screen-share/capture-all-screens/issues

Origin Trial documentation link

https://github.com/screen-share/capture-all-screens

Risks

Interoperability and Compatibility

This API is only available to origins allowlisted by administrators through a policy. The policy itself is non-standard, limiting even theoretical interoperability.This API rejects requests from pages that are not allow-listed through an administrator. The likelihood of this API being adopted by a browser that does not provide administrators mechanisms to manage clients is low.

Gecko: N/A

WebKit: N/A

Web developers: Positive (https://github.com/screen-share/capture-all-screens/issues/9)

Other signals:

Ergonomics

No

Activation

The challenge for developers is the limitation of the API to origins allowlisted by an enterprise policy.

Security

1. Risk of malicious sites exploiting the API and gaining access to sensitive information on users' devices. This risk is mitigated by the API only being accessible to origins allowlisted by an enterprise policy.

2. Risk of users loading private information that gets recorded and made available to apps affiliated with their device's admin. This risk is mitigated by informing users that recording might start at any moment before the API becomes accessible. (In CrOS, this warning is delivered in the log-in screen, and when users log-in despite the warning, this is tantamount to assent.)

3. Risk of users forgetting that their screens are being recorded. This risk is mitigated through a persistent notification.

Goals for experimentation

Learn about the experience of web developers and how this API fulfills their needs.

Reason this experiment is being extended

This API will eventually be released for isolated contexts, which are delayed. Hence, we are asking for an extension of the origin trial.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

This API is initially implemented on CrOS, where demand for it is greatest, and where we have the most flexibility in offering users early warning that their screens may be recorded if they proceed past the log-in screen. Lessons learned from shipping this API on CrOS will be used when deciding how to correctly implement such warnings on other platforms.

Is this feature fully tested by web-platform-tests?

No, as WPTs don’t support setting of managed policies. The API is tested by a number of unit- and browser- tests (Test files).

DevTrial instructions

https://github.com/screen-share/capture-all-screens/blob/main/HOWTO.md

Flag name on chrome://flags

enable-get-all-screens-media

Finch feature name

None

Non-finch justification

None

Requires code in //chrome?

True

Tracking bug

https://issues.chromium.org/issues/40216442

Launch bug

https://launch.corp.google.com/launch/4201060

Estimated milestones

Origin trial desktop first	118
Origin trial desktop last	124
Origin trial extension 1 end milestone	130
DevTrial on desktop	116

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6284029979525120

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEdDZo9N354i6eST0x19TXwpeBtgs5_gJUYVF%2BTKLpiJySDADg%40mail.gmail.com

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/6TRT0XsVOE4/m/NOm-YEQCAgAJ

Simon Hangl

Software Engineer

sim...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Yoav Weiss (@Shopify)]

On Mon, Mar 18, 2024 at 5:28 PM 'Simon Hangl' via blink-dev <blin...@chromium.org> wrote:

Hello blink-dev,

We’d like to ask for an extension to our Origin Trial, from M124 to M130. This is due to the dependency on isolated web apps.

Contact emails

sim...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Specification

https://screen-share.github.io/capture-all-screens

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/13el0NriAUpAzLUw96V7zQiMSjgH9zVaTXUHtuaq8-HI/edit?resourcekey=0-jRPpeLth1odq6M5iFLswig

That's a Google-only doc. Is there a public variant?

 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP0TkgE%2B58Z59SF%3Dpbvrbx8ovaUFC8V0uki7bovVSKg6GAbxOg%40mail.gmail.com.

[Vladimir Levin]

On Mon, Mar 18, 2024 at 11:17 AM 'Simon Hangl' via blink-dev <blin...@chromium.org> wrote:

Hello blink-dev,

We’d like to ask for an extension to our Origin Trial, from M124 to M130. This is due to a dependency on isolated web apps, which are delayed.

The intent process only allows extensions of 3 milestones at a time. It also requires evidence of substantial progress on the feature. It sounds like here, the original experiment did not go as planned due to a dependency. Do you know if the isolated web apps feature is ready now? In other words, is this dependency satisfied? 

Contact emails

sim...@google.com

Explainer

https://github.com/screen-share/capture-all-screens/blob/main/README.md

Specification

https://screen-share.github.io/capture-all-screens

Design docs

https://screen-share.github.io/capture-all-screens

https://github.com/screen-share/capture-all-screens/blob/main/README.md

https://docs.google.com/document/d/13el0NriAUpAzLUw96V7zQiMSjgH9zVaTXUHtuaq8-HI/edit?resourcekey=0-jRPpeLth1odq6M5iFLswig

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAP0TkgF1BfhsLRadATibKed4vQUoV8_PqA_xUUZdXSSFcGZW%2Bw%40mail.gmail.com.

[Simon Hangl]

Apologies for the delay. We'd like to ask for an extension of the origin trial from M129 to M132.

@Yoav, I made the design doc available for all chromium accounts here.

@Vladimir, we are on track with isolated web apps and an intent to ship will be submitted in the next milestones.

[Daniel Bratell]

Any reason to not make it available for everyone? Asking for a friend.

Another thing, when extending experiments we want to see evidence of substantial progress on the feature so that it doesn't just roll along until it's burned in by pure inertia. Could you please tell us about the progress since the last extension?

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/dad681d8-8adb-4530-bf59-3604c8bc5047n%40chromium.org.

[Simon Hangl]

@Daniel, thanks for your questions / comments. We intend to make getAllScreensMedia available for everybody once isolated web apps launch (we are asking to extend the origin trial to already gain insights on the API before isolated web apps launch - see also the "Short term solution until IWAs are available" section in the design doc). This also brings me to the 2nd part of your question: we made significant progress towards isolated web apps (we are mostly code complete and the intent to launch will be submitted within the next 1-3 milestones).

[Simon Hangl]

Oops, upon friendly clarification from a colleague I realized that your comment was probably about making the doc visible to everyone :) . I updated the doc permissions now.

[Yoav Weiss (@Shopify)]

A few things trouble me here.

• Dependency injection
  • The initial intent indicated dependency on Enterprise Policy, rather than IWAs.
  • I see some reasoning for the new dependency in the design doc's security considerations, but it seems incomplete
    • e.g. why couldn't you enforce CSP and TrustedTypes as a requirement for this regardless of IWA? How does bundling help when allowing one app to leak information from others? Wasn't there controls in place limiting the origins that can do that as part of the Enterprise Policy?
    •  I may be missing context as a lot of the links in that doc are still Google-only
• Timelines
  
  • The initial trial went from 118 to 124.
  • On this thread I see you started by asking for an extension from 124 to 130, and then switched to asking for 129 to 132.
  • At the same time, I don't believe the OT was put on hold when 124 was released. 
  • What happened between M124 and M128?
• Progress towards shipping
  • On top of that, no evidence of substantial progress towards shipping was demonstrated. Again, the design doc still contains many Google-only links, so I may be missing context here, but this section feels very much like a soft launch. The Origin Trial risks we are trying to avoid don't seem to have been carefully considered.

Putting all this together, I don't think we should renew the current trial.

[Reilly Grant]

CSP and Trusted Types give you protections against XSS but only the bundling provided by IWAs provides the protection against server compromise that Chrome Security is asking for for this API.

Shipping this API in its final form has been blocked on IWAs being ready to launch (which is imminent).

Reilly Grant | Software Engineer | rei...@chromium.org | Google Chrome

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6aee109d-77a7-4a01-b4d9-3fcbb4e06b36n%40chromium.org.

[Simon Hangl]

Thanks for your response, Yoav. Please find my answers to your questions below:

ad 1) “Why not CSP + trusted types instead of IWAs”: We discussed this with Artur, who initially flagged the vulnerability here. We do enforce these requirements. The API is only exposed in contexts that meet certain requirements on client-side XSS mitigation. These are necessary but not sufficient, as server-side XSS remains a meaningful risk in the absence of the packaging/signing guarantees of IWAs. We're managing that risk during this experimental period via Enterprise policy requirements on the one hand, and OT registration on the other.

ad 2) “What happened between M124 and M128”: We did clarify with the Blink owners whether we could extend the origin trial until M136, to ensure partners can already work with the API followed by the transition to IWAs. The origin trial accidentally was created longer than the formal 6 milestones (see discussions here), which I realized after I applied for extension on this thread. While we did clarify whether we can extend the origin trial with the timelines above, I sincerely apologize for not following the formal process.

ad 3) “Progress towards shipping”: We acknowledge that with our approach we went beyond the intent of an origin trial. We did however check in advance with the Blink owners whether we could follow this approach due to exceptional circumstances in order to

• get this API into the hands of selected web developers and

• timebox the temporary solution through origin trial to make sure this API does not remain on the drive-by web.

The origin trial is essential to keep current developer momentum and grant enough time for the selected developers to prepare the API launch in context of IWA. Good evidence for progress towards shipping can be seen by the multitude of IWA related work to prepare the upcoming launch (IWA Launch).

I hope to have answered your questions sufficiently. Please let me know if you have any further concerns or follow-up questions.

[Vladimir Levin]

As you mentioned, this is not the intended use of OTs. However, since the feature needs IWAs to ship, and IWAs are well on track towards shipping, I'm inclined to allow this extension. That being said, I am hopeful that this will be the last extension to the feature.

Due to the nature of this situation, this intent will require 3 LGTMs.

LGTM1 to extend to M131 inclusive (please correct me if that's not the intended target).

Thanks,

Vlad

[Daniel Bratell]

LGTM2

/Daniel

[Simon Hangl]

Thank you for the quick response. Just one correction: the agreement contained an extension until (including) M136 (see my reponse above).

[Vladimir Levin]

On Wed, Jul 17, 2024 at 1:28 PM 'Simon Hangl' via blink-dev <blin...@chromium.org> wrote:

Thank you for the quick response. Just one correction: the agreement contained an extension until (including) M136 (see my reponse above).

For now, my LGTM only applies up to and including M131. If more time is needed, you can come back for another extension with evidence of substantial progress (e.g. shipping the feature on top of IWA).

Thanks,

Vlad

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e564e02-03a0-463d-8e7b-ed622d12cb3dn%40chromium.org.

[Yoav Weiss (@Shopify)]

LGTM3 to extend experimentation up to and including M131, but no further. Like Vlad, I'd like to see clear evidence of significant progress before considering further extensions.

I'm very unhappy with the way this OT was managed. I'm LGTMing as I don't want your partners to bear the costs, but I want to emphasize that my LGTM does not legitimize the following:

• Creating production reliance on an API while it's in OT.
• Doing the above while it's clear that it is not a safe API to ship.
• This is partially on us, the API owners. Artur's comment on that thread should have triggered us to un-LGTM that initial intent, and should have resulted in the original OT never landing. That would have avoided the mess we're in now.
• Multiple milestones where the OT ran without any public approval from the API owners.

Beyond that, it's still not 100% clear to me why IWAs negate Artur's concerns, but I guess that's something we'd discuss on the intent to ship for this feature under IWAs.

[Drew Wilson]

On Wednesday, July 17, 2024 at 8:39:44 PM UTC+2 Yoav Weiss (@Shopify) wrote:

LGTM3 to extend experimentation up to and including M131, but no further. Like Vlad, I'd like to see clear evidence of significant progress before considering further extensions.

I'm very unhappy with the way this OT was managed. I'm LGTMing as I don't want your partners to bear the costs, but I want to emphasize that my LGTM does not legitimize the following:

• Creating production reliance on an API while it's in OT.
• Doing the above while it's clear that it is not a safe API to ship.
  • This is partially on us, the API owners. Artur's comment on that thread should have triggered us to un-LGTM that initial intent, and should have resulted in the original OT never landing. That would have avoided the mess we're in now.
• Multiple milestones where the OT ran without any public approval from the API owners.

Beyond that, it's still not 100% clear to me why IWAs negate Artur's concerns, but I guess that's something we'd discuss on the intent to ship for this feature under IWAs.

If you're talking about his concerns about XSS attacks - IWAs are a defense against that (it's literally one of the core purposes of that technology).

 

LGTM2

/Daniel

TAG reviewhttps://github.com/w3ctag/design-reviews/issues/856

[Swetha Sivaram]

Just so we have a better understanding and make sure we're working towards it, Vlad, you mention that shipping the API on IWA would constitute evidence of substantial progress, to consider an extension of the OT beyond M131.

 

The focus for the team over the next few weeks is to get Isolated Web Apps and the API to as close to a shipped state (including Blink approvals) as possible. If an extension of the OT is required beyond this point, it is primarily due to the dependency that some partners have on ControlledFrame functionality (an IWA-equivalent of WebViews) being shipped, in order to migrate. We're working on determining the best path forward to get ControlledFrame functionality shipped with a reasonable time built in for Contact Center partners to migrate to IWAs before an additional OT extension expires.

Do you foresee any challenges in approving an extension of the OT to M136 if we reach this state by M131? 

- Swetha 

[Swetha Sivaram]

`Do you foresee any challenges in approving an extension of the OT to M136 if we reach this state by M131? `

To clarify further, by "this state" I mean having Isolated Web Apps and the getAllScreensMedia API shipped / close to being shipped while the path to shipping ControlledFrame is being worked out.

[Vladimir Levin]

Hi Swetha,

Shipping this API in IWAs would indeed show enough progress that allowing some extra time for partners to migrate seems reasonable. I was not aware of the ControlledFrame dependency. Is shipping this also within the team's scope?

Generally, if another extension is required, I'd like to get a good understanding of what is still missing at that time and what steps are being taken to ensure that all of the requirements for partner migration are met.

Thanks,

Vlad

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/677e8001-ce0a-4c23-91e7-364611b3bd72n%40chromium.org.

[Swetha Sivaram]

Hi Vlad

Thank you for clarifying the expectations for another extension. Yes, shipping Controlled Frame is within the broader team's scope. It is not a dependency for all customers but a preferred way forward for some of them. 

Regards

Swetha