Intent to Extend Experiment: Captured Surface Control

[Chromestatus]

Contact emails

elad...@chromium.org, gui...@chromium.org

Explainer

https://github.com/screen-share/captured-surface-control/blob/main/README.md

Specification

https://screen-share.github.io/captured-surface-control

Design docs

https://docs.google.com/document/d/10UojDvTJ6ojBEOP7cgBIIaE7WZEfes_Qv1eN3A2A0nM/edit?usp=sharing

Summary

Web API that allows Web applications to: 1. Produce wheel events in a captured tab or window. 2. Read/write the zoom level of a captured tab.

Blink component

Blink>GetDisplayMedia

TAG review

https://github.com/w3ctag/design-reviews/issues/962

TAG review status

Pending

Chromium Trial Name

CapturedSurfaceControl

Origin Trial documentation link

https://github.com/screen-share/captured-surface-control/blob/main/README.md

WebFeature UseCounter name

kCapturedSurfaceControl

Risks

Interoperability and Compatibility

Gecko: No signal

WebKit: No signal

Web developers: No signals

Other signals:

Security

https://github.com/screen-share/captured-surface-control?tab=readme-ov-file#security-and-privacy-concerns

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Goals for experimentation

Reason this experiment is being extended

The status in the W3C WebRTC WG is that we have consensus on supporting the use case of forwarding zoom and wheel events from the local user to the captured tab. The API currently under trial supports this use case, but the WG does not want support use cases with non-local control (e.g., co-presenting), which the current API does support. See https://www.w3.org/2024/05/21-webrtc-minutes.html#t01 We want to experiment with an alternative API shape to prevent non-local use cases, and with additional restrictions on the existing API shape. These changes will be discussed in upcoming WG meetings.

Reason this experiment is being extended

The experiment was shifted from the original m122-m127 to m124-m129. See https://groups.google.com/a/chromium.org/g/blink-dev/c/7MbIEBdHMck for details.

Ongoing technical constraints

Debuggability

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

No

Supported on all desktop platforms. (Screen-sharing is not currently supported on mobile platforms.)

Is this feature fully tested by web-platform-tests?

No

Flag name on chrome://flags

captured-surface-control

Finch feature name

CapturedDisplaySurface

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1466247

Launch bug

https://launch.corp.google.com/launch/4268170

Estimated milestones

Origin trial desktop first	122
Origin trial desktop last	127
Origin trial extension 1 end milestone	132
Origin trial extension 2 end milestone	129
DevTrial on desktop	122

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5092615678066688?gate=5192731870363648

Links to previous Intent discussions

Intent to Prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDPSgR3kX39drHd9t-JvTKBk%2B7Dg03O6dvowzw-LjQ__1A%40mail.gmail.com
Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-DF_BCtdwiaYqrXsan3YNBfT2pAeboT0f1F9wmViPPWQ%40mail.gmail
Intent to Extend Experiment 2: https://groups.google.com/a/chromium.org/g/blink-dev/c/7MbIEBdHMck/m/UBbWTX-hAgAJ

This intent message was generated by Chrome Platform Status.

[Guido Urdaneta]

Just to clarify, since the automatically generated email from chromestatus might not be 100% clear: The request is to extend the OT to M132 

The email says the milestone for extension 1 is M132 and for extension 2 is M129, but it should be the other way around.

There was a first "extension" that was not really an extension, but a shift in dates since the OT actually started on M124 (and is scheduled to end in M129). Now we want to extend the OT to M132 to experiment with changes that try to address feedback from the WG. 

[Domenic Denicola]

Thanks for the clarification.

Per the requirements for extending an origin trial, I see progress on:

• Draft spec, maybe? Although there have been no updates for 2 months so I'm unsure if it incorporates the changes you're mentioning.
• Outreach for feedback from the spec community (your link to https://www.w3.org/2024/05/21-webrtc-minutes.html#t01)

I see no progress on:

• TAG review (but this is out of your control)
• https://bit.ly/blink-signals requests (it seems like no signals were filed, and no web developers want to use this feature per your "Web developers: No signals")
• WPT tests

Can you work on making progress on gathering signals, writing WPTs, and also clarify whether the spec has been updated to the new design?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxYGAzeOq%3D1OMvL60%2Bu54no%3D6A7gKGb%3DQ4xLmQtbNVTEDw%40mail.gmail.com.

[Guido Urdaneta]

On Fri, Aug 16, 2024 at 8:08 AM Domenic Denicola <dom...@chromium.org> wrote:

Thanks for the clarification.

Per the requirements for extending an origin trial, I see progress on:

• Draft spec, maybe? Although there have been no updates for 2 months so I'm unsure if it incorporates the changes you're mentioning.

• Outreach for feedback from the spec community (your link to https://www.w3.org/2024/05/21-webrtc-minutes.html#t01)

I see no progress on:

• TAG review (but this is out of your control)

A TAG review was requested a couple of months ago. We're still waiting for feedback. 

• https://bit.ly/blink-signals requests (it seems like no signals were filed, and no web developers want to use this feature per your "Web developers: No signals")

We haven't formally requested signals from other browsers because we're still discussing in the WG.

However, an initial signal is that both Mozilla and Apple are supportive of the main use case, so it is likely that we will eventually reach consensus on an API surface for that use case. 

Wrt Web Developers, we haven't contacted OT participants for formal feedback, but here are some positive signals posted on the Web:

• Dialpad: https://github.com/screen-share/captured-surface-control/issues/10. They state that they have successfully used the current version of the API and are requesting some extensions.
• whatpwacando.today: https://x.com/dannymoerkerke/status/1810429064286498859?t=gKxk7pv0QAmV_Z3uXzB5xA&s=19 and https://whatpwacando.today/capture-handle/
  
• Google Meet has also successfully experimented with the API. https://support.google.com/meet/answer/14808197 

• WPT tests

We plan to add these once we have a clearer idea of what the final shape of the API surface will be. This is likely to happen after the August 27 WG Meeting, where this API will be discussed. 

 

Can you work on making progress on gathering signals, writing WPTs, and also clarify whether the spec has been updated to the new design?

The current version of the spec reflects what is currently shipping in the OT. We'll update it once 1) the new additions are ready for inclusion in the OT and 2) we discuss them in the WG meeting later this month.

Thanks,

GU

[Domenic Denicola]

Thanks for the responses.

Per our process, I'm not comfortable approving the OT for extension until formal position requests are filed, and there is at least a spec PR ready to match what you plan to include in the extended Origin Trial.

(Or, if I've misunderstood and you do not plan to change the API shape during the OT, then the spec change is not necessary.)

[Guido Urdaneta]

On Mon, Aug 19, 2024 at 3:32 AM Domenic Denicola <dom...@chromium.org> wrote:

Thanks for the responses.

Per our process, I'm not comfortable approving the OT for extension until formal position requests are filed, and there is at least a spec PR ready to match what you plan to include in the extended Origin Trial.

Here are are the position requests:

• https://github.com/mozilla/standards-positions/issues/1061
• https://github.com/WebKit/standards-positions/issues/388

Here is the spec:

• PR with the new changes: https://github.com/screen-share/captured-surface-control/pull/13/files
• Updated spec including the changes: https://screen-share.github.io/captured-surface-control/

The new changes just landed in Chromium as well.

Please let me know if there are any other issues blocking the extension of the OT.

[Domenic Denicola]

Excellent, thank you! LGTM to extend the experiment.