Intent to Experiment: Element Capture

[Elad Alon]

Contact emails

elad...@chromium.org

Explainer

https://github.com/screen-share/element-capture/blob/main/README.md

Specification

https://screen-share.github.io/element-capture

Summary

API for capturing a subtree of the DOM.

Given a video MediaStreamTrack obtained through pre-existing means to initiate tab-capture, Element Capture allows mutating the track to only capture a subtree of the DOM starting at a given Element.

The API bears some resemblance to the Region Capture API, but affords greater flexibility for applications, because occluding and occluded content are both excluded from the capture.

Blink component

Blink>GetDisplayMedia

TAG review

We are holding off on the request for a TAG review until we get some more developer feedback about the API shape.

TAG review status

Pending

Risks

Interoperability and Compatibility

Gecko: Under consideration (https://github.com/mozilla/standards-positions/issues/857)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/280)

Web developers:

Positive See upvotes and comments on the following:

• https://github.com/WICG/proposals/issues/73
• https://twitter.com/quicksave2k/status/1583388663597015042

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Goals for experimentation

* Solicit more informed Web developer feedback to validate the API shape.

* Ensure that the feature works correctly in conjunction with adjacent features.

Debuggability

No changes to DevTools are intended.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

This API is supported on all desktop platforms. Mobile platforms are unsupported because screen-capture itself is unsupported on those platforms.

Is this feature fully tested by web-platform-tests?

Not yet (but we’re working on extending coverage)

Flag name on chrome://flags

element-capture

Finch feature name

ElementCapture

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1418194

Launch bug

https://launch.corp.google.com/launch/4240472

Estimated milestones

Shipping on desktop

121

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5198989277790208

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDO6y5b6y3q9QEd2scsYPWuWLJBnPLgwm%2BaHpKx36CYMwA%40mail.gmail.com

This intent message was generated by Chrome Platform Status.

[Mike Taylor]

Hi Elad,

Can you clarify which milestones you intend to experiment in?

thanks,
Mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDN8mcO%2BYqaVA5nb5BBv-dZB0wqwfh9580wMc-e%2BNuP7yw%40mail.gmail.com.

[Elad Alon]

Hi Mike. I'm asking for m121 to m126 (inclusive).

[Mike Taylor]

Thanks, LGTM to experiment from 121 to 126 inclusive.

[Jan-Ivar Bruaroey]

The Gecko position has been updated to negative. See https://github.com/mozilla/standards-positions/issues/857

[Elad Alon]

Thank you for updating, Jan-Ivar. I have now updated this information in the ChromeStatus entry. I am looking forward to carry on the discussion about your position in your standards-positions GitHub repo, in the Element Capture spec repo, in the Screen Capture CG and in the WebRTC WG.

[Elad Alon]

Hello Blink owners,

We are asking for a breaking period of 3 weeks to this API, followed by a renewed experiment for the traditional 6 milestones (2024-May-14 to 2024-Jun-04). A request that seems similar to me was granted in January 2022 for another API, and later shipped successfully with consensus with Mozilla and Apple.

At the time of writing, we have public support for this API from such companies as Zoom, Jitsi, Mux, DialPad, Whereby, Intel and Tella. Tango’s co-founder wrote “I can't emphasize enough how instrumental this specification would be for our product and user experience.” However, none of them have signed up for the OT as of yet. 

From the OT perspective:

This API allows Web developers to build novel new features; however, it requires a non-trivial investment. We are hoping that after giving more time, we will see more participation.

From the standardization perspective:

We need time to pick up discussions with Mozilla and Apple again. As Chrome’s Security and Privacy experts do not share the concerns Mozilla and Apple have previously voiced, it stands to reason that additional discussions will allow us to converge - and we will prioritize these discussions now.

Additionally, some possibilities remain for API changes that could perhaps allow for a compromise, mostly around cross-origin isolation. (Full disclosure - this is not my ideal outcome, but it’s a fallback possibility worth mentioning.)

Progress made:

• Spec: The spec has evolved and is now more mature, dealing better with such edge cases as loss of “eligibility for restriction.”

• TAG: Previously we held off on the request for a TAG review until we got some more developer feedback about the API shape. Having received this initial feedback, the TAG request has now been submitted.

• Signals: Signals have been requested. Mozilla responded. We intend to prioritize this discussion with them now.

• Feedback: Outreach for feedback from the spec community - multiple issues were filed on the spec by Web developers.

• WPT: Coverage has recently been extended.

Reasons to run a new trial:

Gain additional feedback from new participants. Examples for remaining areas where new feedback could help include:

• Uncover new edge cases which were not uncovered by the spec authors and reviewers, implementers and current OT participants. The current edge cases here demonstrate how non-obvious these may be.

• Validate (or refute) the assumptions we have made about the viability of an MVP that is missing some functionality. Examples:

  • Events notifying apps when an element stops/starts being “eligible for restriction”.

  • Mechanism to force elements into an “eligible for restriction” state.

• Encourage future OT participation by Web developers, by demonstrating that risks associated with relying on an origin trial, while real, are partially offset by a commitment to keep the OT going until discussions conclude.

Thanks,

Elad

[Mike Taylor]

Note: Because you're asking for a renewal of 6 milestones, this will require 3 LGTMs.

Can you clarify which milestones you're asking for? You've written "(2024-May-14 to 2024-Jun-04)" - that range maps to M125 and M126 stable releases.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/073738e4-cb2f-4a1a-9299-cf134f971a67n%40chromium.org.

[Elad Alon]

I'm asking for a break-off period starting July 1 and ending July 22 (inclusive). That means that the original trial will end on June 30, and be restarted on July 23. This corresponds to breaking the original experiment early during the cycle of what would have otherwise been its final milestone (m126), the restarting the experiment over the m127-m132 range. (CC Panos for feasibility.)

[Mike Taylor]

OK - thanks for the clarification. I would ask that you get in touch with every registered origin - expiring the experiment early may be surprising.

LGTM1 to experiment from M127 to M132 inclusive.

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/45e6f481-db05-48d4-8da9-92b5b05e6447%40chromium.org.

[Yoav Weiss (@Shopify)]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_z5FbVXB-8OLe0O_q7nV9qgaOsrCyio1HcuWCVD9v8CQ%40mail.gmail.com.