Intent to Experiment: Element Capture

[Elad Alon]

Contact emails

elad...@chromium.org

Explainer

https://github.com/screen-share/element-capture/blob/main/README.md

Specification

https://screen-share.github.io/element-capture

Summary

API for capturing a subtree of the DOM.

Given a video MediaStreamTrack obtained through pre-existing means to initiate tab-capture, Element Capture allows mutating the track to only capture a subtree of the DOM starting at a given Element.

The API bears some resemblance to the Region Capture API, but affords greater flexibility for applications, because occluding and occluded content are both excluded from the capture.

Blink component

Blink>GetDisplayMedia

TAG review

We are holding off on the request for a TAG review until we get some more developer feedback about the API shape.

TAG review status

Pending

Risks

Interoperability and Compatibility

Gecko: Under consideration (https://github.com/mozilla/standards-positions/issues/857)

WebKit: No signal (https://github.com/WebKit/standards-positions/issues/280)

Web developers:

Positive See upvotes and comments on the following:

• https://github.com/WICG/proposals/issues/73
• https://twitter.com/quicksave2k/status/1583388663597015042

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Goals for experimentation

* Solicit more informed Web developer feedback to validate the API shape.

* Ensure that the feature works correctly in conjunction with adjacent features.

Debuggability

No changes to DevTools are intended.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

No

This API is supported on all desktop platforms. Mobile platforms are unsupported because screen-capture itself is unsupported on those platforms.

Is this feature fully tested by web-platform-tests?

Not yet (but we’re working on extending coverage)

Flag name on chrome://flags

element-capture

Finch feature name

ElementCapture

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1418194

Launch bug

https://launch.corp.google.com/launch/4240472

Estimated milestones

Shipping on desktop

121

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5198989277790208

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDO6y5b6y3q9QEd2scsYPWuWLJBnPLgwm%2BaHpKx36CYMwA%40mail.gmail.com

This intent message was generated by Chrome Platform Status.

[Mike Taylor]

Hi Elad,

Can you clarify which milestones you intend to experiment in?

thanks,
Mike

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMO6jDN8mcO%2BYqaVA5nb5BBv-dZB0wqwfh9580wMc-e%2BNuP7yw%40mail.gmail.com.

[Elad Alon]

Hi Mike. I'm asking for m121 to m126 (inclusive).

[Mike Taylor]

Thanks, LGTM to experiment from 121 to 126 inclusive.

[Jan-Ivar Bruaroey]

The Gecko position has been updated to negative. See https://github.com/mozilla/standards-positions/issues/857

[Elad Alon]

Thank you for updating, Jan-Ivar. I have now updated this information in the ChromeStatus entry. I am looking forward to carry on the discussion about your position in your standards-positions GitHub repo, in the Element Capture spec repo, in the Screen Capture CG and in the WebRTC WG.

[Elad Alon]

Hello Blink owners,

We are asking for a breaking period of 3 weeks to this API, followed by a renewed experiment for the traditional 6 milestones (2024-May-14 to 2024-Jun-04). A request that seems similar to me was granted in January 2022 for another API, and later shipped successfully with consensus with Mozilla and Apple.

At the time of writing, we have public support for this API from such companies as Zoom, Jitsi, Mux, DialPad, Whereby, Intel and Tella. Tango’s co-founder wrote “I can't emphasize enough how instrumental this specification would be for our product and user experience.” However, none of them have signed up for the OT as of yet. 

From the OT perspective:

This API allows Web developers to build novel new features; however, it requires a non-trivial investment. We are hoping that after giving more time, we will see more participation.

From the standardization perspective:

We need time to pick up discussions with Mozilla and Apple again. As Chrome’s Security and Privacy experts do not share the concerns Mozilla and Apple have previously voiced, it stands to reason that additional discussions will allow us to converge - and we will prioritize these discussions now.

Additionally, some possibilities remain for API changes that could perhaps allow for a compromise, mostly around cross-origin isolation. (Full disclosure - this is not my ideal outcome, but it’s a fallback possibility worth mentioning.)

Progress made:

• Spec: The spec has evolved and is now more mature, dealing better with such edge cases as loss of “eligibility for restriction.”

• TAG: Previously we held off on the request for a TAG review until we got some more developer feedback about the API shape. Having received this initial feedback, the TAG request has now been submitted.

• Signals: Signals have been requested. Mozilla responded. We intend to prioritize this discussion with them now.

• Feedback: Outreach for feedback from the spec community - multiple issues were filed on the spec by Web developers.

• WPT: Coverage has recently been extended.

Reasons to run a new trial:

Gain additional feedback from new participants. Examples for remaining areas where new feedback could help include:

• Uncover new edge cases which were not uncovered by the spec authors and reviewers, implementers and current OT participants. The current edge cases here demonstrate how non-obvious these may be.

• Validate (or refute) the assumptions we have made about the viability of an MVP that is missing some functionality. Examples:

  • Events notifying apps when an element stops/starts being “eligible for restriction”.

  • Mechanism to force elements into an “eligible for restriction” state.

• Encourage future OT participation by Web developers, by demonstrating that risks associated with relying on an origin trial, while real, are partially offset by a commitment to keep the OT going until discussions conclude.

Thanks,

Elad

[Mike Taylor]

Note: Because you're asking for a renewal of 6 milestones, this will require 3 LGTMs.

Can you clarify which milestones you're asking for? You've written "(2024-May-14 to 2024-Jun-04)" - that range maps to M125 and M126 stable releases.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/073738e4-cb2f-4a1a-9299-cf134f971a67n%40chromium.org.

[Elad Alon]

I'm asking for a break-off period starting July 1 and ending July 22 (inclusive). That means that the original trial will end on June 30, and be restarted on July 23. This corresponds to breaking the original experiment early during the cycle of what would have otherwise been its final milestone (m126), the restarting the experiment over the m127-m132 range. (CC Panos for feasibility.)

[Mike Taylor]

OK - thanks for the clarification. I would ask that you get in touch with every registered origin - expiring the experiment early may be surprising.

LGTM1 to experiment from M127 to M132 inclusive.

[Chris Harrelson]

LGTM2

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/45e6f481-db05-48d4-8da9-92b5b05e6447%40chromium.org.

[Yoav Weiss (@Shopify)]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_z5FbVXB-8OLe0O_q7nV9qgaOsrCyio1HcuWCVD9v8CQ%40mail.gmail.com.

[Elad Alon]

Thank you for the LGTMs.

A quick update that we have discussed the matter with the OT Core team, and it appears to us all better to have the breaking period between June 24 (first day of break) and July 15 (first day of renewed experiment). This ensures that partners have time to mint a new token and roll it out to their production enviornment ahead of m127 shipping to Stable.

[Danny Moerkerke]

I registered a token for the origin trial but it doesn't seem to work. Is it not active yet?

Demo: https://whatpwacando.today/element-capture

Op dinsdag 21 mei 2024 om 12:05:58 UTC+2 schreef Elad Alon:

[François Beaufort]

Note that I can also reproduce this issue with https://element-capture-demo.glitch.me/

@origin-tri...@google.com Can you help us fix this?

My understanding was that the breaking period was between June 24 and July 15. The issue happens for both old and new tokens.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9cfa01b4-a346-4b13-8d36-35b03d7875ben%40chromium.org.

[François Beaufort]

I see "Origin Trials - Version: 1.0.0.17" in chrome://components.
@James Scott Can you also reproduce on your side by going to https://element-capture-demo.glitch.me/?

On Mon, Jul 22, 2024 at 4:04 PM James Scott <james...@google.com> wrote:

Hi all,

The latest configuration (v1.0.0.17) to re-enable the feature went out last week. Could you go to chrome://components/ and check the version of the Origin Trials component and report back what you see?

Users will need to restart their browser after getting the latest version.

Best,

James

[James Scott]

Hi François,

On my Chromebook, the feature is enabled at https://element-capture-demo.glitch.me/. However, it only started working after I restarted my Chromebook following the 1.0.0.17 update.  The trial is also active on my Linux machine. Similarly, I had to run the `killall chrome` command in my Linux terminal to get it to work.

I have attached two screenshots.

Please let me know if you are still experiencing issues, and if I can be of any further assistance.

Best,

James

 

[James Scott]

Hi all,

The latest configuration (v1.0.0.17) to re-enable the feature went out last week. Could you go to chrome://components/ and check the version of the Origin Trials component and report back what you see?

Users will need to restart their browser after getting the latest version.

Best,

James

On Mon, Jul 22, 2024 at 2:47 AM François Beaufort <fbea...@google.com> wrote:

[Danny Moerkerke]

Thanks James, that works! 

So users should just restart their browser once they have v1.0.0.17 of the component?

Op maandag 22 juli 2024 om 17:56:54 UTC+3 schreef James Scott:

[James Scott]

Hi Danny,

Happy to help! Yes, restarting the browser after updating to v1.0.0.17 is the best way to make sure the previous version's configuration is cleared and the component's new settings take effect immediately.

Best,

James