Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

[Guido Urdaneta]

Contact emails

h...@chromium.org, gui...@chromium.org, agp...@chromium.org

Explainer

https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

Specification

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor

Summary

Allow WebRTC Encoded Transform API to manipulate audio and video frame metadata. Some WebRTC Encoded Transform use cases involve manipulation of not only the payload of encoded video / audio frames but also its metadata. For example, if a peer connection negotiates a custom codec and an encoded transform is used to implement part or all of the the custom codec and needs to set the output codec type as part of the metadata of the output frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01

Blink component

Blink>WebRTC

TAG review

The original full spec was reviewed by TAG here: https://github.com/w3ctag/design-reviews/issues/531
No TAG review requested yet for the setMetadata method (during the Working Group discussion it was decided to use a constructor, but interest in the method version was recently revived).

Other use cases: 

https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media 

https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media 

https://w3c.github.io/webrtc-nv-use-cases/#auction 

TAG review status

Pending

Chromium Trial Name

RTCEncodedFrameSetMetadata

Origin Trial documentation link

https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

WebFeature UseCounter name

RTCEncodedFrameSetMetadata

Risks

Interoperability and Compatibility

Interoperability risk: There is always the risk that other browsers will not implement this feature. This risk is mitigated by alignment across browser vendors in the W3C WebRTC Working Group around the spec. Compatibility risk: This is a new feature intended to support new use cases. It introduces no breaking changes, so we do not expect any compatibility issues.

Gecko: No signal

However, they have shown interest in reviving the discussion for the setMetadata() method after achieving consensus on the Custom Codec negotiation API for WebRTC Encoded Transform.

WebKit: No signal

Web developers: Positive

Other signals:

Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is an extension to WebRTC/RTCPeerConnection.

Activation

No significant risks identified.

Security

No new security risks identified.

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

No

Goals for experimentation

Determine if the proposed API properly supports the intended use case.

Reason this experiment is being extended

There are two reasons to request this extension: 1. This proposal initially started as a setMetadata() method on encoded frames, but the result of discussions in the W3C WebRTC Working Group was that introducing a constructor (instead of a method) was a better fit for the use cases for which there was consensus in the WG. After a few iterations over the constructor API shape, the WG achieved consensus recently and we have sent an Intent to Ship for that. However, the final version of the constructor only became available in M126 (the last milestone of the Origin Trial) and we would like to give developers a little more time to migrate to the shipped version of the API. 2. After achieving consensus on the constructor with custom metadata, a new use case has been discussed in the WG that has revived interest in the original setMetadata() proposal. The WG has achieved consensus on a new API for custom codec negotiation for which setMetadata() looks like a better fit than the constructor since it doesn't require copying the payload of the encoded frame. So the WG might achieve consensus on adding setMetadata() after all. See the resolution of https://www.w3.org/2024/04/23-webrtc-minutes.html#t01 Since setMetadata() might be added to the spec in addition to the constructor, we would like to extend the trial to allow developers to continue experimenting with it.

Ongoing technical constraints

None

Debuggability

N/A

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedAudioFrame-metadata.https.html?label=master&label=experimental&aligned https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedVideoFrame-metadata.https.html?label=master&label=experimental&aligned

Flag name on chrome://flags

Finch feature name

RTCEncodedFrameSetMetadata

Non-finch justification

Guarded by a Blink RuntimeEnabledFeature.

Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/40248396

Estimated milestones

Shipping on desktop	126
Origin trial desktop first	118
Origin trial desktop last	126
Origin trial extension 1 end milestone	126
Origin trial extension 2 end milestone	129

Shipping on Android	126
OriginTrial Android last	126
OriginTrial Android first	118

Shipping on WebView	126
OriginTrial webView last	126
OriginTrial webView first	118

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5116073827893248?gate=4892642281521152

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/x2ZACgXrqp0 Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxazRts59rCgrOHm2yDKwpGkXqsd-_5Wkurxid34FknDiQ%40mail.gmail.com
Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/dA4TndGG4VQ
Intent to Ship: https://groups.google.com/a/chromium.org/g/blink-dev/c/pKTAFZBMF_M

[Yoav Weiss (@Shopify)]

Do I understand correctly that you want to extend the OT for 3 more milestones, up to 129 (inclusive)?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxZ2Q8t_x2jVUKe4Ug%3DPE%3D_oeubMFx%2BgEGmDnPuQDUOq2A%40mail.gmail.com.

[Guido Urdaneta]

Yes, but since the main motivation is to let developers migrate to the final version of the API (which changed in the last milestone of the original OT), it would be acceptable to have a shorter extension. This is, of course, assuming the I2S for the final version of the API is approved. 

[Yoav Weiss (@Shopify)]

LGTM to extend experimentation until M129 inclusive

On Fri, May 3, 2024 at 5:05 PM Guido Urdaneta <gui...@chromium.org> wrote:

Yes, but since the main motivation is to let developers migrate to the final version of the API (which changed in the last milestone of the original OT), it would be acceptable to have a shorter extension. This is, of course, assuming the I2S for the final version of the API is approved. 

That's fine! :)