Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions (setMetadata method)

85 views
Skip to first unread message

Guido Urdaneta

unread,
May 2, 2024, 8:46:02 AMMay 2
to blink-dev

Contact emails

h...@chromium.orggui...@chromium.orgagp...@chromium.org

Explainer

https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

Specification

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor

Summary

Allow WebRTC Encoded Transform API to manipulate audio and video frame metadata. Some WebRTC Encoded Transform use cases involve manipulation of not only the payload of encoded video / audio frames but also its metadata. For example, if a peer connection negotiates a custom codec and an encoded transform is used to implement part or all of the the custom codec and needs to set the output codec type as part of the metadata of the output frame. See https://www.w3.org/2024/04/23-webrtc-minutes.html#t01



Blink component

Blink>WebRTC

TAG review

The original full spec was reviewed by TAG here: https://github.com/w3ctag/design-reviews/issues/531
No TAG review requested yet for the setMetadata method (during the Working Group discussion it was decided to use a constructor, but interest in the method version was recently revived).


Other use cases: 

https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media 

https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media 

https://w3c.github.io/webrtc-nv-use-cases/#auction 


TAG review status

Pending

Chromium Trial Name

RTCEncodedFrameSetMetadata

Origin Trial documentation link

https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

WebFeature UseCounter name

RTCEncodedFrameSetMetadata

Risks



Interoperability and Compatibility

Interoperability risk: There is always the risk that other browsers will not implement this feature. This risk is mitigated by alignment across browser vendors in the W3C WebRTC Working Group around the spec. Compatibility risk: This is a new feature intended to support new use cases. It introduces no breaking changes, so we do not expect any compatibility issues.


Gecko: No signal
However, they have shown interest in reviving the discussion for the setMetadata() method after achieving consensus on the Custom Codec negotiation API for WebRTC Encoded Transform.

WebKit: No signal

Web developers: Positive

Other signals:

Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is an extension to WebRTC/RTCPeerConnection.



Activation

No significant risks identified.



Security

No new security risks identified.



WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

No



Goals for experimentation

Determine if the proposed API properly supports the intended use case.



Reason this experiment is being extended

There are two reasons to request this extension: 1. This proposal initially started as a setMetadata() method on encoded frames, but the result of discussions in the W3C WebRTC Working Group was that introducing a constructor (instead of a method) was a better fit for the use cases for which there was consensus in the WG. After a few iterations over the constructor API shape, the WG achieved consensus recently and we have sent an Intent to Ship for that. However, the final version of the constructor only became available in M126 (the last milestone of the Origin Trial) and we would like to give developers a little more time to migrate to the shipped version of the API. 2. After achieving consensus on the constructor with custom metadata, a new use case has been discussed in the WG that has revived interest in the original setMetadata() proposal. The WG has achieved consensus on a new API for custom codec negotiation for which setMetadata() looks like a better fit than the constructor since it doesn't require copying the payload of the encoded frame. So the WG might achieve consensus on adding setMetadata() after all. See the resolution of https://www.w3.org/2024/04/23-webrtc-minutes.html#t01 Since setMetadata() might be added to the spec in addition to the constructor, we would like to extend the trial to allow developers to continue experimenting with it.



Ongoing technical constraints

None



Debuggability

N/A



Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedAudioFrame-metadata.https.html?label=master&label=experimental&aligned https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedVideoFrame-metadata.https.html?label=master&label=experimental&aligned



Flag name on chrome://flags



Finch feature name

RTCEncodedFrameSetMetadata

Non-finch justification

Guarded by a Blink RuntimeEnabledFeature.



Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/40248396

Estimated milestones

Shipping on desktop126
Origin trial desktop first118
Origin trial desktop last126
Origin trial extension 1 end milestone126
Origin trial extension 2 end milestone129

Shipping on Android126
OriginTrial Android last126
OriginTrial Android first118
Shipping on WebView126
OriginTrial webView last126
OriginTrial webView first118


Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5116073827893248?gate=4892642281521152

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/x2ZACgXrqp0 Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxazRts59rCgrOHm2yDKwpGkXqsd-_5Wkurxid34FknDiQ%40mail.gmail.com
Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/dA4TndGG4VQ
Intent to Ship: https://groups.google.com/a/chromium.org/g/blink-dev/c/pKTAFZBMF_M

Yoav Weiss (@Shopify)

unread,
May 3, 2024, 8:39:53 AMMay 3
to Guido Urdaneta, blink-dev
Do I understand correctly that you want to extend the OT for 3 more milestones, up to 129 (inclusive)?



--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxZ2Q8t_x2jVUKe4Ug%3DPE%3D_oeubMFx%2BgEGmDnPuQDUOq2A%40mail.gmail.com.

Guido Urdaneta

unread,
May 3, 2024, 11:06:09 AMMay 3
to Yoav Weiss (@Shopify), Guido Urdaneta, blink-dev
Yes, but since the main motivation is to let developers migrate to the final version of the API (which changed in the last milestone of the original OT), it would be acceptable to have a shorter extension. This is, of course, assuming the I2S for the final version of the API is approved. 

Yoav Weiss (@Shopify)

unread,
May 3, 2024, 11:56:17 AMMay 3
to Guido Urdaneta, blink-dev
LGTM to extend experimentation until M129 inclusive

On Fri, May 3, 2024 at 5:05 PM Guido Urdaneta <gui...@chromium.org> wrote:
Yes, but since the main motivation is to let developers migrate to the final version of the API (which changed in the last milestone of the original OT), it would be acceptable to have a shorter extension. This is, of course, assuming the I2S for the final version of the API is approved. 

That's fine! :)
Reply all
Reply to author
Forward
0 new messages