Intent to Ship: WebRTC encoded transform - Constructor with custom Metadata (originally Modify Metadata functions)

229 views
Skip to first unread message

Guido Urdaneta

unread,
May 2, 2024, 7:40:31 AMMay 2
to blink-dev

Contact emails

h...@chromium.orggui...@chromium.orgagp...@chromium.org

Explainer

https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

Specification

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedaudioframe-constructor

Summary

Allow WebRTC Encoded Transform API to create encoded audio and video frames specifying custom metadata. This is achieved by introducing constructors for encoded frames that take the original frame and custom metadata as input. This supports use cases that involve manipulation of not only the payload of encoded video / audio frames but also its metadata. Some examples: * Changing the mime type of the frame if the transform changes the type of the payload * Forwarding of media to a new peer connection set up to use different metadata values * Altering the timestamp of a frame to introduce a delay
Use cases: https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#auction Issue link: https://github.com/w3c/webrtc-nv-use-cases/issues/77


This change has consensus in the WebRTC Working Group and has been merged into the WebRTC Encoded Transform spec.

Blink component

Blink>WebRTC

TAG review

TAG review request for this specific change: https://github.com/w3ctag/design-reviews/issues/942 

The original version of the full spec was reviewed by TAG here: https://github.com/w3ctag/design-reviews/issues/531

TAG review status

Pending

Chromium Trial Name

RTCEncodedFrameSetMetadata

Origin Trial documentation link

https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

WebFeature UseCounter name

RTCEncodedFrameSetMetadata

Risks



Interoperability and Compatibility

Interoperability risk: There is always the risk that other browsers will not implement this feature. This risk is mitigated by alignment across browser vendors in the W3C WebRTC Working Group around the spec. Compatibility risk: This is a new feature intended to support new use cases. It introduces no breaking changes, so we do not expect any compatibility issues.


Gecko: No signal (https://github.com/mozilla/standards-positions/issues/1009) During WebRTC WG meetings, Mozilla has shown positive signals and agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223

WebKit: Positive (https://github.com/WebKit/standards-positions/issues/334
Marked as "Invalid" in the position request because this is a small change and the position was addressed in the PR review. The comments from WebKit in the PR review and during WebRTC WG meetings are positive and they have agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223

Web developers: Positive

Other signals:

Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is an extension to WebRTC/RTCPeerConnection.



Activation

No significant risks identified.



Security

No new security risks identified.



WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

No



Debuggability

N/A



Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedAudioFrame-metadata.https.html?label=master&label=experimental&aligned https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedVideoFrame-metadata.https.html?label=master&label=experimental&aligned



Flag name on chrome://flags



Finch feature name

RTCEncodedFrameSetMetadata

Non-finch justification

Guarded by a Blink RuntimeEnabledFeature.



Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/40248396

Estimated milestones

Shipping on desktop126
Origin trial desktop first118
Origin trial desktop last126
Origin trial extension 1 end milestone126
Shipping on Android126
OriginTrial Android last126
OriginTrial Android first118
Shipping on WebView126
OriginTrial webView last126
OriginTrial webView first118


Anticipated spec changes

None for the changes covered by this intent.


Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5116073827893248?gate=5122458632323072

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/x2ZACgXrqp0 Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxazRts59rCgrOHm2yDKwpGkXqsd-_5Wkurxid34FknDiQ%40mail.gmail.com
Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/dA4TndGG4VQ


This intent message was generated by Chrome Platform Status.

Alex Russell

unread,
May 8, 2024, 11:45:27 AMMay 8
to blink-dev, Guido Urdaneta
Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Guido Urdaneta

unread,
May 14, 2024, 8:05:50 AMMay 14
to Alex Russell, blink-dev, Guido Urdaneta
Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.



On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:
Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Mike Taylor

unread,
May 14, 2024, 3:47:32 PMMay 14
to Guido Urdaneta, Alex Russell, blink-dev

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxa8OyREc7SGWbwSB9xPkwrbpwjkw%2Bhz%2BPJtPezDpOmu_w%40mail.gmail.com.

Domenic Denicola

unread,
May 15, 2024, 1:34:47 AMMay 15
to blink-dev, Mike Taylor, blink-dev, Guido Urdaneta, Alex Russell
LGTM2

On Wednesday, May 15, 2024 at 4:47:32 AM UTC+9 Mike Taylor wrote:

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

On 5/14/24 8:05 AM, Guido Urdaneta wrote:
Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.



On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:
Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Daniel Bratell

unread,
May 15, 2024, 8:57:49 AMMay 15
to Domenic Denicola, blink-dev, Mike Taylor, Guido Urdaneta, Alex Russell

The document seems to be an internal one. Is there anything you can share publicly?

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c04d3cba-f2de-420a-9938-6848adba9568n%40chromium.org.

Guido Urdaneta

unread,
May 15, 2024, 9:48:23 AMMay 15
to Daniel Bratell, Domenic Denicola, blink-dev, Mike Taylor, Guido Urdaneta, Alex Russell
I just opened access to the doc.

On Wed, May 15, 2024 at 2:57 PM Daniel Bratell <bra...@sarasas.se> wrote:

The document seems to be an internal one. Is there anything you can share publicly?

/Daniel

On 2024-05-15 07:34, Domenic Denicola wrote:
LGTM2

On Wednesday, May 15, 2024 at 4:47:32 AM UTC+9 Mike Taylor wrote:

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

On 5/14/24 8:05 AM, Guido Urdaneta wrote:
Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.



On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:
Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxa8OyREc7SGWbwSB9xPkwrbpwjkw%2Bhz%2BPJtPezDpOmu_w%40mail.gmail.com.

Daniel Bratell

unread,
May 15, 2024, 10:19:38 AMMay 15
to Guido Urdaneta, Domenic Denicola, blink-dev, Mike Taylor, Alex Russell

LGTM3

/Daniel

On 2024-05-15 15:47, Guido Urdaneta wrote:
I just opened access to the doc.

Reply all
Reply to author
Forward
0 new messages