Intent to Ship: WebRTC encoded transform - Constructor with custom Metadata (originally Modify Metadata functions)

[Guido Urdaneta]

Contact emails

h...@chromium.org, gui...@chromium.org, agp...@chromium.org

Explainer

https://github.com/guidou/webrtc-extensions/blob/main/constructor-explainer.md

Specification

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedvideoframe-constructor

https://w3c.github.io/webrtc-encoded-transform/#dom-rtcencodedaudioframe-constructor

Summary

Allow WebRTC Encoded Transform API to create encoded audio and video frames specifying custom metadata. This is achieved by introducing constructors for encoded frames that take the original frame and custom metadata as input. This supports use cases that involve manipulation of not only the payload of encoded video / audio frames but also its metadata. Some examples: * Changing the mime type of the frame if the transform changes the type of the payload * Forwarding of media to a new peer connection set up to use different metadata values * Altering the timestamp of a frame to introduce a delay
Use cases: https://w3c.github.io/webrtc-nv-use-cases/#live-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#stored-encoded-media https://w3c.github.io/webrtc-nv-use-cases/#auction Issue link: https://github.com/w3c/webrtc-nv-use-cases/issues/77

This change has consensus in the WebRTC Working Group and has been merged into the WebRTC Encoded Transform spec.

Blink component

Blink>WebRTC

TAG review

TAG review request for this specific change: https://github.com/w3ctag/design-reviews/issues/942 

The original version of the full spec was reviewed by TAG here: https://github.com/w3ctag/design-reviews/issues/531

TAG review status

Pending

Chromium Trial Name

RTCEncodedFrameSetMetadata

Origin Trial documentation link

https://github.com/palak8669/webrtc-encoded-transform/blob/create-encoded-explainer/create-encoded-explainer.md

WebFeature UseCounter name

RTCEncodedFrameSetMetadata

Risks

Interoperability and Compatibility

Interoperability risk: There is always the risk that other browsers will not implement this feature. This risk is mitigated by alignment across browser vendors in the W3C WebRTC Working Group around the spec. Compatibility risk: This is a new feature intended to support new use cases. It introduces no breaking changes, so we do not expect any compatibility issues.

Gecko: No signal (https://github.com/mozilla/standards-positions/issues/1009) During WebRTC WG meetings, Mozilla has shown positive signals and agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223

WebKit: Positive (https://github.com/WebKit/standards-positions/issues/334) 

Marked as "Invalid" in the position request because this is a small change and the position was addressed in the PR review. The comments from WebKit in the PR review and during WebRTC WG meetings are positive and they have agreed with merging the PR in the main spec. See the exchange in https://github.com/w3c/webrtc-encoded-transform/pull/223

Web developers: Positive

Other signals:

Ergonomics

This feature is an extension to WebRTC Encoded Transform, which itself is an extension to WebRTC/RTCPeerConnection.

Activation

No significant risks identified.

Security

No new security risks identified.

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

No

Debuggability

N/A

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedAudioFrame-metadata.https.html?label=master&label=experimental&aligned https://wpt.fyi/results/webrtc-encoded-transform/tentative/RTCEncodedVideoFrame-metadata.https.html?label=master&label=experimental&aligned

Flag name on chrome://flags

Finch feature name

RTCEncodedFrameSetMetadata

Non-finch justification

Guarded by a Blink RuntimeEnabledFeature.

Requires code in //chrome?

False

Tracking bug

https://issues.chromium.org/issues/40248396

Estimated milestones

Shipping on desktop	126
Origin trial desktop first	118
Origin trial desktop last	126
Origin trial extension 1 end milestone	126

Shipping on Android	126
OriginTrial Android last	126
OriginTrial Android first	118

Shipping on WebView	126
OriginTrial webView last	126
OriginTrial webView first	118

Anticipated spec changes

None for the changes covered by this intent.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5116073827893248?gate=5122458632323072

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/x2ZACgXrqp0 Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxazRts59rCgrOHm2yDKwpGkXqsd-_5Wkurxid34FknDiQ%40mail.gmail.com
Intent to Extend Experiment 1: https://groups.google.com/a/chromium.org/g/blink-dev/c/dA4TndGG4VQ

This intent message was generated by Chrome Platform Status.

[Alex Russell]

Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Contact emails

h...@chromium.org, guidou@chromium.org, agpalak@chromium.org

[Guido Urdaneta]

Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.

https://docs.google.com/document/d/1QSYbrlgE_6ZZag_VRd8Mn863Slb5-GLlJ_-X4WOiym0/edit?usp=sharing

On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:

Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Contact emails

h...@chromium.org, gui...@chromium.org, agp...@chromium.org

[Mike Taylor]

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxa8OyREc7SGWbwSB9xPkwrbpwjkw%2Bhz%2BPJtPezDpOmu_w%40mail.gmail.com.

[Domenic Denicola]

LGTM2

On Wednesday, May 15, 2024 at 4:47:32 AM UTC+9 Mike Taylor wrote:

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

On 5/14/24 8:05 AM, Guido Urdaneta wrote:

Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.

https://docs.google.com/document/d/1QSYbrlgE_6ZZag_VRd8Mn863Slb5-GLlJ_-X4WOiym0/edit?usp=sharing

On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:

Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Contact emails

h...@chromium.org, guidou@chromium.org, agpalak@chromium.org

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Daniel Bratell]

The document seems to be an internal one. Is there anything you can share publicly?

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c04d3cba-f2de-420a-9938-6848adba9568n%40chromium.org.

[Guido Urdaneta]

I just opened access to the doc.

On Wed, May 15, 2024 at 2:57 PM Daniel Bratell <bra...@sarasas.se> wrote:

The document seems to be an internal one. Is there anything you can share publicly?

/Daniel

On 2024-05-15 07:34, Domenic Denicola wrote:

LGTM2

On Wednesday, May 15, 2024 at 4:47:32 AM UTC+9 Mike Taylor wrote:

Thanks for the doc. It sounds like the design evolved during the OT based on fedback from the WG, and at least one partner was satisfied with the shape and functionality.

LGTM1

On 5/14/24 8:05 AM, Guido Urdaneta wrote:

Here is a doc with feedback from the Origin Trial. There were two registrants reported with a large usage scale. We received feedback from one of them and will update the doc if/when we get feedback from the other one.

https://docs.google.com/document/d/1QSYbrlgE_6ZZag_VRd8Mn863Slb5-GLlJ_-X4WOiym0/edit?usp=sharing

On Wed, May 8, 2024 at 5:45 PM Alex Russell <sligh...@chromium.org> wrote:

Hey Guido,

This is a cool feature! The Milestones section shows that an OT was run; is there a summary someplace of what we learned from the OT?

Best,

Alex

On Thursday, May 2, 2024 at 4:40:31 AM UTC-7 Guido Urdaneta wrote:

Contact emails

h...@chromium.org, gui...@chromium.org, agp...@chromium.org

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxa8OyREc7SGWbwSB9xPkwrbpwjkw%2Bhz%2BPJtPezDpOmu_w%40mail.gmail.com.

[Daniel Bratell]

LGTM3

/Daniel

On 2024-05-15 15:47, Guido Urdaneta wrote:

I just opened access to the doc.