Intent to Extend Experiment: WebRTC encoded transform - Modify Metadata functions
Guido Urdaneta
Contact emails
gui...@chromium.org, he...@google.com, agp...@chromium.org
Explainer
Specification
https://github.com/w3c/webrtc-encoded-transform/pull/223/files
Original I2E
https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ
Summary
Add a feature to the WebRTC Encoded Transform API that allows manipulating audio and video frame metadata.
Background: A number of use cases have been identified that require the manipulation of the metadata of WebRTC encoded frames without decoding them first.
These include:
- Receiving data in encoded form and forwarding it.
- Sending data that has been encoded previously
- Sending data that has been received in encoded form
In particular, we want to support the use case of glitch-free forwarding of media coming from multiple redundant peer connections that provide the same media payloads but with different metadata.
https://github.com/w3c/webrtc-nv-use-cases/issues/77
https://github.com/w3c/webrtc-nv-use-cases/issues/122
https://github.com/w3c/webrtc-encoded-transform/issues/211
Blink component
TAG review
The original full spec was reviewed by TAG here: https://github.com/w3ctag/design-reviews/issues/531
No TAG review has been requested for this incremental change, since it is a small addition and its API surface might change as a result of feedback from the origin trial and ongoing discussions in the WebRTC working group.
TAG review status
Issues addressed for the original spec. Not yet requested for the incremental change.
Chromium Trial Name
RTCEncodedFrameSetMetadata
WebFeature UseCounter name
V8RTCEncodedVideoFrame_SetMetadata_Method
Link to origin trial feedback summary
Risks
Interoperability and Compatibility
Interoperability risk: There is the risk that other browsers will not implement this feature. We are mitigating this risk by pursuing consensus in the W3C WebRTC Working Group so that it becomes part of the WebRTC encoded transform spec supported by all browsers.
Compatibility risk: This is a new feature intended to support a new use case. It introduces no breaking changes, so we do not expect any compatibility issues.
Gecko: No signal requested yet. Positive initial feedback during WG discussions.
WebKit: No signal requested yet. Positive initial feedback during WG discussions.
Web developers: Positive
Other signals:
Ergonomics
This feature is an extension to WebRTC Encoded Transform, which itself is an extension to WebRTC/RTCPeerConnection.
Activation
N/A
Security
No new security risks identified.
WebView application risks
None
Reason this experiment is being extended
We have received positive developer feedback and have validated that the proposal supports the intended use case, but the API requires some changes in order to achieve consensus in the W3C WebRTC Working Group. This is the current status:
1. There is consensus to support the use case, which is already part of the official "WebRTC Extended Use Cases" document from the WG. See requirement N39 in https://www.w3.org/TR/webrtc-nv-use-cases/#auction
2. There is consensus to develop a spec proposal that includes changes to the API surface subject to this origin trial. More specifically, the initial consensus is to remove the setMetadata method in frames (as originally proposed) and instead introduce a constructor for creating new frames given an existing frame and new metadata, following a pattern already established by WebCodecs. In addition, the WG agreed to the introduction of a class that allows attaching a source of encoded frames to the sender of an RTCRtpPeerConnection. See https://github.com/w3c/webrtc-encoded-transform/issues/211#issuecomment-1853419603 and https://www.w3.org/2023/12/12-webrtc-minutes.html#t06. From the WG's point of view, both changes are seen as parts of the same package (i.e., they will be looked at together, not separately).
In this Intent to Extend Experiment, we are requesting to extend the origin trial for 3 milestones (124 to 126 inclusive). During this period we'll introduce the updated frame API (i.e., the constructor) and will temporarily keep the setMetadata method to allow developers to migrate to the new API. The encoded frame source will be pursued by a separate intent.
Ongoing technical constraints
None.
Debuggability
N/A
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?
Yes
Is this feature fully tested by web-platform-tests?
Currently tested using unit and Web tests. Web tests will be moved to WPT once the final API shape is accepted by the WebRTC WG.
Flag name on chrome://flags
None
Finch feature name
RTCEncodedFrameSetMetadata
Requires code in //chrome?
False
Tracking bug
Estimated milestones
Link to entry on the Chrome Platform Status
https://chromestatus.com/feature/5116073827893248
Links to previous Intent discussions
Intent to prototype: https://groups.google.com/a/chromium.org/g/blink-dev/c/x2ZACgXrqp0
Intent to experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/yqwA-jXv6VY/m/fCkBkclxAAAJ
This intent message was generated by Chrome Platform Status.
Domenic Denicola
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxY7yNLBte5CVtk_%3DZfwpZnTn%3DWPCWbt%3DGCn4wWXCr2wtQ%40mail.gmail.com.
Guido Urdaneta
Thanks for updating the explainer!I'm uneasy about this working group policy prohibiting TAG reviews. Can you link to where this policy is stated? If it's indeed not possible for WebRTC WG members to request TAG review due to Working Group restrictions, then that's probably something the API Owners need to discuss more broadly, and consider for the list of acceptable exceptions.Nevertheless, given that the design here has changed significantly over the course of the Origin Trial, I think it's reasonable to delay TAG review until there's been more developer and community feedback on the new API. Just be aware that this could be a blocker when it comes time for Intent to Ship, and please file for review early enough that we can get a reply before then!So, LGTM to extend.On Mon, Jan 29, 2024 at 5:40 PM Guido Urdaneta <gui...@chromium.org> wrote:
On Mon, Jan 29, 2024 at 8:35 AM Domenic Denicola <dom...@chromium.org> wrote:It's exciting to see the Origin Trial system working as intended, giving us time to get in-the-field feedback but also stay responsive to ongoing standards discussions about changing the shape of the API.Recalling the requirements for extending an origin trial, we need to see substantial progress demonstrated in all of the following areas: draft spec, TAG review, signals requests, outreach for feedback from the spec community, and WPT tests. I think these have mostly been met, with the exception of TAG review. Do you think it'd be reasonable to start the TAG review on the constructor proposal now, or do you think there's still a significant chance it might change due to WG feedback, such that we're better of continuing to delay the review?
I think it's too early for a TAG review because the proposal hasn't landed in the spec yet and the WebRTC WG policy nowadays is to wait for specs to be complete before requesting a TAG review.
Also, a question on the explainer below.On Fri, Jan 26, 2024 at 8:27 PM Guido Urdaneta <gui...@chromium.org> wrote:Is this explainer up to date with the newest specification proposal? The specification PR you link below introduces RTCEncodedVideoFrame and RTCEncodedAudioFrame constructors, but the explainer doesn't seem to reference either of those.
It is updated now :)For simplicity, the explainer uses only the RTCEncodedVideoFrame constructor.
Additionally, the explainer seems to talk about a clone "operator" (is that a way of saying method?), but I don't see that in the spec PR.
That part is now removed because a clone operation is already part of the spec via structuredClone() and is no longer part of this proposal.
Domenic Denicola
To clarify about TAG reviews, there is no written policy prohibiting TAG reviews. The policy (or rather the recommended practice) in a case like this is to wait until there is consensus in the WG (i.e., the proposed changes have landed in the official spec) before requesting the TAG review.
BTW, can you send the LGTM again? I forgot to cc blink-dev in my initial reply and the LGTM was sent only to my email address.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxbE1-3Zd%2BDSJ-r%2BMKBCwZ%2Bw0tXDA-BgUC-do%2BA4j5YyRg%40mail.gmail.com.