Intent to Ship: timestamp field on RTCEncodedVideoFrameMetadata

[Tony Herre]

Contact emails

he...@google.com, gui...@chromium.com

Spec

https://w3c.github.io/webrtc-encoded-transform/#RTCEncodedVideoFrameMetadata particularly

PR#173.

Summary

Add a 'timestamp' field to RTCEncodedVideoFrameMetadata containing the presentation timestamp of the associated encoded video frame.

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Risks

Interoperability and Compatibility

Positive response from all members at W3C WebRTC WG April 2023 Interim, PR landed with no open issues.

Ergonomics

Added specifically to align with the timestamp field on the WebCodecs EncodedVideoChunk, and allow matching video frames with the timestamp in the WebCodecs VideoFrame object.

Will also provide the same timestamp already exposed in requestVideoFrameCallback's 'mediaTime'.

Is this feature fully tested by web-platform-tests?

Tested internally by RTCEncodedVideoFrame-capture-timestamp-id.html, upstreaming to WPT tracked in crbug.com/1441888.

Entry on the feature dashboard

None, small delta to launched API.

[Nicola Tommasi]

Hi Tony,

Thanks for this intent, could you please clarify why this new field is needed and important?Without an explainer is hard to follow this change.

Cheers,

Nicola

[Tony Herre]

Thanks for the reply - hard to judge how much context to include for small launches like this.

The motivation is in helping WebRTC web applications which are using both Mediacapture Transforms to modify raw audio/video (adding video effects, background replace etc) and WebRTC Encoded Transforms for modifying the WebRTC encoded data which gets sent on over the network (to do client-side encryption, appending additional application metadata within the media, observing per-frame WebRTC metadata like ContributingSources etc).

Adding this `timestamp` field to RTCEncodedVideoFrameMetadata means that, when getting an encoded video frame in the encoded transform, the app can now tell which raw video frame in the mediacapture transform was encoded to produce this (or when receiving video, which raw frame is later produced after decoding this encoded frame), by matching RTCEncodedVideoFrameMetadata.timestamp with VideoFrame.timestamp.

With this, the application can coordinate the two transforms a lot better - eg changing how raw frames are processed or rendered on the exact frame in which webrtc metadata changed, or change details of the encoded data on the exact frame when something in the raw video transform was changed.

As for developer interest, I'm working closely with at least one quick adopter, and the quick agreement in the WebRTC Working Group shows this is generally useful and a part of the roadmap.

Thanks!

[Nicola Tommasi]

I didn't have much background on this topic to be honest and I wanted to understand the real usage scenario. Thanks a lot for clarifying this. 

Nicola

[Yoav Weiss]

I suspect this resulted in this intent not being captured in our tooling :/

+Jason Robbins - FYI 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAArnMxH%2BNJKi80S4822wmjtQUqid8czA9AefCC_fm%3DHk3sTuiw%40mail.gmail.com.

[Yoav Weiss]

As pointed out, this seems to be missing a bunch of fields from the broader template..

On Thu, May 11, 2023 at 2:20 PM Yoav Weiss <yoav...@chromium.org> wrote:

On Tue, May 2, 2023 at 4:29 PM 'Tony Herre' via blink-dev <blin...@chromium.org> wrote:

Contact emails

he...@google.com, gui...@chromium.com

An explainer is missing, but you provided an inline one above, which is good enough from my perspective.

A TAG review checkpoint is also missing. At the same time, we probably don't need to file for one here.

 

Spec

https://w3c.github.io/webrtc-encoded-transform/#RTCEncodedVideoFrameMetadata particularly

PR#173.

Summary

Add a 'timestamp' field to RTCEncodedVideoFrameMetadata containing the presentation timestamp of the associated encoded video frame.

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Risks

Interoperability and Compatibility

Positive response from all members at W3C WebRTC WG April 2023 Interim, PR landed with no open issues.

While I agree that there seems to be general support on the PR, we typically require formal https://bit.ly/blink-signals

 

Ergonomics

Added specifically to align with the timestamp field on the WebCodecs EncodedVideoChunk, and allow matching video frames with the timestamp in the WebCodecs VideoFrame object.

Will also provide the same timestamp already exposed in requestVideoFrameCallback's 'mediaTime'.

Is this feature fully tested by web-platform-tests?

Tested internally by RTCEncodedVideoFrame-capture-timestamp-id.html, upstreaming to WPT tracked in crbug.com/1441888.

It seems easier to land WPT directly with a ".tentative" suffix, and then rename them.

[Tony Herre]

Re TAG, the original review for this API was https://github.com/w3ctag/design-reviews/issues/531.

Re signals, looks like this would have been better expressed as "no signal" from other UAs to capture the formal stance, with just a note about informal support in the WG.

I would suggest it wouldn't be worth requesting due to the "Change is too small to justify this effort" caveat in the blink-signals doc - please let me know if others disagree and I can get this kicked off.

Thanks for all the process references! I'll be sure to leave the template headings in-place in future Intents, so hopefully will go through a lot smoother.

Thanks,

Tony

[Tony Herre]

Bubbling this up as I suspect it has fallen through the process again.

Is there any further information or actions I need to take here to unblock?

Thanks

Tony

[Chris Harrelson]

Hi Tony,

Please make a chromestatus.com entry for this change, we need them for all changes to the platform, so they can be tracked in release notes and elsewhere. (And as Yoav mentioned, it's why this intent was missed during our API owners triage process.)

As for signals: please file one for Gecko and Webkit, thanks.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAArnMxEzMsTFwiO%3DXYz7bZf4rjuv6mHzUubOrPke6jZ%3DukdpLQ%40mail.gmail.com.

[Tony Herre]

Thanks all for the guidance! Sounds best at this point.
I've started a new chromestatus entry, and will request shipping there when we have all the fields complete.

On Wed, 31 May 2023 at 20:41, Saddam Molla Saddam Molla <dinonter...@gmail.com> wrote:

3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw8H-CcAeZap7GOWkhCsxSin%3DJNEc35%3D%2Bz2KDcsRqftukA%40mail.gmail.com.