Intent to Ship: timestamp field on RTCEncodedVideoFrameMetadata

167 views
Skip to first unread message

Tony Herre

unread,
May 2, 2023, 10:29:22 AM5/2/23
to blin...@chromium.org

Contact emails

he...@google.com, gui...@chromium.com


Spec

PR#173.


Summary

Add a 'timestamp' field to RTCEncodedVideoFrameMetadata containing the presentation timestamp of the associated encoded video frame.


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes


Risks

Interoperability and Compatibility

Positive response from all members at W3C WebRTC WG April 2023 Interim, PR landed with no open issues.


Ergonomics

Added specifically to align with the timestamp field on the WebCodecs EncodedVideoChunk, and allow matching video frames with the timestamp in the WebCodecs VideoFrame object.

Will also provide the same timestamp already exposed in requestVideoFrameCallback's 'mediaTime'.


Is this feature fully tested by web-platform-tests?

Tested internally by RTCEncodedVideoFrame-capture-timestamp-id.html, upstreaming to WPT tracked in crbug.com/1441888.



Entry on the feature dashboard

None, small delta to launched API.

Nicola Tommasi

unread,
May 8, 2023, 10:22:03 AM5/8/23
to blink-dev, Tony Herre
Hi Tony,

Thanks for this intent, could you please clarify why this new field is needed and important?Without an explainer is hard to follow this change.

Cheers,
Nicola

Tony Herre

unread,
May 9, 2023, 5:05:36 AM5/9/23
to Nicola Tommasi, blink-dev
Thanks for the reply - hard to judge how much context to include for small launches like this.

The motivation is in helping WebRTC web applications which are using both Mediacapture Transforms to modify raw audio/video (adding video effects, background replace etc) and WebRTC Encoded Transforms for modifying the WebRTC encoded data which gets sent on over the network (to do client-side encryption, appending additional application metadata within the media, observing per-frame WebRTC metadata like ContributingSources etc).
Adding this `timestamp` field to RTCEncodedVideoFrameMetadata means that, when getting an encoded video frame in the encoded transform, the app can now tell which raw video frame in the mediacapture transform was encoded to produce this (or when receiving video, which raw frame is later produced after decoding this encoded frame), by matching RTCEncodedVideoFrameMetadata.timestamp with VideoFrame.timestamp.

With this, the application can coordinate the two transforms a lot better - eg changing how raw frames are processed or rendered on the exact frame in which webrtc metadata changed, or change details of the encoded data on the exact frame when something in the raw video transform was changed.

As for developer interest, I'm working closely with at least one quick adopter, and the quick agreement in the WebRTC Working Group shows this is generally useful and a part of the roadmap.

Thanks!

Nicola Tommasi

unread,
May 10, 2023, 12:14:25 PM5/10/23
to blink-dev, Tony Herre, blink-dev, Nicola Tommasi
I didn't have much background on this topic to be honest and I wanted to understand the real usage scenario. Thanks a lot for clarifying this. 

Nicola

Yoav Weiss

unread,
May 11, 2023, 8:21:05 AM5/11/23
to Tony Herre, Jason Robbins, blin...@chromium.org
I suspect this resulted in this intent not being captured in our tooling :/

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAArnMxH%2BNJKi80S4822wmjtQUqid8czA9AefCC_fm%3DHk3sTuiw%40mail.gmail.com.

Yoav Weiss

unread,
May 11, 2023, 10:22:25 AM5/11/23
to Tony Herre, Jason Robbins, blin...@chromium.org
As pointed out, this seems to be missing a bunch of fields from the broader template..


On Thu, May 11, 2023 at 2:20 PM Yoav Weiss <yoav...@chromium.org> wrote:


On Tue, May 2, 2023 at 4:29 PM 'Tony Herre' via blink-dev <blin...@chromium.org> wrote:

An explainer is missing, but you provided an inline one above, which is good enough from my perspective.

A TAG review checkpoint is also missing. At the same time, we probably don't need to file for one here.
 

Spec

PR#173.


Summary

Add a 'timestamp' field to RTCEncodedVideoFrameMetadata containing the presentation timestamp of the associated encoded video frame.


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes


Risks

Interoperability and Compatibility

Positive response from all members at W3C WebRTC WG April 2023 Interim, PR landed with no open issues.


While I agree that there seems to be general support on the PR, we typically require formal https://bit.ly/blink-signals
 

Ergonomics

Added specifically to align with the timestamp field on the WebCodecs EncodedVideoChunk, and allow matching video frames with the timestamp in the WebCodecs VideoFrame object.

Will also provide the same timestamp already exposed in requestVideoFrameCallback's 'mediaTime'.


Is this feature fully tested by web-platform-tests?

Tested internally by RTCEncodedVideoFrame-capture-timestamp-id.html, upstreaming to WPT tracked in crbug.com/1441888.


It seems easier to land WPT directly with a ".tentative" suffix, and then rename them.

Tony Herre

unread,
May 12, 2023, 5:35:30 AM5/12/23
to blink-dev, Yoav Weiss, blin...@chromium.org, Tony Herre, Jason Robbins
Re TAG, the original review for this API was https://github.com/w3ctag/design-reviews/issues/531.

Re signals, looks like this would have been better expressed as "no signal" from other UAs to capture the formal stance, with just a note about informal support in the WG.
I would suggest it wouldn't be worth requesting due to the "Change is too small to justify this effort" caveat in the blink-signals doc - please let me know if others disagree and I can get this kicked off.


Thanks for all the process references! I'll be sure to leave the template headings in-place in future Intents, so hopefully will go through a lot smoother.

Thanks,
Tony

Tony Herre

unread,
May 31, 2023, 3:47:22 AM5/31/23
to blink-dev, Yoav Weiss, Jason Robbins
Bubbling this up as I suspect it has fallen through the process again.

Is there any further information or actions I need to take here to unblock?

Thanks
Tony

Chris Harrelson

unread,
May 31, 2023, 2:37:31 PM5/31/23
to Tony Herre, blink-dev, Yoav Weiss, Jason Robbins
Hi Tony,

Please make a chromestatus.com entry for this change, we need them for all changes to the platform, so they can be tracked in release notes and elsewhere. (And as Yoav mentioned, it's why this intent was missed during our API owners triage process.)

As for signals: please file one for Gecko and Webkit, thanks.



Tony Herre

unread,
Jun 2, 2023, 7:27:26 AM6/2/23
to Saddam Molla Saddam Molla, Chris Harrelson, blink-dev, Yoav Weiss, Jason Robbins
Thanks all for the guidance! Sounds best at this point.
I've started a new chromestatus entry, and will request shipping there when we have all the fields complete.

On Wed, 31 May 2023 at 20:41, Saddam Molla Saddam Molla <dinonter...@gmail.com> wrote:
3

Reply all
Reply to author
Forward
0 new messages