Intent to Implement: WebGL last uploaded video frame metadata

[Kai Ninomiya]

Contact emails

kai...@chromium.org, k...@chromium.org

Spec

This is an experimental change, hidden behind a flag, with no formal spec. Any future version of this API which is intended to ship will have a different design, to be determined.

When a frame of an HTMLVideoElement is uploaded to a WebGLTexture, the metadata (width, height, and timestamp) of that frame shall be snapshotted. These values are accessed via the following attributes added to WebGLTexture, and shall be visible immediately after the return of the uploading function (e.g. texImage2D):

interface WebGLTexture {

    readonly attribute unsigned long lastUploadedVideoWidth;

    readonly attribute unsigned long lastUploadedVideoHeight;

    readonly attribute double lastUploadedVideoTimestamp;

};

Summary

Allows WebGL developers to query metadata about the exact frame that is visible to their WebGL application via a WebGLTexture.

Motivation

This experimental API is a request from the YouTube 360 team. It will allow them to experiment with getting better alignment of their metadata and video stream with the frame that's currently being displayed. A future, specified, cross-browser API will replace this API for real use.

Interoperability and Compatibility Risk

Never intended to launch. The API which replaces this experiment will be vetted by all UAs for implementability.

Ongoing technical constraints

This is expected to be removed after prototyping is complete, in preparation for a formally specified API to take its place.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

OWP launch tracking bug

Will never launch.

Link to entry on the feature dashboard

N/A

Requesting approval to ship?

No.

[Harald Alvestrand]

what's the definition of the timestamp?

In WebRTC, I've been wondering about timestamps for a lot of cases, and there seems to be a plethora of them.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Kai Ninomiya]

I don't know a lot about media and different timestamp definitions. The one being used in the implementation is:

media/base/video_frame.h:384

// The time span between the current frame and the first frame of the stream.

// This is the media timestamp, and not the reference time.

// See VideoFrameMetadata::REFERENCE_TIME for details.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

[Dale Curtis]

Timestamp in this case should be the packet/frame presentation timestamp as muxed into the media container. I'm pretty sure this is untouched along the path from demuxing to rendering.

Note: For poorly muxed files, src= (i.e. not Media Source Extensions) demuxing may swap presentation and display timestamps, https://cs.chromium.org/chromium/src/third_party/ffmpeg/libavcodec/utils.c?l=2071

- dale

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Joe Medley]

Kai,

I need a way to follow this. Do you have any kind of bug I can star?

[David Dorwin]

The related bug is https://bugs.chromium.org/p/chromium/issues/detail?id=639174

[Kai Ninomiya]

We're making a revision to this prototype API. https://crrev.com/c/616188

 

Contact emails

kai...@chromium.org, k...@chromium.org

Spec

This is an experimental change, hidden behind a flag, with no formal spec. Any future version of this API which is intended to ship will have a different design, to be determined.

When a frame of an HTMLVideoElement is uploaded to a WebGLTexture, the metadata (width, height, and timestamp) of that frame shall be snapshotted. These values are accessed via the following attributes added to WebGLTexture, and shall be visible immediately after the return of the uploading function (e.g. texImage2D):

interface WebGLTexture {

    readonly attribute unsigned long lastUploadedVideoWidth;

    readonly attribute unsigned long lastUploadedVideoHeight;

    readonly attribute double lastUploadedVideoTimestamp;

    readonly attribute boolean lastUploadedVideoFrameWasSkipped;

};

Summary

Allows WebGL developers to query metadata about the exact frame that is visible to their WebGL application via a WebGLTexture.

Motivation

This is to be used for experimental prototyping efforts to make sure that a similar API will provide the necessary info to applications.

Interoperability and Compatibility Risk

Never intended to launch. The API which replaces this experiment will be vetted by all UAs for implementability.

Ongoing technical constraints

This is expected to be removed after prototyping is complete, in preparation for a formally specified API to take its place.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

OWP launch tracking bug

Never intended to launch.