Intent to Ship: WebGPU WebCodecs integration

[François Beaufort]

Contact emails

cwa...@google.com, kai...@google.com, baj...@google.com

Explainer

https://developer.chrome.com/blog/new-in-webgpu-113/#use-webcodecs-videoframe-source-in-importexternaltexture

https://github.com/gpuweb/gpuweb/issues/1380

https://github.com/gpuweb/gpuweb/issues/4165

https://gpuweb.github.io/gpuweb/explainer/#image-input

Specification

https://gpuweb.github.io/gpuweb/#gpuexternaltexture

Design docs

https://github.com/gpuweb/gpuweb/issues/1380

Summary

WebGPU exposes an API to create opaque "external texture" objects from HTMLVideoElement. These object can be used to sample the video frames efficiently, potentially in a zero-copy way directly from the source YUV data. However the WebGPU specification for the first version of WebGPU does not allow creating GPUExternalTextures from WebCodecs VideoFrame objects. This capability is important for advanced video processing applications that are already using WebCodecs and would like to integrate WebGPU in the video processing pipeline. This feature adds support for using a VideoFrame as the source for a GPUExternalTexture and a copyExternalImageToTexture call.

Blink component

Blink>WebGPU

TAG review

A new TAG review is not needed in this case as WebGPU had one recently already: https://github.com/w3ctag/design-reviews/issues/626 

This small but important addition is about adding VideoFrame support on top of HTMLVideoElement support to GPUExternalTextures and copyExternalImageToTexture call.

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Positive (https://github.com/gpuweb/gpuweb/wiki/Minutes-2023-04-19#investigation-import-videoframe-from-webcodec-to-webgpu-1380:~:text=KG%3A%20the%20proposal%20above%20makes%20sense) WebCodecs is listed as "worth prototyping" which likely means this intergration is the same.

WebKit: In development (https://github.com/WebKit/WebKit/pull/14055)

Web developers: Positive

Other signals:

Ergonomics

No ergonomic risk. This API would be used at the intersection of WebGPU and WebCodec. It is designed to keep performance as high as possible by allowing zero-copy sampling of YUV frame data.

Security

The lifetime management of VideoFrame was taken into account of this feature. No other security considerations.

WebView application risks

N/A

Debuggability

No support.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

It will be supported where WebGPU is supported. For now, it is only ChromeOS, macOS, and Windows.

Is this feature fully tested by web-platform-tests?

Yes

DevTrial instructions

https://github.com/gpuweb/gpuweb/issues/1380

Requires code in //chrome?

False

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1455083

Availability expectation

This feature will be part of the WebGPU API in all browsers when they ship it.

Sample links

https://webgpu.github.io/webgpu-samples/samples/videoUploadingWebCodecs

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5078348864159744

Links to previous Intent discussions

Intent to Experiment: https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGdfWNPH6Jk-g%2B3Nkx0F_rmn05kWcWU%3Dd3cOJNpjfZ3nAWjXcQ%40mail.gmail.com

This intent message was generated by Chrome Platform Status.

[Alex Russell]

LGTM1, but does this need to bounce through main thread video decode for every frame sampled? Is it not possible to target a WebGPU texture with the decoded buffer contents without needing to lean on the main thread's `requestVideoFrameCallback()`?

[François Beaufort]

Thank you Alex.

WebGPU can use VideoFrames in any worker so as long as they can be produced off-main thread then it shouldn't block.

VideoFrames are also possible to postMessage between workers if needed.

[Yoav Weiss]

LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAPpwU5Kb%3DNMqiZURzoUK%2BJ45Au2P_rY7yAzK8-OeCZ-g3%2Bo98g%40mail.gmail.com.

[Mike Taylor]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfXecSRht7k4wzc22bBF-GtX9nWSko3nPpTB4j9d-s%3DDmw%40mail.gmail.com.