Intent to Implement and Ship: Conversion to RGB in VideoFrame.copyTo()

[Eugene Zemtsov]

Contact emails

ezem...@google.com

Explainer

https://gist.github.com/Djuffin/9e2f98025ead49998524510cfeed8d33

Specification

https://www.w3.org/TR/webcodecs/#dom-videoframe-copyto

Summary

VideoFrame.copyTo() can convert pixel data to RGB pixel format Converting YUV video frames to RGB is often required for processing them in libraries like TensorFlow.js and OpenCV.js. Previously the only possible way to achieve this was rendering the frame on a canvas. Specifying VideoFrameCopyToOptions.format and VideoFrameCopyToOptions.colorSpace makes it possible to convert frames to RGB pixel formats by calling VideoFrame.copyTo() without having to use an extra canvas.

Blink component

Blink>Media>WebCodecs

Initial public proposal

https://github.com/w3c/webcodecs/issues/92

TAG review

N/A since the change is minor

Risks

Interoperability and Compatibility

None

Gecko: Positive (https://github.com/w3c/webcodecs/pull/754#pullrequestreview-2008590591)

WebKit: Positive TPAC 2023. Media working group session.

Web developers: 

https://github.com/w3c/webcodecs/issues/92#issuecomment-1594083978 

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

None

Is this feature fully tested by web-platform-tests?

Yes

https://wpt.fyi/results/webcodecs videoFrame-copyTo-rgb.any.js

Estimated milestones

Shipping on desktop

126

Shipping on Android

126

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/4668827056209920

[Mike Taylor]

Can you please request reviews for privacy, security, enterprise, etc in the chromestatus entry?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrGyVMDipu_nqd%3DKw_9eE2UMtdbiWjbqac0NquwNmm4DMg%40mail.gmail.com.

[Eugene Zemtsov]

> Can you please request reviews for privacy, security, enterprise, etc in the chromestatus entry?

Done

--

Thanks,

Eugene Zemtsov.

[Alex Russell]

hey Eugene,

This is an exciting an useful addition! Have you looked into other platform APIs that could benefit from being able to explicitly specify intermediate format hinting and/or transformation? It's a place where (had the TAG been consulted) I would have expected to see a larger chain of additions to make this work in other areas, or at least a discussion of why they aren't (or are?) being pursued. Have you looked into that?

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrGyVMDipu_nqd%3DKw_9eE2UMtdbiWjbqac0NquwNmm4DMg%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

[Mike Taylor]

+1. Would you mind also filing gecko and webkit positions? I expect them to be positive, given the informal signals you have in the spec PRs already - but this also lets them know we're moving ahead with shipping. Thanks - Mike

[Eugene Zemtsov]

I've filed issues for TAG review and firefox and webkit positions.

>  Have you looked into other platform APIs that could benefit from being able to explicitly specify intermediate format hinting and/or transformation?

I don't think this kind of review can be done for all web APIs by one person. It's up to spec editors in each area to identify these small changes that improve dev experience and performance. 

For media related things: 

-  VideoFrame is a versatile tool that can be created from HTMLOrSVGImageElement, HTMLVideoElement, HTMLCanvasElement, ImageBitmap,

         OffscreenCanvas without a copy. In a way this change extends variety of export formats for all of them.

    So lots of things can be exported to RGB bitmaps without a canvas now. 

-  AudioData has a way to be exported into different sampling formats.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrGyVMDipu_nqd%3DKw_9eE2UMtdbiWjbqac0NquwNmm4DMg%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

[Eugene Zemtsov]

This feature has Privacy, Security, Enterprise, and Debuggability approvals. Webkit gave a positive signal. 

Any objections or questions from the API owners? 

--

Thanks,

Eugene Zemtsov.

[Mike Taylor]

I see that WebKit raised an API concern around detectability. In your reply, you said "we should consider...". Has that consideration happened, perhaps as follow-up work? :)

[Eugene Zemtsov]

In that discussion Marcos Cáceres asked for a synchronous way to detect if format conversion is supported. 

We have a synchronous call allocationSize() that throws an unsupported error in cases where format conversion is not supported. 

I added a WPT test around it in a recent CL.

canCopyTo() might be needed if we have more settings in VideoFrameCopyToOptions and might want to know which part exactly is causing an error. 

--

Thanks,

Eugene Zemtsov.

[Mike Taylor]

Ergonomics-wise, it does not seem intuitive to use a method called allocationSize() to feature detect copyTo support, even if the semantics are the same. Besides a very careful reading of the spec, how do we expect developers to know about it?

[Eugene Zemtsov]

By reading an MDN article that I'll update after the launch. 

Even now allocationSize() is supposed to be called before copyTo() anyway, to figure out the size of the buffer for the output. (see example1, example2) 

That's why I don't think that it's an ergonomic burden for developers. 

--

Thanks,

Eugene Zemtsov.

[Domenic Denicola]

LGTM1.

I have a small non-blocking request: update your Chrome Status entry to avoid using ClassName.staticMethod() syntax for what appears to be an instance method. (See e.g. this related discussion and this bug I filed on Chrome Status a year ago.) This will avoid a confusing experience for web developers when this information makes its way into the beta blog post and other documentation.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrGyVMDipu_nqd%3DKw_9eE2UMtdbiWjbqac0NquwNmm4DMg%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

[Daniel Bratell]

LGTM2

/Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/221f4af8-b6b4-4d36-9225-3a8594dc5b14n%40chromium.org.

[Yoav Weiss (@Shopify)]

LGTM3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrGyVMDipu_nqd%3DKw_9eE2UMtdbiWjbqac0NquwNmm4DMg%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/221f4af8-b6b4-4d36-9225-3a8594dc5b14n%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/c6cdcdbd-1a21-4202-95b1-78eca60d0c5f%40gmail.com.