Intent to Prototype And Ship: Allow transferring ArrayBuffer into VideoFrame, AudioData, EncodedVideoChunk, EncodedAudioChunk, ImageDecoder constructors

[Eugene Zemtsov]

Contact emails

ezem...@google.com

Explainer

https://gist.github.com/Djuffin/1c8fac486ca9f402be85074166e89a16

Specification

https://www.w3.org/TR/webcodecs/#dictdef-videoframeinit

Summary

This will allow detaching array buffers and using corresponding buffers inside VideoFrame, ImageDecoder, EncodedVideoChunk, EncodedAudioChunk, AudioData without a copy.

Blink component

Blink>Media>WebCodecs

TAG review

None

Risks

Interoperability and Compatibility

None

Gecko: N/A (https://www.w3.org/2023/05/30-mediawg-minutes.html#t01) Change is too small to justify this effort. The change was discussed and approved by the w3c media working group.

WebKit: N/A (https://www.w3.org/2023/05/30-mediawg-minutes.html#t01) Change is too small to justify this effort. The change was discussed and approved by the w3c media working group.

Web developers: No signals

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

None

Is this feature fully tested by web-platform-tests?

No

Flag name on chrome://flags

None

Finch feature name

None

Non-finch justification

None

Requires code in //chrome?

False

Tracking bug

https://crbug.com/1446808

Estimated milestones

Shipping on desktop

120

Shipping on Android

120

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5075602045927424

--

Thanks,

Eugene Zemtsov.

[Jonathan Hao]

Hi Eugene,

Just to double check.  Once an ArrayBuffer is transferred and detached, any further access will result in some sort of TypeError, right?

Thanks,

Jonathan

[Yoav Weiss]

On Tuesday, August 22, 2023 at 11:08:24 AM UTC+2 Jonathan Hao wrote:

Hi Eugene,

Just to double check.  Once an ArrayBuffer is transferred and detached, any further access will result in some sort of TypeError, right?

Thanks,

Jonathan

On Wednesday, August 16, 2023 at 10:22:00 PM UTC+1 Eugene Zemtsov wrote:

Contact emailsezem...@google.com

Explainerhttps://gist.github.com/Djuffin/1c8fac486ca9f402be85074166e89a16

Specificationhttps://www.w3.org/TR/webcodecs/#dictdef-videoframeinit

Summary

This will allow detaching array buffers and using corresponding buffers inside VideoFrame, ImageDecoder, EncodedVideoChunk, EncodedAudioChunk, AudioData without a copy.

Blink componentBlink>Media>WebCodecs

TAG reviewNone

Risks

Interoperability and Compatibility

None

Gecko: N/A (https://www.w3.org/2023/05/30-mediawg-minutes.html#t01) Change is too small to justify this effort. The change was discussed and approved by the w3c media working group.

WebKit: N/A (https://www.w3.org/2023/05/30-mediawg-minutes.html#t01) Change is too small to justify this effort. The change was discussed and approved by the w3c media working group.

I somewhat share Youenn's concerns about the API shape, but I'm not familiar with the examples this is supposed to be consistent with. Would it be possible to explore different API shapes in the explainer? (e.g. a boolean that detaches the input buffer after init would be my first choice)

Typically we defer such questions to a TAG review. I'd hate to introduce significant delay at this point, but it might be possible to expedite this specific question and get it in front of them.

 

Web developers: No signals

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None

Debuggability

None

Is this feature fully tested by web-platform-tests?No

Flag name on chrome://flagsNone

Finch feature nameNone

Non-finch justificationNone

Requires code in //chrome?False

Tracking bughttps://crbug.com/1446808

Estimated milestonesShipping on desktop120Shipping on Android120

Link to entry on the Chrome Platform Statushttps://chromestatus.com/feature/5075602045927424

--

Thanks,

Eugene Zemtsov.

[Eugene Zemtsov]

A transfer list is consistent with the approach taken by structuredClone and postMessage. 

And it's already a part of the WebCodecs spec. 

--

Thanks,

Eugene Zemtsov.

[Jonathan Hao]

Thanks for the clarification!

[Yoav Weiss]

On Wed, Aug 23, 2023 at 12:26 PM Jonathan Hao <ph...@chromium.org> wrote:

Thanks for the clarification!

On Tue, Aug 22, 2023 at 9:20 PM Eugene Zemtsov <ezem...@google.com> wrote:

A transfer list is consistent with the approach taken by structuredClone and postMessage. 

And it's already a part of the WebCodecs spec. 

Can you clarify if this was in response to my questions or to Jonathan's?

[Eugene Zemtsov]

> Can you clarify if this was in response to my questions or to Jonathan's?

Yours.

Sorry, I missed Jonathan's question.

>  Once an ArrayBuffer is transferred and detached, any further access will result in some sort of TypeError, right?

Detached ArrayBuffers look like an empty ArrayBuffers. 

you can play with them using this code

let ab = new ArrayBuffer(100);
let ab2 = structuredClone(ab,  { transfer: [ab] })

ab is empty now

--

Thanks,

Eugene Zemtsov.

[Alex Russell]

Hey Eugene,

I'm a little worried that we're debating API shape here when there hasn't been any guidance from the TAG on design consistency. Have you either asked the TAG to weigh in (didn't see a review link in the Intent) or asked Chromium (ex)TAG members to give the API a once-over?

Best,

Alex

[Daniel Bratell]

In addition to Alex's question, I also noticed that you answered the web-platform-tests with a "no", which is a bit unexpected to me. Is there a reason this cannot or won't be tested in web-platform-tests?

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/ca4852cc-e0ab-4685-99d9-84d2f8316b90n%40chromium.org.

[Dale Curtis]

Alex, I assume you mean TAG's views on consistency regarding transfer ergonomics? Otherwise https://www.w3.org/TR/design-principles/#consistency is what we followed here. We have not asked, given that we felt this was a small performance improvement, with pre-existing ergonomics, and already has Media WG approval. We can certainly file a TAG request, but as you know, litigating minor features like this through TAG is unlikely to have a timely resolution.

Regarding Yoav's proposal above of a single boolean, that might make sense today where we have a single transfer, but we expect more input ArrayBuffers over time for some of these APIs, which would mean it becomes all-or-nothing for developers. E.g., we are likely to accept arrays of metadata, HDR data, etc. The boolean would mean they must transfer everything, which may lead to them making temporary copies of smaller buffers to get transfer effects on the larger ones.

Daniel, sorry, that's just an oversight in the chromestatus entry. There are tests added (here's the one for videoFrame):

https://chromium-review.googlesource.com/c/chromium/src/+/4529012/17/third_party/blink/web_tests/external/wpt/webcodecs/videoFrame-construction.any.js

- dale

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/fbe4d8ba-2d6a-f085-6608-25a2eeef6d22%40gmail.com.

[Eugene Zemtsov]

TAG review: https://github.com/w3ctag/design-reviews/issues/889

--

Thanks,

Eugene Zemtsov.

[Sangwhan Moon]

Thank you for filing this. If it is urgent, I could flag it as time constrained.

On Aug 31, 2023, at 7:14, 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:



To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF_jrf-1aRNk1AshPHDzUsiJeS3zoeuwXwuznZMpJxx_w%40mail.gmail.com.

[Dale Curtis]

I wouldn't say it's urgent, but I would hope for feedback within a week or two. Thanks!

- dale

[Yoav Weiss]

LGTM1 given the consistency here with other platform APIs and the need to extend this in the future.

[Chris Harrelson]

LGTM2

Would be great to file official vendor signals just so they know this is happening, if you're willing.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfX1Vd7t%3DpFtYTB0gojxEBL92Uaj73xt5o63FCTw5hRcZA%40mail.gmail.com.

[Mike Taylor]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw_xc8E0%2BrhP322Gaym3-yi1Z_k5y9po%2BM4OsW0CcRQW-A%40mail.gmail.com.