Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

[Eugene Zemtsov]

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Blink component

Blink>Media>WebCodecs

TAG review

None. 

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612 

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Neutral (https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364) Paul Adenot (Mozilla) expressed that minor changes to WebCodecs spec don't need to go through the full "Request for Mozilla Position" process assuming they were approved by the Media Working Group.

WebKit: Positive (https://www.w3.org/2023/03/07-mediawg-minutes.html#t02) The issue was discussed on 07 March 2023 by w3c Media working group. Jer Noble (Apple) was actively participating and provided input for spec details.

Web developers: Positive (https://github.com/w3c/webcodecs/issues/56) People ask for this on GitHub

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1424154

Estimated milestones

Shipping on desktop

117

Shipping on Android

117

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5783986600673280

--

Thanks,

Eugene Zemtsov.

[Mike Taylor]

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Assuming I know very little about video codecs, what use cases does this enable for developers?

Blink component

Blink>Media>WebCodecs

TAG review

None. 

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612

This is a new addition since that review, but it's a trivial addition (IMHO) so a new review request isn't needed.

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Neutral (https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364) Paul Adenot (Mozilla) expressed that minor changes to WebCodecs spec don't need to go through the full "Request for Mozilla Position" process assuming they were approved by the Media Working Group.

WebKit: Positive (https://www.w3.org/2023/03/07-mediawg-minutes.html#t02) The issue was discussed on 07 March 2023 by w3c Media working group. Jer Noble (Apple) was actively participating and provided input for spec details.

Can we request a formal position from WebKit, at least to let them know we're intending to ship?

Web developers: Positive (https://github.com/w3c/webcodecs/issues/56) People ask for this on GitHub

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1424154

Estimated milestones

Shipping on desktop

117

Shipping on Android

117

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5783986600673280

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

[Eugene Zemtsov]

>  what use cases does this enable for developers?

Controlling the quantizer for every frame makes possible at least two new use cases for VideoEncoder:

1. Constant quality encoding when complex frames and parts get more bandwidth than less complex ones.

2. External rate control algorithms implemented by web developers. Such algorithms can take into account a website's knowledge about contents of each frame or quick changes in available network bandwidth.

> Can we request a formal position from WebKit, at least to let them know we're intending to ship?

https://github.com/WebKit/standards-positions/issues/223

--

Thanks,

Eugene Zemtsov.

[Sangwhan Moon]

(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Assuming I know very little about video codecs, what use cases does this enable for developers?

Blink component

Blink>Media>WebCodecs

TAG review

None. 

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612

This is a new addition since that review, but it's a trivial addition (IMHO) so a new review request isn't needed.

I agree, I don't see anything particularly contentious in the proposal that would warrant a full review.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

[Eugene Zemtsov]

Any new feedback or resolution on this one? 

--

Thanks,

Eugene Zemtsov.

[Yoav Weiss]

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:

Any new feedback or resolution on this one? 

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:

(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.

Could you write one or add an inline explanation outlining that?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

[Philipp Hancke]

Am Mi., 19. Juli 2023 um 14:25 Uhr schrieb Yoav Weiss <yoav...@chromium.org>:

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:

Any new feedback or resolution on this one? 

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:

(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.

Could you write one or add an inline explanation outlining that?

Explaining that is tough without going into the details "what is quantization for video codecs. See

  https://www.vcodex.com/h264avc-4x4-transform-and-quantization/

for a very detailed explanation for H264.

the tl;dr is that folks who encode video like tuning all kinds of knobs to get the "best" result and qp is one of those.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

[Yoav Weiss]

On Wed, Jul 19, 2023 at 2:51 PM Philipp Hancke <philipp...@googlemail.com> wrote:

Am Mi., 19. Juli 2023 um 14:25 Uhr schrieb Yoav Weiss <yoav...@chromium.org>:

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:

Any new feedback or resolution on this one? 

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:

(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.

Could you write one or add an inline explanation outlining that?

Explaining that is tough without going into the details "what is quantization for video codecs. See

  https://www.vcodex.com/h264avc-4x4-transform-and-quantization/

for a very detailed explanation for H264.

the tl;dr is that folks who encode video like tuning all kinds of knobs to get the "best" result and qp is one of those.

An explainer doesn't have to assume folks reading it don't know what quantization means :)

Basically, clicking through the specs, it's still not clear to me if the quantization values are provided as a single int that quantization tables are supposed to be divided by. a "quality" int that represents a certain quantization table, or something else entirely. Clarity on that would be great.

[Alex Russell]

I *think* I grok what this is for, and I'm still pretty frustrated that there isn't a crisp summary along the lines of "this parameter helps sites implement custom bitrate vs. quality vs. CPU use tradeoffs for different kinds of media and streams, which are important to customers like X, Y, and Z".

Eugene, Philipp: it's important that the Blink process show that we are shipping important features that solve real problems, particularly when we're in the position of shipping first. We *want* to trust the WebRTC/media community to work with us to launch API changes quickly, and demonstrating need is part of that. Can you respond with an overview (perhaps in the form of an Explainer), and/or perhaps have potential users of this API chime in?

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Eugene Zemtsov]

Thanks for the feedback. 

I put together an explainer and linked it on the ChromeStatus feature page: 

https://gist.github.com/Djuffin/3722232679b977058be787be0dff4254 

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

[Yoav Weiss]

LGTM1

Thanks for the explainer!!

[Daniel Bratell]

LGTM2

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

[Chris Harrelson]

LGTM3. Thank you!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9e95972f-14de-2195-7723-d6c16a0a1f89%40gmail.com.

[Mike Taylor]

Thanks Eugene for the additional explainer text - and thanks to Alex for encouraging more work here.

[Alex Russell]

+1; thank you, Eugene.

LGTM3. Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Bernard Aboba]

For the WEBRTC/MEDIA WG joint meeting at TPAC, Eugene has put together some slides and a demo of QP-based rate control: 

https://docs.google.com/presentation/d/1FpCAlxvRuC0e52JrthMkx-ILklB5eHszbk8D3FIuSZ0/edit#slide=id.g2397bc70323_0_17

As noted in the graph, "actual" rate remains close to the "desired" rate (e.g. the target rate that might be provided in a Bandwidth Estimate) both for ramp-up and ramp-down, which should translate to better video quality.  For example, see: https://snr.stanford.edu/salsify/

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.