Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Eugene Zemtsov

unread,

Jul 6, 2023, 5:05:19 AM7/6/23

to blin...@chromium.org, Dale Curtis, Thomas Guilbert

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Blink component

Blink>Media>WebCodecs

TAG review

None.

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Neutral (https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364) Paul Adenot (Mozilla) expressed that minor changes to WebCodecs spec don't need to go through the full "Request for Mozilla Position" process assuming they were approved by the Media Working Group.

WebKit: Positive (https://www.w3.org/2023/03/07-mediawg-minutes.html#t02) The issue was discussed on 07 March 2023 by w3c Media working group. Jer Noble (Apple) was actively participating and provided input for spec details.

Web developers: Positive (https://github.com/w3c/webcodecs/issues/56) People ask for this on GitHub

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1424154

Estimated milestones

Shipping on desktop

117

Shipping on Android

117

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5783986600673280

--

Thanks,

Eugene Zemtsov.

Mike Taylor

unread,

Jul 6, 2023, 11:32:21 AM7/6/23

to Eugene Zemtsov, blin...@chromium.org, Dale Curtis, Thomas Guilbert

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Assuming I know very little about video codecs, what use cases does this enable for developers?

Blink component

Blink>Media>WebCodecs

TAG review

None.

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612

This is a new addition since that review, but it's a trivial addition (IMHO) so a new review request isn't needed.

TAG review status

Not applicable

Risks

Interoperability and Compatibility

Gecko: Neutral (https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364) Paul Adenot (Mozilla) expressed that minor changes to WebCodecs spec don't need to go through the full "Request for Mozilla Position" process assuming they were approved by the Media Working Group.

WebKit: Positive (https://www.w3.org/2023/03/07-mediawg-minutes.html#t02) The issue was discussed on 07 March 2023 by w3c Media working group. Jer Noble (Apple) was actively participating and provided input for spec details.

Can we request a formal position from WebKit, at least to let them know we're intending to ship?

Web developers: Positive (https://github.com/w3c/webcodecs/issues/56) People ask for this on GitHub

Other signals:

WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1424154

Estimated milestones

Shipping on desktop

117

Shipping on Android

117

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5783986600673280

--

Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

Eugene Zemtsov

unread,

Jul 6, 2023, 7:12:13 PM7/6/23

to Mike Taylor, blin...@chromium.org, Dale Curtis, Thomas Guilbert

> what use cases does this enable for developers?

Controlling the quantizer for every frame makes possible at least two new use cases for VideoEncoder:
1. Constant quality encoding when complex frames and parts get more bandwidth than less complex ones.
2. External rate control algorithms implemented by web developers. Such algorithms can take into account a website's knowledge about contents of each frame or quick changes in available network bandwidth.

> Can we request a formal position from WebKit, at least to let them know we're intending to ship?

https://github.com/WebKit/standards-positions/issues/223

--

Thanks,

Eugene Zemtsov.

Sangwhan Moon

unread,

Jul 7, 2023, 8:53:39 AM7/7/23

to Mike Taylor, Eugene Zemtsov, blin...@chromium.org, Dale Curtis, Thomas Guilbert

(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

Specification

https://www.w3.org/TR/webcodecs/#video-encoder-bitrate-mode

Summary

Adds "quantizer" VideoEncoderBitrateMode for VideoEncoder. This allows to specify a quantizer parameter for each frame for AV1, VP9, and AVC video codecs. The quantizer parameter is set via codec specific extensions for VideoEncoderEncodeOptions.

Assuming I know very little about video codecs, what use cases does this enable for developers?

Blink component

Blink>Media>WebCodecs

TAG review

None.

Previously WebCodecs API had TAG review as a whole:

https://github.com/w3ctag/design-reviews/issues/612

This is a new addition since that review, but it's a trivial addition (IMHO) so a new review request isn't needed.

I agree, I don't see anything particularly contentious in the proposal that would warrant a full review.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

Eugene Zemtsov

unread,

Jul 13, 2023, 4:53:03 PM7/13/23

to blin...@chromium.org, Mike Taylor, Dale Curtis, Thomas Guilbert, Sangwhan Moon

Any new feedback or resolution on this one?

--

Thanks,

Eugene Zemtsov.

Yoav Weiss

unread,

Jul 19, 2023, 8:25:14 AM7/19/23

to Eugene Zemtsov, blin...@chromium.org, Mike Taylor, Dale Curtis, Thomas Guilbert, Sangwhan Moon

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:

Any new feedback or resolution on this one?

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:
(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.

Could you write one or add an inline explanation outlining that?

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

Philipp Hancke

unread,

Jul 19, 2023, 8:52:04 AM7/19/23

to Yoav Weiss, Eugene Zemtsov, blin...@chromium.org, Mike Taylor, Dale Curtis, Thomas Guilbert, Sangwhan Moon

Am Mi., 19. Juli 2023 um 14:25 Uhr schrieb Yoav Weiss <yoav...@chromium.org>:

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:
Any new feedback or resolution on this one?

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:
(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.
Could you write one or add an inline explanation outlining that?

Explaining that is tough without going into the details "what is quantization for video codecs. See

https://www.vcodex.com/h264avc-4x4-transform-and-quantization/

for a very detailed explanation for H264.

the tl;dr is that folks who encode video like tuning all kinds of knobs to get the "best" result and qp is one of those.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

Yoav Weiss

unread,

Jul 19, 2023, 11:14:16 AM7/19/23

to Philipp Hancke, Eugene Zemtsov, blin...@chromium.org, Mike Taylor, Dale Curtis, Thomas Guilbert, Sangwhan Moon

On Wed, Jul 19, 2023 at 2:51 PM Philipp Hancke <philipp...@googlemail.com> wrote:

Am Mi., 19. Juli 2023 um 14:25 Uhr schrieb Yoav Weiss <yoav...@chromium.org>:

On Thu, Jul 13, 2023 at 10:53 PM 'Eugene Zemtsov' via blink-dev <blin...@chromium.org> wrote:
Any new feedback or resolution on this one?

On Fri, Jul 7, 2023 at 5:53 AM Sangwhan Moon <s...@chromium.org> wrote:
(resending from correct email)

On 2023年07月07日 00時32分12秒 (+09:00), Mike Taylor wrote:

On 7/5/23 8:57 PM, 'Eugene Zemtsov' via blink-dev wrote:

Intent to Implement and Ship: Per-frame quantizer in VideoEncoder

Contact emails

ezem...@google.com

Explainer

None

I think an explainer can be significantly helpful in helping us understand how developers will be using this feature and what use cases it'd cover.
Could you write one or add an inline explanation outlining that?

Explaining that is tough without going into the details "what is quantization for video codecs. See
https://www.vcodex.com/h264avc-4x4-transform-and-quantization/
for a very detailed explanation for H264.

the tl;dr is that folks who encode video like tuning all kinds of knobs to get the "best" result and qp is one of those.

An explainer doesn't have to assume folks reading it don't know what quantization means :)

Basically, clicking through the specs, it's still not clear to me if the quantization values are provided as a single int that quantization tables are supposed to be divided by. a "quality" int that represents a certain quantization table, or something else entirely. Clarity on that would be great.

Alex Russell

unread,

Jul 19, 2023, 11:57:35 AM7/19/23

to blink-dev, Yoav Weiss, Eugene Zemtsov, blin...@chromium.org, Mike Taylor, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

I *think* I grok what this is for, and I'm still pretty frustrated that there isn't a crisp summary along the lines of "this parameter helps sites implement custom bitrate vs. quality vs. CPU use tradeoffs for different kinds of media and streams, which are important to customers like X, Y, and Z".

Eugene, Philipp: it's important that the Blink process show that we are shipping important features that solve real problems, particularly when we're in the position of shipping first. We *want* to trust the WebRTC/media community to work with us to launch API changes quickly, and demonstrating need is part of that. Can you respond with an overview (perhaps in the form of an Explainer), and/or perhaps have potential users of this API chime in?

Best,

Alex

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--
Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Eugene Zemtsov

unread,

Jul 19, 2023, 7:19:00 PM7/19/23

to Alex Russell, blink-dev, Yoav Weiss, Mike Taylor, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

Thanks for the feedback.

I put together an explainer and linked it on the ChromeStatus feature page:

https://gist.github.com/Djuffin/3722232679b977058be787be0dff4254

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--
Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,

Eugene Zemtsov.

Yoav Weiss

unread,

Jul 19, 2023, 11:53:50 PM7/19/23

to Eugene Zemtsov, Alex Russell, blink-dev, Mike Taylor, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

LGTM1

Thanks for the explainer!!

Daniel Bratell

unread,

Jul 20, 2023, 8:46:28 AM7/20/23

to Yoav Weiss, Eugene Zemtsov, Alex Russell, blink-dev, Mike Taylor, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

LGTM2

/Daniel

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

Chris Harrelson

unread,

Jul 20, 2023, 10:04:54 AM7/20/23

to Daniel Bratell, Yoav Weiss, Eugene Zemtsov, Alex Russell, blink-dev, Mike Taylor, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

LGTM3. Thank you!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/9e95972f-14de-2195-7723-d6c16a0a1f89%40gmail.com.

Mike Taylor

unread,

Jul 24, 2023, 2:24:29 PM7/24/23

to Chris Harrelson, Daniel Bratell, Yoav Weiss, Eugene Zemtsov, Alex Russell, blink-dev, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com

Thanks Eugene for the additional explainer text - and thanks to Alex for encouraging more work here.

Alex Russell

unread,

Jul 26, 2023, 11:45:38 AM7/26/23

to blink-dev, Mike Taylor, Yoav Weiss, Eugene Zemtsov, Alex Russell, blink-dev, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com, Chris Harrelson, Daniel Bratell

+1; thank you, Eugene.

LGTM3. Thank you!

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Bernard Aboba

unread,

Sep 5, 2023, 6:41:34 PM9/5/23

to blink-dev, Alex Russell, Mike Taylor, Yoav Weiss, Eugene Zemtsov, blink-dev, Dale Curtis, tgui...@google.com, Sangwhan Moon, philipp...@googlemail.com, Peter Thatcher

For the WEBRTC/MEDIA WG joint meeting at TPAC, Eugene has put together some slides and a demo of QP-based rate control:

https://docs.google.com/presentation/d/1FpCAlxvRuC0e52JrthMkx-ILklB5eHszbk8D3FIuSZ0/edit#slide=id.g2397bc70323_0_17

As noted in the graph, "actual" rate remains close to the "desired" rate (e.g. the target rate that might be provided in a Bandwidth Estimate) both for ramp-up and ramp-down, which should translate to better video quality. For example, see: https://snr.stanford.edu/salsify/

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrF0PKfpVbaYWX-hR0wJ%2Bb9H4YtwFBUc6Y6JGSmFT7pVgQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/0a1d0f96-c5f0-fdf8-44b0-b49c3f180839%40chromium.org.

--

Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAK8JDrFxVaQnqU4dGQesuw%2B_AaBV763a0-t%3Dp0273LPNbyespQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfUBdFesjccKhgcoQ5Zs%2BJBEqMtNjFthN2UpVsUF_hEC4w%40mail.gmail.com.

--

Thanks,
Eugene Zemtsov.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAL5BFfWmDomb-KRB2kXns%3DxTkdMdYE4WH8twPKXTVVCywNNtog%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Reply all

Reply to author

Forward