Intent to Ship: WebCodecs support for enabling AV1 screen content coding tools

564 views
Skip to first unread message

Philipp Hancke

unread,
Oct 20, 2023, 2:27:50 PM10/20/23
to blink-dev, Philipp Hancke, Bernard Aboba

Contact emails

pha...@microsoft.com, bernar...@microsoft.com


Explainer

None


Specification

https://www.w3.org/TR/webcodecs-av1-codec-registration/#dom-av1encoderconfig-forcescreencontenttools


Design docs

https://github.com/fippo/webrtc-explainers/blob/gh-pages/av1-screencontent/index.md


Summary

Adds AV1EncoderConfig (a dictionary containing forceScreenContentTools) to the VideoEncoderConfig along these lines:

encoder.configure({ codec: 'av01.0.04M.08', av1: { forceScreenContentTools: true}, width: 1920, height: 1080, bitrate: 2_000_000, framerate: 5, });


This allows an application to encode “screen content”, in particular presentation slides, in a more efficient way supported by the AV1 codec.

This material is typically static, often includes text, a limited set of colors, lots of repetitive content (e.g. straight lines, shapes) for which the encoder can optimize.

See the explainer for a lot of visual examples. This AV1 feature is already supported by WebRTC and enabled for screen sharing MediaStreamTracks so this increases platform consistency.



Blink component

Blink>Media>WebCodecs


TAG review

None


TAG review status

Not applicable


Risks



Interoperability and Compatibility

None


Gecko: Neutral similar to this situation https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364


WebKit: No signal (https://github.com/WebKit/standards-positions/issues/239)


Web developers: No signals Presented in the screen capture community group during TPAC 2023, slides here: https://docs.google.com/presentation/d/10i4HFYZ4CylpFUuoJcigfiI5uS7pK3uaseQpcYhugAY/edit#slide=id.g27ea3fad398_0_0


Other signals:


WebView application risks

Does this intent deprecate or change behavior of existing APIs, such that it has potentially high risk for Android WebView-based applications?

None


Debuggability

None


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes


Is this feature fully tested by web-platform-tests?

No

Unclear to what degree this is testable. Enabling this feature reduces the encoded frame size which makes it tough to test, it might be possible with a canvas source that provides predictable input similar to https://webrtc.github.io/samples/src/content/insertable-streams/video-processing/



Flag name on chrome://flags

None


Finch feature name

None


Non-finch justification

None


Requires code in //chrome?

False


Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1464862


Estimated milestones

Shipping on desktop

121


Anticipated spec changes

None


Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/6307770441138176


Links to previous Intent discussions

n/a

This intent message was generated by Chrome Platform Status.


Mike Taylor

unread,
Oct 25, 2023, 12:40:19 PM10/25/23
to Philipp Hancke, blink-dev, Philipp Hancke, Bernard Aboba

On 10/20/23 2:27 PM, 'Philipp Hancke' via blink-dev wrote:

Contact emails

pha...@microsoft.com, bernar...@microsoft.com


Explainer

None

For those looking for an explainer, hope down to the design docs section (nice explainer, btw!).


Specification

https://www.w3.org/TR/webcodecs-av1-codec-registration/#dom-av1encoderconfig-forcescreencontenttools


Design docs

https://github.com/fippo/webrtc-explainers/blob/gh-pages/av1-screencontent/index.md


Summary

Adds AV1EncoderConfig (a dictionary containing forceScreenContentTools) to the VideoEncoderConfig along these lines:

encoder.configure({ codec: 'av01.0.04M.08', av1: { forceScreenContentTools: true}, width: 1920, height: 1080, bitrate: 2_000_000, framerate: 5, });

This allows an application to encode “screen content”, in particular presentation slides, in a more efficient way supported by the AV1 codec.

This material is typically static, often includes text, a limited set of colors, lots of repetitive content (e.g. straight lines, shapes) for which the encoder can optimize.

See the explainer for a lot of visual examples. This AV1 feature is already supported by WebRTC and enabled for screen sharing MediaStreamTracks so this increases platform consistency.



Blink component

Blink>Media>WebCodecs


TAG review

None


TAG review status

Not applicable

An FYI review would be nice (especially given the discussions around per-frame/per-codec options), can you send one?


Risks



Interoperability and Compatibility

None


Gecko: Neutral similar to this situation https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364

Can we go ahead and request a position? It's not clear to me that Mozilla was present when the feature was discussed at https://lists.w3.org/Archives/Public/public-media-wg/2023Apr/0007.html.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADxkKiJc34b%3DMrX7HhXQP_gZ3wCFg43qHbLJ7mKQqOPe7WYVwA%40mail.gmail.com.

Philipp Hancke

unread,
Nov 1, 2023, 10:07:22 AM11/1/23
to Mike Taylor, blink-dev, Philipp Hancke, Bernard Aboba
Am Mi., 25. Okt. 2023 um 18:40 Uhr schrieb Mike Taylor <mike...@chromium.org>:

On 10/20/23 2:27 PM, 'Philipp Hancke' via blink-dev wrote:

Contact emails

pha...@microsoft.com, bernar...@microsoft.com


Explainer

None

For those looking for an explainer, hope down to the design docs section (nice explainer, btw!).


Specification

https://www.w3.org/TR/webcodecs-av1-codec-registration/#dom-av1encoderconfig-forcescreencontenttools


Design docs

https://github.com/fippo/webrtc-explainers/blob/gh-pages/av1-screencontent/index.md


Summary

Adds AV1EncoderConfig (a dictionary containing forceScreenContentTools) to the VideoEncoderConfig along these lines:

encoder.configure({ codec: 'av01.0.04M.08', av1: { forceScreenContentTools: true}, width: 1920, height: 1080, bitrate: 2_000_000, framerate: 5, });

This allows an application to encode “screen content”, in particular presentation slides, in a more efficient way supported by the AV1 codec.

This material is typically static, often includes text, a limited set of colors, lots of repetitive content (e.g. straight lines, shapes) for which the encoder can optimize.

See the explainer for a lot of visual examples. This AV1 feature is already supported by WebRTC and enabled for screen sharing MediaStreamTracks so this increases platform consistency.



Blink component

Blink>Media>WebCodecs


TAG review

None


TAG review status

Not applicable

An FYI review would be nice (especially given the discussions around per-frame/per-codec options), can you send one?


Risks



Interoperability and Compatibility

None


Gecko: Neutral similar to this situation https://github.com/mozilla/standards-positions/issues/837#issuecomment-1614666364

Can we go ahead and request a position? It's not clear to me that Mozilla was present when the feature was discussed at https://lists.w3.org/Archives/Public/public-media-wg/2023Apr/0007.html.

Looks like this was discussed at the editors meeting: https://github.com/w3c/webcodecs/pull/662#issuecomment-1526010045 but since those don't have notes... ;-)

Yoav Weiss

unread,
Nov 8, 2023, 5:16:42 AM11/8/23
to blink-dev, philipp...@googlemail.com, blink-dev, Philipp Hancke, Bernard Aboba, Mike Taylor
LGTM1 - seems like a simple and useful addition to the API surface.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Mike Taylor

unread,
Nov 8, 2023, 10:30:45 AM11/8/23
to Yoav Weiss, blink-dev, philipp...@googlemail.com, Philipp Hancke, Bernard Aboba

LGTM2

Chris Harrelson

unread,
Nov 8, 2023, 11:09:50 AM11/8/23
to Mike Taylor, Yoav Weiss, blink-dev, philipp...@googlemail.com, Philipp Hancke, Bernard Aboba
LGTM3

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/6c589524-5dba-4d8e-b94a-f594be3f4d83%40chromium.org.

Philipp Hancke

unread,
Dec 6, 2023, 11:31:46 AM12/6/23
to Chris Harrelson, Mike Taylor, Yoav Weiss, blink-dev, Philipp Hancke, Bernard Aboba
We ended up *not* shipping this here as the underlying feature ended up being integrated into contentHint by Eugene.
Which is another way to get consistency with WebRTC.

https://jsfiddle.net/fippo/r392ohwz/24 works great in Canary with a significantly reduced bitrate \o/
Reply all
Reply to author
Forward
0 new messages