Intent to Ship: WebCodecs

[Chris Cunningham]

Contact emails

chcunn...@google.com, sand...@google.com, eri...@google.com

Explainer

https://github.com/w3c/webcodecs/blob/main/explainer.md 

Specification

https://w3c.github.io/webcodecs/

Design docs

https://docs.google.com/document/d/1-AwmoXL0Cg1ZFGLHchjS48_dxgezHqmXW3YRYtcF53s

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

Blink component

Blink>Media>WebCodecs

TAG reviews

https://github.com/w3ctag/design-reviews/issues/612

https://github.com/w3ctag/design-reviews/issues/433

TAG review status

Feedback addressed. Awaiting final approval. 

Risks

Things to note

We would like to request an exemption from the origin trial breakage period. WebCodecs enables significant performance improvements for some OT registrants such that the breakage period would create a major regression for their users. This has not prevented us from making breaking changes to the API. See earlier discussion of breaking changes here.

The origin trial also included Media Capture Transform (a.k.a. Breakout Box) APIs from https://w3c.github.io/mediacapture-transform/. We are not seeking approval to ship Media Capture Transform in this thread. 

Interoperability and Compatibility

Gecko: Positive, co-editing the spec (Paul Adenot)

https://github.com/mozilla/standards-positions/issues/209

Edge: Positive, co-editing the spec (Bernard Aboba)

WebKit: No signal

Expressed mixed interest/concerns.
https://lists.webkit.org/pipermail/webkit-dev/2020-May/031191.html

https://lists.webkit.org/pipermail/webkit-dev/2021-February/031691.html 

More recently seeing increased WebKit engagement on github for WebCodecs and adjacent media APIs. 

Web developers: Positive 

Initial positive response to proposal: https://discourse.wicg.io/t/webcodecs-proposal/3662/ 

Positive response from several origin trial participants included at the bottom.

Ergonomics

The proposed API shape enables the core use cases (see explainer) in a performant manner. We've left room for future optimizations and generally minimized complexity, even if that means we don't support all codec features. We intend for this shape to be friendly to both WASM and JS users. 

Decoder outputs will typically be rendered with Canvas/WebGL and WebAudio (AudioWorklet). Developers have asked for this low level rendering control.

Encoder inputs will often come from getUserMedia() and getDisplayMedia() via the sister API: MediaStreamTrackProcessor. 

https://w3c.github.io/mediacapture-transform/#track-processor 

Activation

WebCodecs would benefit from having libraries built on top that do things such as: 

+ containerize (mux) and de-containerize (demux) to/from media file formats (e.g. mp4, webm).

+ render media with low latency from an unreliable media transport (in other words, a jitter buffer)

+ RTC client logic / signaling 

WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs. Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability.

Security

The implementation is thoroughly fuzz tested. There may be marginally increased fingerprinting surface due to support for detecting the presence of hardware encoder capabilities. This has been reviewed and deemed acceptable by the privacy sandbox team.

Is this feature fully tested by web-platform-tests?

Yes. https://github.com/web-platform-tests/wpt/tree/master/webcodecs 

Flag name

--enable-blink-features=WebCodecs

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=897297

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1097106

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5669293909868544

Links to previous Intent discussions

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ/m/oyb3oFVaAAAJ

https://groups.google.com/a/chromium.org/g/blink-dev/c/55QViQdz5Rc/m/UxytBsZ5AgAJ (extend)

https://groups.google.com/a/chromium.org/g/blink-dev/c/oisKIPVdLQw/m/KAikvxDQAQAJ (extend)

Statements from origin trial participants

Citrix 

https://citrix.com

"Citrix plan to use this API primarily to offload graphically intensive CPU encoding & decoding operations to the GPU.

This ultimately will provide a better user experience for our HTML5 / Chrome App users, for example, higher frame rates during video playback or interactive graphics. Also, reducing CPU usage in graphics has positive knock on effects for other aspects of our solution, for example, better audio synchronization and smoother audio playback.

Going forwards, WebCodecs will be the preferred way for our encoding and decoding requirements on our HTML5 and Chrome App CWA."

Clipchamp 

https://clipchamp.com

"Roadmap for WebCodecs adoption @ Clipchamp: We are currently under an origin trial and have been closely tracking your progress in building out this feature. We are currently using WebCodecs (or are preparing to us it) in the following manner:

•  Video export: we have integrated WebCodecs encoding into our video encoding toolchain, based on a WebAssembly port of FFmpeg. This will target H.264 and be a drop-in replacement for our current (software-based) H.264 encoding approach. This is essentially ready to go. We plan to initially launch it to a small subset of users (under the origin trial) next week. 

• Interactive playback: we plan to replace the current (HTML <video> based) interactive playback/seeking of the video project's "timeline". That is, we will use WebCodecs-based decoding in combination with FFmpeg to implement a video playback and seeking widget that offers more granular control over a video's loading lifecycle and seeking behavior. Our prototyping indicates that this will greatly improve performance and robustness of the current approach - especially on low-end devices like Chromebooks. Work on this has started and I will meet with the team working on this today. We have decided to bring this work forward to address your feedback re performance of Clipchamp on Chromebooks.

Value-add for our users: adopting WebCodecs in the aforementioned areas of our stack will benefit our users in terms of (1) video export performance and (2) interaction performance on the timeline. And while these improvements should be felt on all platforms, we expect them to be especially pronounced on low-end devices, such as many entry-level Chromebooks. Other than that, WebCodecs will help us to normalize our stack between the (interactive) timeline interactions and the video export process, basing both on the aforementioned FFmpeg+WebCodecs stack. While not directly relevant for our end users, we can realistically expect knock-on effects from the unified programming model, benefitting our users. That is, our existing FFmpeg (software decoding) stack used to be too slow to replace the HTML <video>  elements underpinning the timeline widget. This is going to change thanks to WebCodecs and will then allow us to avoid the challenges of using HTML <video> for timeline interactions (happy to provide more details here - please let me know).

Plan to continue using it: definitely! We are extremely happy with your team pushing to build and release WebCodecs. I can confidently say, it is a certainty at this point that we will prominently use it inside our stack. We have yet to learn the detailed characteristics across different platforms to determine the breadth of the initial rollout. That being said, we expect that we will immediately be able to use it for our default/draft export preset."

Eagle Eye Networks (EEN) 

https://een.com/

"EEN is a cloud-based video surveillance company streaming low latency live video in the browser. The WebCodec API allows us to develop a low latency live player, without browser plugins, with full control over video buffering and playback with a simple and flexible Javascript API. The value for our users is that live video will load faster, with improved error handling (e.g. error due to decoding delay) and with support for all common video/audio codecs. We will ship a video player based on the WebCodecs API as soon as the feature flag is removed in Chrome. This player will replace the HLS based player."

Google Docs

https://docs.google.com/ 

Google Docs is using ImageDecoder to reduce user-visible loading jank and simplify their architecture. They've found that, for docs with animated GIFs, ImageDecoder improves the input latency compared to using <img>. See this doc for their in depth analysis. They commit to using ImageDecoder immediately upon it shipping in Chrome.

Grass Valley

http://grassvalley.com/

"We require fast, random-access to decoded H.264 and AAC samples to provide browser-based editing and metadata logging workflows. Currently, this need is fulfilled by WebAssembly, however this has a high resource cost – particularly CPU. The WebCodecs API will give us the ability to target lower-end devices and/or increase the quality of the media we can support (higher resolutions, bit-rates, etc), which are things our customers have been demanding."

- James Pearce, Principal Software Engineer, Grass Valley.

"With the promise of  The WebCodecs API, the browser evolves from just supporting video consumption, to a platform suitable for professional video production. We're really keen to get this to our customers, we have code ready to drop as soon as the API is released."

- Dr James Westland Cain, Chief Software Architect, Grass Valley

Vysor 

https://www.vysor.io

"Vysor is using the WebCodecs (and WebUSB) API to mirror and control Android+iOS devices in a browser/PWA. The availability of these new APIs have: 

• decreased video latency

• allowed access to hardware accelerated decoding to improve performance and battery life

• decreased the web application size by several megabytes by eliminating native and webassembly dependencies

• consolidated code across the both native Electron app and web application

WebCodecs is an across the board massive improvement and Vysor's default decoder moving forward.

I'm extremely happy that the Chrome team delivered a 5 year old wish list request: https://twitter.com/koush/status/786278158794227712?s=20"

Zoom

https://zoom.com 

“Zoom is using the WebCodecs API for video encoding and decoding. The greater access to the host system hardware capabilities has improved the performance of our web client and enhanced the user experience with fast, clean video. We will continue to build on this work as we grow and evolve our real-time communication platform in the browser.” 

[Yoav Weiss]

LGTM1 to ship without OT breakage period. Looking forward to faster media processing on the web!! :)

Thanks for the OT testimonies. That's super helpful! 

[Daniel Bratell]

LGTM2

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/855a38ff-3d67-422c-bdc8-697a5458c5ccn%40chromium.org.

[Mike West]

LGTM3. Thank you for iterating on this to get to a state in which Mozilla's on board.

-mike

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d324666b-e599-8a47-92d2-74eb65890c8c%40gmail.com.

[Jan-Ivar Bruaroey]

Mozilla is NOT on board with shipping WebCodecs in its current state, because the current state would expose WebCodecs APIs on Window (main-thread).

Mozilla considers the following issue to be blocking:

• https://github.com/w3c/webcodecs/issues/211 - Should WebCodecs be exposed in Window environments?

This issue was hotly debated in the Media WG just last week: https://www.w3.org/2021/06/15-mediawg-minutes.html#t01 — with Apple and Mozilla opposed to such exposure, preferring to limit exposure to workers for now.

Once such exposure has shipped, it cannot be undone.

The Media WG is supposed to issue a "Call for Consensus" on #211 this week. I propose resetting/holding off on LGTMs until that CfC has concluded and the specification updated to reflect Media WG consensus. Otherwise, any support for this "Intent to Ship" may be misconstrued as support for main thread exposure (which I assume Chrome's implementation consists of, and MAY consist of at time of shipping) leaving us no choice but to withhold support. We encourage Apple to do the same.

> LGTM3. Thank you for iterating on this to get to a state in which Mozilla's on board.

mk...@chromium.org you may wish to withdraw your LGTM3 if it is based on the impression of Mozilla's support at this time.

[Dale Curtis]

Hi Jan-Ivar, can you please sync with Paul internally? Chris had already approved this I2S with Paul @ Mozilla with the understanding that we wouldn't ship without the comment you link being resolved.

- dale

[Jan-Ivar Bruaroey]

I did sync with Paul beforehand, so it's possible there was some miscommunication. Glad to hear there is such an understanding, since I heard from other parties who were confused by this as well.

For the benefit of all parties involved, I think we'd prefer this be explicitly acknowledged in the "Intent to Ship". That way release owners know the commitments they're signing onto as well.

Thanks,

[Dale Curtis]

This was covered in a previous thread and linked above ("See discussion of breaking changes here ..."), but could have been called out better in this message. The understanding reached was that we wouldn't ship until these sets of issues reached consensus:

Reason this experiment is being extended

We did not reach cross UA consensus on blocking GitHub spec issues to ship in 92. Some areas of disagreement were discovered too late. We've since revamped our issue tracking / labels and are firmly aligned between spec co-editors on the outstanding work (labels: breaking + need-definition) and the proposed timeline to ship in  93. Having said that, we are requesting OT extension through 93, ending with release of 94 to allow for additional room. 

 

- dale 

[Jan-Ivar Bruaroey]

Thanks Dale, I'm glad to hear such understanding was reached, except I'm not seeing https://github.com/w3c/webcodecs/issues/211 listed in what you quote. Chris, would you mind confirming?

Sorry to be so pedantic about this,

[Dale Curtis]

211 is listed under: https://github.com/w3c/webcodecs/issues?q=is%3Aissue+is%3Aopen+label%3Abreaking

- dale

[Paul Adenot]

All,

I indeed specifically said to Chris Cunningham, during one of our weekly editor calls, that I thought this list was doable in the allotted time and that I agreed with its content, but that of course consensus would have to be achieved, so I guess we all agree here.

Sorry for the mis-communication, I'm just back from a week of PTO and wasn't looking at my Mozilla email when this was posted.

Thanks,

Paul.

[Dale Curtis]

The chairs of the media WG have ruled in favor of window exposure:

https://lists.w3.org/Archives/Public/public-media-wg/2021Jul/0017.html

All "breaking" and "need-definition" issues have PRs assigned and agreement on direction. After reviewing the issues in today's editor's call, Paul from Mozilla and Bernard from Microsoft reaffirmed their approval of our intent to ship. As such, we'll be shipping WebCodecs in M94 (which reaches stable ~Sep 24th) -- we'll be flipping the flag by the end of the week.

Thanks to our editors (Chris Cunningham, Paul Adenot, and Bernard Aboba) for bringing the spec together. Thanks to the Media WG (especially our chairs Jer Noble and Chris Needham) for feedback and helping guide discussions. Thanks to Chrome's implementation team (Dan Sanders, Thomas Guilbert, and Eugene Zemtsov). Finally, thanks to our partners and the many others who participated in the issue tracker, integration with other standards, and code reviews.

- dale