chcunn...@google.com, sand...@google.com, eri...@google.com
Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.
Things to note
We would like to request an exemption from the origin trial breakage period. WebCodecs enables significant performance improvements for some OT registrants such that the breakage period would create a major regression for their users. This has not prevented us from making breaking changes to the API. See earlier discussion of breaking changes here.
The origin trial also included Media Capture Transform (a.k.a. Breakout Box) APIs from https://w3c.github.io/mediacapture-transform/. We are not seeking approval to ship Media Capture Transform in this thread.
Interoperability and Compatibility
Gecko: Positive, co-editing the spec (Paul Adenot)
Edge: Positive, co-editing the spec (Bernard Aboba)
WebKit: No signal
Expressed mixed interest/concerns.
More recently seeing increased WebKit engagement on github for WebCodecs and adjacent media APIs.
Web developers: Positive
Initial positive response to proposal: https://discourse.wicg.io/t/webcodecs-proposal/3662/
Positive response from several origin trial participants included at the bottom.
The proposed API shape enables the core use cases (see explainer) in a performant manner. We've left room for future optimizations and generally minimized complexity, even if that means we don't support all codec features. We intend for this shape to be friendly to both WASM and JS users.
Decoder outputs will typically be rendered with Canvas/WebGL and WebAudio (AudioWorklet). Developers have asked for this low level rendering control.
Encoder inputs will often come from getUserMedia() and getDisplayMedia() via the sister API: MediaStreamTrackProcessor.
WebCodecs would benefit from having libraries built on top that do things such as:
+ containerize (mux) and de-containerize (demux) to/from media file formats (e.g. mp4, webm).
+ render media with low latency from an unreliable media transport (in other words, a jitter buffer)
+ RTC client logic / signaling
WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs. Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability.
The implementation is thoroughly fuzz tested. There may be marginally increased fingerprinting surface due to support for detecting the presence of hardware encoder capabilities. This has been reviewed and deemed acceptable by the privacy sandbox team.
Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ/m/oyb3oFVaAAAJ
"Citrix plan to use this API primarily to offload graphically intensive CPU encoding & decoding operations to the GPU.
This ultimately will provide a better user experience for our HTML5 / Chrome App users, for example, higher frame rates during video playback or interactive graphics. Also, reducing CPU usage in graphics has positive knock on effects for other aspects of our solution, for example, better audio synchronization and smoother audio playback.
Going forwards, WebCodecs will be the preferred way for our encoding and decoding requirements on our HTML5 and Chrome App CWA."
"Roadmap for WebCodecs adoption @ Clipchamp: We are currently under an origin trial and have been closely tracking your progress in building out this feature. We are currently using WebCodecs (or are preparing to us it) in the following manner:
Video export: we have integrated WebCodecs encoding into our video encoding toolchain, based on a WebAssembly port of FFmpeg. This will target H.264 and be a drop-in replacement for our current (software-based) H.264 encoding approach. This is essentially ready to go. We plan to initially launch it to a small subset of users (under the origin trial) next week.
Interactive playback: we plan to replace the current (HTML <video> based) interactive playback/seeking of the video project's "timeline". That is, we will use WebCodecs-based decoding in combination with FFmpeg to implement a video playback and seeking widget that offers more granular control over a video's loading lifecycle and seeking behavior. Our prototyping indicates that this will greatly improve performance and robustness of the current approach - especially on low-end devices like Chromebooks. Work on this has started and I will meet with the team working on this today. We have decided to bring this work forward to address your feedback re performance of Clipchamp on Chromebooks.
Value-add for our users: adopting WebCodecs in the aforementioned areas of our stack will benefit our users in terms of (1) video export performance and (2) interaction performance on the timeline. And while these improvements should be felt on all platforms, we expect them to be especially pronounced on low-end devices, such as many entry-level Chromebooks. Other than that, WebCodecs will help us to normalize our stack between the (interactive) timeline interactions and the video export process, basing both on the aforementioned FFmpeg+WebCodecs stack. While not directly relevant for our end users, we can realistically expect knock-on effects from the unified programming model, benefitting our users. That is, our existing FFmpeg (software decoding) stack used to be too slow to replace the HTML <video> elements underpinning the timeline widget. This is going to change thanks to WebCodecs and will then allow us to avoid the challenges of using HTML <video> for timeline interactions (happy to provide more details here - please let me know).
Plan to continue using it: definitely! We are extremely happy with your team pushing to build and release WebCodecs. I can confidently say, it is a certainty at this point that we will prominently use it inside our stack. We have yet to learn the detailed characteristics across different platforms to determine the breadth of the initial rollout. That being said, we expect that we will immediately be able to use it for our default/draft export preset."
Eagle Eye Networks (EEN)
Google Docs is using ImageDecoder to reduce user-visible loading jank and simplify their architecture. They've found that, for docs with animated GIFs, ImageDecoder improves the input latency compared to using <img>. See this doc for their in depth analysis. They commit to using ImageDecoder immediately upon it shipping in Chrome.
"We require fast, random-access to decoded H.264 and AAC samples to provide browser-based editing and metadata logging workflows. Currently, this need is fulfilled by WebAssembly, however this has a high resource cost – particularly CPU. The WebCodecs API will give us the ability to target lower-end devices and/or increase the quality of the media we can support (higher resolutions, bit-rates, etc), which are things our customers have been demanding."
- James Pearce, Principal Software Engineer, Grass Valley.
"With the promise of The WebCodecs API, the browser evolves from just supporting video consumption, to a platform suitable for professional video production. We're really keen to get this to our customers, we have code ready to drop as soon as the API is released."
- Dr James Westland Cain, Chief Software Architect, Grass Valley
"Vysor is using the WebCodecs (and WebUSB) API to mirror and control Android+iOS devices in a browser/PWA. The availability of these new APIs have:
decreased video latency
allowed access to hardware accelerated decoding to improve performance and battery life
decreased the web application size by several megabytes by eliminating native and webassembly dependencies
consolidated code across the both native Electron app and web application
WebCodecs is an across the board massive improvement and Vysor's default decoder moving forward.
I'm extremely happy that the Chrome team delivered a 5 year old wish list request: https://twitter.com/koush/status/786278158794227712?s=20"
“Zoom is using the WebCodecs API for video encoding and decoding. The greater access to the host system hardware capabilities has improved the performance of our web client and enhanced the user experience with fast, clean video. We will continue to build on this work as we grow and evolve our real-time communication platform in the browser.”
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/855a38ff-3d67-422c-bdc8-697a5458c5ccn%40chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/d324666b-e599-8a47-92d2-74eb65890c8c%40gmail.com.
Reason this experiment is being extended
We did not reach cross UA consensus on blocking GitHub spec issues to ship in 92. Some areas of disagreement were discovered too late. We've since revamped our issue tracking / labels and are firmly aligned between spec co-editors on the outstanding work (labels: breaking + need-definition) and the proposed timeline to ship in 93. Having said that, we are requesting OT extension through 93, ending with release of 94 to allow for additional room.