Intent to Implement WebCodecs

[Peter Thatcher]

Intent to Implement: WebCodecs

Contact emails

ptha...@google.com, sand...@google.com, chcunn...@google.com 

Explainer

https://github.com/WICG/web-codecs/blob/master/explainer.md

Design doc/Spec

No spec or design doc yet, but there is some rough WebIDL at https://github.com/WICG/web-codecs/blob/master/webcodecs.idl.

TAG review

https://github.com/w3ctag/design-reviews/issues/433

Summary

Efficient, low-level access to built-in (software and hardware) media encoders and decoders.

Motivation

Existing media APIs (HTMLMediaElement, Media Source Extensions, WebAudio, MediaRecorder, WebRTC) are high-level and narrowly-focused. A low-level codec API will better support emerging applications, such as latency-sensitive game streaming, client-side effects or transcoding, and polyfillable media container support, without the increased network and CPU cost of JavaScript or WebAssembly codec implementations.

Risks

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

Edge: Public support  (co-editing the spec)

Firefox: Public support (more specifically, "under consideration" more details at https://github.com/mozilla/standards-positions/issues/209 )

Safari: Negative signals (concerns over keeping it fingerprinting neutral)

Web / Framework developers: Positive (https://discourse.wicg.io/t/webcodecs-proposal/3662/)

Wikipedia: https://discourse.wicg.io/t/webcodecs-proposal/3662/9 

Twitch: https://discourse.wicg.io/t/webcodecs-proposal/3662/12

Sony: https://discourse.wicg.io/t/webcodecs-proposal/3662/13

Ergonomics

Decoders will frequently be used with HtmlMediaElement for rendering media.  Making sure it easy to render easily, performantly, and with low latency is important.

Many developers have requested that WebCodecs be easily used from WASM.  Making sure that works easily, performantly, and with low latency is important.

Developers have requested that video decoders be easily used with WebGL for using decoded video frames on WebGL textures.  Making sure that works easily, performantly, and with low latency is important.

Activation

WebCodecs would benefit from having libraries built on top that do things such as:

• Encode and then containerize (mux) encoded media into formats such as mp4

• Decontainerize (demux) and then decode media from formats as HLS

• Buffer, decode, and playout media with low latency  from an unreliable media transport (in other words, a jitter buffer)

WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs.

Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability

Security

There may be increased fingerprinting surface 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Link to entry on the Chrome Platform Status

https://www.chromestatus.com/feature/5669293909868544

Requesting approval to ship?

No

[Daniel Bratell]

This is likely to amplify the codec fragmentation that already troubles the web. What are your thoughts on that?

/Daniel

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAJrXDUE_E0P4GbwwRfWA1TtL1QTp65C2pDaHrffsN%3D2RjScmsg%40mail.gmail.com.

[Boris Zbarsky]

On 10/21/19 9:38 PM, 'Peter Thatcher' via blink-dev wrote:
> Firefox: Public support (more specifically, "under consideration" more
> details at https://github.com/mozilla/standards-positions/issues/209

> <https://github.com/mozilla/standards-positions/issues/209>)

"under consideration" means "there is no position yet; we are figuring
it out". It explicitly does NOT mean "public support" nor indeed any
implication of support at all. I means the issue is in the first stage
of a process that will end up with one of the other positions listed at
the bottom of <https://github.com/mozilla/standards-positions>.

-Boris

[Peter Thatcher]

Why would it amplify codec fragmentation?  The set of codecs that will be supported will be the same set as it supported by existing web APIs.  

[Daniel Bratell]

By exposing more low level information, I think that there will be more diversion points which will further reduce the chance of web developers having the time and energy to cover all variants. It is also plausible that a browser may support different sets of codecs on different abstraction levels. For instance, a platform might have APIs to get video data from a stream to the screen, but lack the ability to get individual pixels.

Since the codec fragmentation is a one of the web's weak spots, I wonder if you have considered the implications. You have probably thought more about video decoding than I have recently.

/Daniel

[bernar...@gmail.com]

A similar clarification with respect to the position of the Edge team.  

Edge is supportive of the incubation of WebCodecs within WICG and is willing to contribute to the development of the specification. Since WebCodecs offers unified handling of media on the web (in both streaming as well as realtime scenarios), we believe it has the potential to empower developers to do things that are not possible in today's architecture, as well as potentially improving the reliability and maintainability of web media by allowing innovation (and bug fixing) to occur outside native browser implementations.

However, WebCodecs is still relatively immature compared with WebTransport and so in the absence of a specification and broader review it would be premature for us to take a formal "position".   

Bernard Aboba

Microsoft

[Harald Alvestrand]

I didn't think of the fragmentation issue before this, but thinking about it for 1 minute, I think there are multiple sides to the coin here.

We have fragmentation at the platform, browser and browser version level today - because of differing hardware capabilities, and differing software capabilities - and the developer has no tools to deal with it except "negotiate to the common subset".

The larger exposed control surface, and the ability to apply software tweaks before/after the codec, allows another layer of working around interoperability issues without waiting for the browser to fix them.

This may increase interoperability.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/07e3a1c0-2c67-79b6-dcaf-86762071a472%40gmail.com.

[Peter Thatcher]

Thank you to both Boris and Bernard for clarifying the positions of their respective organizations.

[Peter Thatcher]

I see what you're saying, and I think the most likely scenario where such a thing would happen is with more advanced codec modes, in particular SVC and temporal scalability.  Assuming that eventually all browser support WebCodecs someday, it's possible the support for such modes may differ between them.   That's the same as with WebRTC, but being a lower-level API, there may be more controls in WebCodecs which would allow for more exposure to things that aren't available everywhere.

We could mitigate this by not providing advanced controls if we can't guarantee they are always available, but it seems like it would be better to simply have a good capabilities mechanism so that web apps can adapt to what's available (much like MediaCapabilities).