Intent to Extend Origin Trial: WebCodecs

[Husain Bengali]

Contact emails

chcunn...@google.com, sand...@google.com, hben...@google.com

 

Explainer

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Specification

https://wicg.github.io/web-codecs/ 

 

Design docs

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

 

Blink component

Blink>Media

 

TAG review

https://github.com/w3ctag/design-reviews/issues/433

 

TAG review status

Issues addressed

 

Risks

 

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

  

Gecko: Positive co-editing the spec (Paul Adenot)

 

WebKit: No signal (https://docs.google.com/document/d/1AJmgUNzPqbu8yub8e0HxB9G9op16Bfcp6UaxYKv6Xag/edit#heading=h.oml89rehr07t) "We have some concerns about WebCodecs, but don’t think we’ve stated a public position on it. Concerns about fingerprinting and design of the API, but I don’t think it’s fair to say “negative”. " - Myles Maxfield

 

Web developers: Positive (https://discourse.wicg.io/t/webcodecs-proposal/3662/) Wikipedia: https://discourse.wicg.io/t/webcodecs-proposal/3662/9 

Twitch: https://discourse.wicg.io/t/webcodecs-proposal/3662/12 

Sony: https://discourse.wicg.io/t/webcodecs-proposal/3662/13

 

Ergonomics

The proposed API shape enables the core use cases (see explainer) in a performant manner. We've left room for future optimizations and generally minimized complexity, even if that means we don't support all codec features. We intend for this shape to be friendly to both WASM and JS users. 

 

Decoder outputs will typically be rendered with Canvas and Web Audio. Developers have asked for this low level rendering control. The Canvas rendering path also allows VideoFrames to be manipulated via WebGL. The WebAudio rendering path will often leverage AudioWorklet. The AudioDecoder output uses Web Audio primitives to make this easy.

 

Encoder inputs will often come from getUserMedia() and getDisplayMedia(). 

 

Activation

 

WebCodecs would benefit from having libraries built on top that do things such as: 

+ containerize (mux) and de-containerize (demux) to/from media file formats (e.g. mp4, webm).

+ render media with low latency from an unreliable media transport (in other words, a jitter buffer)

+ RTC client logic / signaling 

 

WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs. Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability.

 

 

Security

The implementation is thoroughly fuzz tested. There may be marginally increased fingerprinting surface due to support for detecting the presence of hardware encode capabilities. This has been reviewed and deemed acceptable by the privacy sandbox team.

 

 

Goals for experimentation

We need feedback on most of the API surfaces, which roughly amounts to decode for video, audio and images, and encode for video and audio. 

 

Multiple partners have committed (and shown a great deal of eagerness) to participate in our origin trial and we will look for their feedback and validation of the appropriateness of the API shape for their use cases (e.g. real time communications, low latency streaming).

 

 

Experimental timeline

The initial goal of the origin trial was to run for 3 milestones, from M86 Stable thru till M88 Stable. We are now looking to extend the origin trial thru till M91 Stable

 

Reason this experiment is being extended

The origin trial has been successful in generating strong interest and valuable feedback. We need additional time to fill API gaps (e.g. worker support, pixel format conversion, audio encoding support) and collect feedback on those areas. This additional time will also allow us to land performance improvements.

 

Ongoing technical constraints

None.

 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

  

Is this feature fully tested by web-platform-tests?

Yes. We have coverage for all basic uses of the API, including exception and error states, which should ensure a base-level uniformity across browser implementation of the API surface. Some deeper aspects of the API's implementation, such as which codecs are supported, vary from browser to browser and hence may not need to be tested by WPTs. Test status can be found at: https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&q=webcodecs

 

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=897297

 

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1097106

 

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5669293909868544

 

Links to previous Intent discussions

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ/m/oyb3oFVaAAAJ

 

This intent message was generated by Chrome Platform Status.

[Yoav Weiss]

LGTM to extend experimentation till M91, but not beyond.

In case that won't be sufficient, please reach out to discuss further.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACEBcBNVA48x%3DGzj3HJfovcYH59AXXrCUxx-GvMjh7Y%2BqfzHUA%40mail.gmail.com.

[Thomas Guilbert]

Hello,

As suggested by the OT core group, continuing a discussion here:

We've made design changes to WebCodecs, we decided that portions of the Breakout Box project should replace a small portion of WebCodecs, to deduplicate functionalities and to allow better performance in workers long term.

Specifically, before WebCodec ships, we will deprecate VideoTrackReader/VideoTrackWriter in favor of using TrackProcessor/TrackGenerator. 

The Breakout Box launch bug has passed through Security and Privacy reviews, and is pending legal review and final approvals.

We're looking to bring those primitives under the WebCodecs OT flag, since our launches are now coupled, and so partners can experiment ahead of time. Ideally, we'd like to merge the flag changes back to M89 (pending launch approvals), to get feedback from the Stable channel during our M91 bug fix push.

Can you advise as to what needs to happen next?

Thanks!
Thomas

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEgXS_8jB0VPU8ZVLTkpsk5vgim5GBz-NZsEEFi_FAHXeQ%40mail.gmail.com.

[Yoav Weiss]

Apologies for missing this...

Merging back those changes makes sense, but as it exposes a new API, an "Intent to Experiment" for that new API seems warranted. Once approved, you can expose that new API under the old OT.

[Guido Urdaneta]

I just sent an Intent to Experiment to blink-dev. 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CACj%3DBEiAj%2BZr5r3oOdL3f3m1TtNMzNPSA1R-APvT1PBNDWUUMA%40mail.gmail.com.