Intent to Extend Origin Trial: WebCodecs

[Chris Cunningham]

Contact emails

chcunn...@google.com, sand...@google.com, eri...@google.com

 

Explainer

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Specification

https://wicg.github.io/web-codecs/ 

 

Design docs

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

 

Blink component

Blink>Media

 

TAG reviews

https://github.com/w3ctag/design-reviews/issues/433
https://github.com/w3ctag/design-reviews/issues/612

 

TAG review status

Initial feedback addressed. Awaiting feedback from latest review request. 

Risks

 

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

  

Gecko: Positive co-editing the spec (Paul Adenot)

Edge: Positive co-editing the spec (Bernard Aboba)

WebKit: No signal (https://docs.google.com/document/d/1AJmgUNzPqbu8yub8e0HxB9G9op16Bfcp6UaxYKv6Xag/edit#heading=h.oml89rehr07t) "We have some concerns about WebCodecs, but don’t think we’ve stated a public position on it. Concerns about fingerprinting and design of the API, but I don’t think it’s fair to say “negative”. " - Myles Maxfield

 

Web developers: Positive (https://discourse.wicg.io/t/webcodecs-proposal/3662/) 

Wikipedia: https://discourse.wicg.io/t/webcodecs-proposal/3662/9 

Twitch: https://discourse.wicg.io/t/webcodecs-proposal/3662/12 

Sony: https://discourse.wicg.io/t/webcodecs-proposal/3662/13

 

Ergonomics

The proposed API shape enables the core use cases (see explainer) in a performant manner. We've left room for future optimizations and generally minimized complexity, even if that means we don't support all codec features. We intend for this shape to be friendly to both WASM and JS users. 

 

Decoder outputs will typically be rendered with Canvas and Web Audio. Developers have asked for this low level rendering control. The Canvas rendering path also allows VideoFrames to be manipulated via WebGL. The WebAudio rendering path will often leverage AudioWorklet. The AudioDecoder output uses Web Audio primitives to make this easy.

 

Encoder inputs will often come from getUserMedia() and getDisplayMedia(). 

 

Activation

 

WebCodecs would benefit from having libraries built on top that do things such as: 

+ containerize (mux) and de-containerize (demux) to/from media file formats (e.g. mp4, webm).

+ render media with low latency from an unreliable media transport (in other words, a jitter buffer)

+ RTC client logic / signaling 

 

WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs. Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability.

 

 

Security

The implementation is thoroughly fuzz tested. There may be marginally increased fingerprinting surface due to support for detecting the presence of hardware encode capabilities. This has been reviewed and deemed acceptable by the privacy sandbox team.

 

 

Goals for experimentation

We need feedback on most of the API surfaces, which roughly amounts to decode for video, audio and images, and encode for video and audio. 

 

Multiple partners have committed (and shown a great deal of eagerness) to participate in our origin trial and we will look for their feedback and validation of the appropriateness of the API shape for their use cases (e.g. real time communications, low latency streaming).

 

 

Experimental timeline

The origin trial is currently expected to conclude with the release of M91 stable. We are now requesting an extension through M91, with the trial concluding with the release of M92 stable.

 

Reason this experiment is being extended

The origin trial has been successful in generating strong interest and valuable feedback. We need more time to (1) resolve a handful of API shape questions, (2) respond to pending TAG feedback, and (3) build x-UA consensus.

 

Ongoing technical constraints

None.

 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

  

Is this feature fully tested by web-platform-tests?

Yes. We have coverage for all basic uses of the API, including exception and error states, which should ensure a base-level uniformity across browser implementation of the API surface. Some deeper aspects of the API's implementation, such as which codecs are supported, vary from browser to browser and hence may not need to be tested by WPTs. Test status can be found at: https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&q=webcodecs

 

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=897297

 

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1097106

 

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5669293909868544

 

Links to previous Intent discussions

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ/m/oyb3oFVaAAAJ

First Experiment Extension: https://groups.google.com/a/chromium.org/g/blink-dev/c/3MNyU3OAacI/m/sj8IdIMRAwAJ 

[Mike West]

LGTM to extend this OT again. There are open questions around the API's shape, and you're getting active feedback from partners and other vendors. That falls pretty squarely into #2 under https://www.chromium.org/blink/origin-trials/running-an-origin-trial#TOC-What-is-the-process-to-extend-an-origin-trial- as a reasonable ground upon which to ask for an extension.

I've left a few comments inline below which I hope you can take a look at before (presumably?) coming back with an intent-to-ship.

-mike

On Sat, Mar 13, 2021 at 1:14 AM Chris Cunningham <chcunn...@chromium.org> wrote:

Contact emails

chcunn...@google.com, sand...@google.com, eri...@google.com

 

Explainer

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Specification

https://wicg.github.io/web-codecs/ 

 

Design docs

https://github.com/WICG/web-codecs/blob/master/explainer.md

 

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

 

Blink component

Blink>Media

 

TAG reviews

https://github.com/w3ctag/design-reviews/issues/433
https://github.com/w3ctag/design-reviews/issues/612

 

TAG review status

Initial feedback addressed. Awaiting feedback from latest review request. 

Risks

 

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

  

Gecko: Positive co-editing the spec (Paul Adenot)

Comments in the TAG review suggest that there might be more than a little additional work to do to align with Mozilla on the API's shape: https://github.com/w3ctag/design-reviews/issues/612#issuecomment-782216485. Are those conversations actively happening? A one-milestone extension doesn't give y'all a ton of leeway.

 

Edge: Positive co-editing the spec (Bernard Aboba)

WebKit: No signal (https://docs.google.com/document/d/1AJmgUNzPqbu8yub8e0HxB9G9op16Bfcp6UaxYKv6Xag/edit#heading=h.oml89rehr07t) "We have some concerns about WebCodecs, but don’t think we’ve stated a public position on it. Concerns about fingerprinting and design of the API, but I don’t think it’s fair to say “negative”. " - Myles Maxfield

Please obtain signals via the official channels our friends at other vendors have pointed us towards: https://bit.ly/blink-signals.

Did you consider adding those non-core codecs as "tentative" tests, and/or using assert_implements_optional to verify their existence before verifying their behavior?

 

 

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=897297

 

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1097106

 

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5669293909868544

 

Links to previous Intent discussions

Intent to Experiment: https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ/m/oyb3oFVaAAAJ

First Experiment Extension: https://groups.google.com/a/chromium.org/g/blink-dev/c/3MNyU3OAacI/m/sj8IdIMRAwAJ 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSrX3oJByJ17foGnHbFw%3DKgUbu1zUtt3Cv0j8YPDG9ESag%40mail.gmail.com.

[Chris Cunningham]

Thank you Mike!

> Comments in the TAG review suggest that there might be more than a little additional work to do to align with Mozilla on the API's shape: https://github.com/w3ctag/design-reviews/issues/612#issuecomment-782216485. Are those conversations actively happening? A one-milestone extension doesn't give y'all a ton of leeway.

Paul is just returning from vacation, but we've generally had a standing weekly meeting and lots of engagement on issues and pull requests. We are well aligned on the API shape and we agree that there are some must-fix issues to address before shipping. M92 is not a ton of leeway, but I estimate that there is room to complete the work with about 20 days of buffer. If you prefer, we can go for a longer extension and simply ship before OT ends if things go to plan? 

> Please obtain signals via the official channels our friends at other vendors have pointed us towards: https://bit.ly/blink-signals.

Apologies, I should have given more context. We do have a thread on webkit-dev where they express interest with some questions / reservations. We've not heard from them recently though (though we have asked here and in WG calls). 

> Did you consider adding those non-core codecs as "tentative" tests, and/or using assert_implements_optional to verify their existence before verifying their behavior?

Presently the codec-specific tests are in wpt-internal. I was not aware of assert_implements_optional, but I think its a great fit. We'll pursue migrating tests accordingly. 

Best,

Chris