Intent to Extend Origin Trial: WebCodecs

[Chris Cunningham]

Contact emails

chcunn...@chromium.org, sand...@chromium.org, eri...@google.com, dalec...@chromium.org

 

Explainer

https://github.com/w3c/webcodecs/blob/main/explainer.md 

 

Specification

https://w3c.github.io/webcodecs/

 

Design docs

https://docs.google.com/document/d/1-AwmoXL0Cg1ZFGLHchjS48_dxgezHqmXW3YRYtcF53s/edit?usp=sharing

 

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

 

Blink component

Blink>Media

 

TAG reviews

https://github.com/w3ctag/design-reviews/issues/612
https://github.com/w3ctag/design-reviews/issues/433

 

TAG review status

Feedback addressed. Awaiting final approval. 

Risks

 

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

  

Gecko: Positive co-editing the spec (Paul Adenot)

Edge: Positive co-editing the spec (Bernard Aboba)

WebKit: No signal

Expressed mixed interest/concerns in request for position
https://lists.webkit.org/pipermail/webkit-dev/2020-May/031191.html

https://lists.webkit.org/pipermail/webkit-dev/2021-February/031691.html 

More recently seeing increased WebKit engagement on github for WebCodecs and adjacent media APIs. 

 

Web developers: Positive. https://discourse.wicg.io/t/webcodecs-proposal/3662/ plus lots of enthusiastic positive feedback from OT participants.

 

Ergonomics

The proposed API shape enables the core use cases (see explainer) in a performant manner. We've left room for future optimizations and generally minimized complexity, even if that means we don't support all codec features. We intend for this shape to be friendly to both WASM and JS users. 

 

Decoder outputs will typically be rendered with Canvas and Web Audio. Developers have asked for this low level rendering control. The Canvas rendering path also allows VideoFrames to be manipulated via WebGL. The WebAudio rendering path will often leverage AudioWorklet.

 

Encoder inputs will often come from getUserMedia() and getDisplayMedia(). 

 

Activation

 

WebCodecs would benefit from having libraries built on top that do things such as: 

+ containerize (mux) and de-containerize (demux) to/from media file formats (e.g. mp4, webm).

+ render media with low latency from an unreliable media transport (in other words, a jitter buffer)

+ RTC client logic / signaling 

 

WebCodecs could benefit somewhat from polyfills to experiment with the API, but that would not bring any of the performance benefits or access to hardware codecs. Significant documentation and outreach would likely be helpful, especially for advanced uses of codecs, such as spatial and temporal scalability.

 

 

Security

The implementation is thoroughly fuzz tested. There may be marginally increased fingerprinting surface due to support for detecting the presence of hardware encoding capabilities. This has been reviewed and deemed acceptable by the privacy sandbox team.

 

 

Goals for experimentation

We need feedback on most of the API surfaces, which roughly amounts to decode for video, audio and images, and encode for video and audio. 

 

Multiple partners have committed (and shown a great deal of eagerness) to participate in our origin trial and we will look for their feedback and validation of the appropriateness of the API shape for their use cases (e.g. real time communications, low latency streaming).

 

Reason this experiment is being extended

We did not reach cross UA consensus on blocking GitHub spec issues to ship in 92. Some areas of disagreement were discovered too late. We've since revamped our issue tracking / labels and are firmly aligned between spec co-editors on the outstanding work (labels: breaking + need-definition) and the proposed timeline to ship in  93. Having said that, we are requesting OT extension through 93, ending with release of 94 to allow for additional room. 

 

Experimental timeline

The origin trial is currently expected to conclude with the release of M92 stable. We are now requesting that it run for an additional 2 milestones (M92 and M93), concluding with the release of M94 stable. 

 

Ongoing technical constraints

None.

 

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

  

Is this feature fully tested by web-platform-tests?

Yes. Test status can be found at: https://wpt.fyi/results/webcodecs?label=master&label=experimental&aligned&q=webcodecs 

 

Tracking bug

https://bugs.chromium.org/p/chromium/issues/detail?id=897297

 

Launch bug

https://bugs.chromium.org/p/chromium/issues/detail?id=1097106

 

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5669293909868544

 

Links to previous Intent discussions

Intent to Experiment: 

https://groups.google.com/a/chromium.org/g/blink-dev/c/7OdxQf5HnlQ

Earlier Extension threads: 

https://groups.google.com/a/chromium.org/g/blink-dev/c/3MNyU3OAacI 

https://groups.google.com/a/chromium.org/g/blink-dev/c/55QViQdz5Rc 

[Yoav Weiss]

Running the OT till M94 would run the OT for 9 milestones, which is rather long.

Can you outline which breaking changes were done during the OT that would reduce the risk of burn-in?

On Wed, Jun 2, 2021 at 1:23 AM Chris Cunningham <chcunn...@chromium.org> wrote:

Contact emails

chcunn...@chromium.org, sand...@chromium.org, eri...@google.com, dalec...@chromium.org

 

Explainer

https://github.com/w3c/webcodecs/blob/main/explainer.md 

 

Specification

https://w3c.github.io/webcodecs/

 

Design docs

https://docs.google.com/document/d/1-AwmoXL0Cg1ZFGLHchjS48_dxgezHqmXW3YRYtcF53s/edit?usp=sharing

 

Summary

Provides efficient, low-level access to built-in (software and hardware) media encoders and decoders.

 

Blink component

Blink>Media

 

TAG reviews

https://github.com/w3ctag/design-reviews/issues/612
https://github.com/w3ctag/design-reviews/issues/433

 

TAG review status

Feedback addressed. Awaiting final approval. 

Risks

 

Interoperability and Compatibility

The main risk is that other browsers do not implement it.

  

Gecko: Positive co-editing the spec (Paul Adenot)

A few months back, Paul didn't seem excited about us shipping as is. Is the current shape of the OT something we have more consensus on?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSrk47%3DG9ZSs3kn9Sys_2P7F8tmjReVU%2B2JTfDeyzv2ewg%40mail.gmail.com.

[Chris Cunningham]

Hi Yoav,

> Can you outline which breaking changes were done during the OT that would reduce the risk of burn-in?

Sure thing. Here's an outline (but not an exhaustive list). 

All of these are major breaks to core interfaces. 

• completely deprecated the VideoTrackReader API in favor to MediaStreamTrackProcessor https://github.com/w3c/webcodecs/issues/131
• redesign of how we read pixels from VideoFrame https://github.com/w3c/webcodecs/issues/157
• full redesign of how raw audio data is exposed https://github.com/w3c/webcodecs/issues/168 https://github.com/w3c/webcodecs/issues/223
• making "chunk" interfaces immutable https://github.com/w3c/webcodecs/issues/80

And then there are many smaller (but still significant) breaks like

• require frame.close() after encode(frame) https://github.com/w3c/webcodecs/pull/163
• require 'key' chunks after flush() and configure() https://github.com/w3c/webcodecs/pull/247
• renaming config dictionary members https://github.com/w3c/webcodecs/pull/186 
• deprecating/replacing config dictionary members https://github.com/w3c/webcodecs/pull/250

> A few months back, Paul didn't seem excited about us shipping as is. Is the current shape of the OT something we have more consensus on?

Yes, firmly. We've aligned on both outstanding work and timeline. 

[guest271314]

How to decode and play encoded 'opus' outside of Chromium/Chrome, or any other implementation? See https://github.com/xiph/opus/issues/221.

[Yoav Weiss]

LGTM to experiment M92-M94

The list of breaking changes gives me confidence that the risk of burn-in is small, as participating origins had to modify their implementation multiple times throughout the trial.

As such, it feels safe to extend the OT, even beyond the typical time limits of OTs.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[โต่ง ทองจันดี]

ในวันที่ พฤ. 3 มิ.ย. 2021 22:41 น. Yoav Weiss <yoav...@chromium.org> เขียนว่า:

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSrk47%3DG9ZSs3kn9Sys_2P7F8tmjReVU%2B2JTfDeyzv2ewg%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/62c30e7d-7d8f-4463-a475-b5fbeb991befn%40chromium.org.

[Dan McArdle]

Hello, WebCodecs team!

I think I've found a discrepancy between the explainer and spec.  The explainer lists "clocking" as a goal, and defines it as "the ability to drive the decoding or encoding using a specific clock domain, to be able to control clock drifts."  I was curious about potential privacy implications of clocking, but I found no mention of it in the spec.  I figure one of these documents needs to be updated :)

If clocking does belong in the spec, could you add some text to explain the meaning of clock domain?

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CALG6eSrk47%3DG9ZSs3kn9Sys_2P7F8tmjReVU%2B2JTfDeyzv2ewg%40mail.gmail.com.

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/62c30e7d-7d8f-4463-a475-b5fbeb991befn%40chromium.org.

[Dale Curtis]

That just means a client can control the rate of presentation beyond e.g., what HTMLMediaElement.playbackRate allows today. I agree it can be written less confusingly or just removed entirely: https://github.com/w3c/webcodecs/issues/273

- dale