Intent to Experiment: MediaStreamTrack Insertable Streams (a.k.a. Breakout Box)

[Guido Urdaneta]

Contact emails

gui...@chromium.org, tgui...@chromium.org, h...@chromium.org

Explainer

https://github.com/w3c/mediacapture-insertable-streams/blob/main/explainer.md

Specification

https://w3c.github.io/mediacapture-insertable-streams/

API spec

Yes

Summary

This feature defines an API surface for manipulating raw media carried by MediaStreamTracks such as the output of a camera, microphone, screen capture, or the decoder part of a codec and the input to the decoder part of a codec. It uses WebCodecs interfaces to represent raw media frames and exposes them using streams, similarly to the way the WebRTC Insertable Streams spec exposes encoded data from RTCPeerConnections.

Blink component

Blink>MediaStream

TAG review

https://github.com/w3ctag/design-reviews/issues/603

TAG review status

Pending

Risks

Interoperability and Compatibility

As with all new features, the main interoperability risk is that other browsers do not implement it. There is no compatibility risk since the existing behavior is unaffected if the feature is not used.

Gecko: No signal
Edge: No signal
WebKit: No signal
Web developers: Positive

Ergonomics

* Are there any other platform APIs this feature will frequently be used in tandem with? This API will be used together with other MediaStream and WebRTC related APIs, such as getUserMedia, getDisplayMedia, and RTCPeerConnection. * Could the default usage of this API make it hard for Chrome to maintain good performance (i.e. synchronous return, must run on a certain thread, guaranteed return timing)? No.

Activation

* Will it be challenging for developers to take advantage of this feature immediately, as-is? No. * Would this feature benefit from having polyfills, significant documentation and outreach, and/or libraries built on top of it to make it easier to use? No.

Security

This feature introduces more ergonomic ways to expose data that is already exposed in Chrome via other APIs (e.g., MediaStreams in combination with media elements, canvas and Web Audio). In this sense, it does not introduce new attack surfaces other than the API implementation itself plus the proposed explicit signaling streams. An application may try to DoS a MediaStream source by issuing many control signals, but since these signals are just hints they are not a reliable way to launch a DoS attack. This feature relies on existing security mechanisms for MediaStreamTracks. For example, all camera or screen-capture tracks require user authorization, and element-capture tracks do not allow access to cross-origin data. All the code for this feature runs in the sandboxed renderer process and therefore respects the Rule of Two.

Goals for experimentation

We are looking for feedback with regards to the API shape and its performance, especially for the video processing use case. We also want to learn if the WebCodecs interfaces for raw data need adjustment to better support the use cases enabled by this API. A number of partners have expressed interest in participating in the Origin Trial.

Since this API can be seen as an extension to WebCodecs, we would like to reuse the WebCodecs Origin Trial token in order to make this API easier to use for existing participants in the ongoing WebCodecs trial.

Ongoing technical constraints

None.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

No, but WPTs will be added before an intent to ship.

Tracking bug

https://crbug.com/1142955

Launch bug

https://crbug.com/1146805

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5499415634640896

Links to previous Intent discussions

Intent to prototype: https://groups.google.com/a/chromium.org/forum/#!topic/blink-dev/gkBkLvGKuzc

This intent message was generated by Chrome Platform Status.

[Mike West]

The TAG review opened yesterday. I think it's reasonable to give them a chance to weigh in.

I'm also curious about signals from other vendors. Have you asked?

-mike

[Mike West]

Ah! This is an intent to experiment, not ship. Nevermind! What milestones do you want the experiment to run through?

Regarding reusing the existing WebCodecs experiment: is it the case that this mechanism is only useful in conjunction with Web Codecs? It's a little strange to tie two unrelated features together (philosophically, and practically with regard to the documentation). Does this implicitly require an extension to the Web Codecs experiment (which ends in M90)? If you ship Web Codecs, will you keep the experiment around for this mechanism? It might be cleaner to simply separate them in the case that one might be ready to ship sooner than the other.

[Dale Curtis]

This API reuses the VideoFrame and AudioFrame structures that WebCodecs introduces. It's effectively a better designed replacement for the {Audio|Video}Track{Reader|Writer} primitives that WebCodecs was planning to introduce. If we ship WebCodecs, we'll likely ship this alongside.

 - dale

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/98240f5b-aac5-4dea-b554-e4cca1f87b7an%40chromium.org.

[Guido Urdaneta]

Hi,

We would like the experiment to run from 89 to 91. Since 89 already branched, 90 to 92 is also acceptable if a merge to enable the trial is not accepted.

With regards to reusing the WebCodecs trial, one of the reasons is that some partners using VideoTrackReader/Writer in the WebCodecs trial will need to migrate to this API. It's not a hard requirement to be part of the same trial, but since this API replaces a functionality that was originally part of the WebCodecs spec and trial, there is a case for it.

Thanks,

GU

[Guido Urdaneta]

Friendly ping to API owners.

[Mike West]

We talked about this in our meeting tonight; it's a little weird to tie this OT to the WebCodecs OT, but if it's suitable for your use cases and your partners, there's no reason we wouldn't do it.

LGTM1 to experiment. If you get approval to merge things back to 89, then 89-91 sounds reasonable, also as 1 milestone extension to the Web Codecs OT.

That said, I'm a bit more concerned with extending the Web Codecs OT out 2 milestones. It's not the end of the world, but is there a reason to do so rather than making a decision to ship or not? Is there additional information you expect to receive from partners that would influence the API's design?

-mike

[Dale Curtis]

WebCodecs OT ends in 91, we don't anticipate any further extensions -- hopefully shipping will happen in 91/92.

- dale

--

You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAKXHy%3Dd%2BkrwgJ5D-AjQn5Qp6h9D5%3DgqUOk8eHO5Cwh2D0KQprg%40mail.gmail.com.

[Theodore Olsauskas-Warren]

Hi All,

In the combined OWP Security and Privacy weekly meeting, the reviewers (+aarontag, +arthursonzogni) agreed that the S&P section of the spec could be improved by further detailing exactly what existing mechanisms can be used, and how, to extract the information exposed directly by this new API.

If the spec wants to lean on the idea that the data is exposable through other APIs, then that link should be made strong enough that if circumstances around those other APIs change in the future, we can more readily confirm the privacy position of this spec is still valid.

Theo.