Intent to Implement: WebRTC Insertable Streams

381 views
Skip to first unread message

Harald Alvestrand

unread,
Oct 1, 2019, 9:13:03 AM10/1/19
to blink-dev


Contact emails

h...@chromium.org, gui...@chromium.org


Explainer


https://github.com/alvestrand/webrtc-media-streams/blob/master/explainer.md



Design doc/Spec


In the process of being written - the explainer gives the basic design.


Summary


This API enables the insertion of hooks for user-defined processing steps in the encoding and decoding of a WebRTC MediaStreamTrack, supplementing or replacing the encoding process.


This is a very early Intent to Implement, the API specification is still in flux. We will ask for TAG review of the feature when a little more documentation work has been done.


Motivation


A number of applications for real time media on the Web (see, for instance, the webrtc-nv-use-cases document) require that functions not available in the browser are executed on the media stream, either before or after encoding for transmission or decoding on reception. Some of these cases are possible to do inefficiently today using the painting of media stream tracks to canvas and recapturing the track from the canvas; other applications are not possible.


This API aims to provide a minimally-disruptive means of making these use cases possible while maintaining compatibility with the existing RTCPeerConnection-based programming model.


Risks


This might not be possible to do in a performant manner. The basic requirement is that one should be able to reliably invoke Javascript-defined functionality on every video frame (cadence below 50 ms per frame), not only fast enough, but without glitches. The experiment will give some basis for evaluating whether this is possible or not.


Interoperability and Compatibility


Most browser engines use at least parts of the WebRTC project core libraries. Once it’s been proven that it is possible to do this, it should be more likely that others will follow suit.


Edge: No signals (but positive on WebCodecs)

Firefox: No signals (but positive on WebCodecs)

Safari: No signals

Web / Framework developers: No signals (but use cases contributed to nv-use-cases)


The proposal was presented at TPAC 2019.


Ergonomics


This API depends on the WebRTC RTCPeerConnection API, and on the WebCodec API.


Activation


It is possible to polyfill (to some degree) pre-encoding / post-decoding processing steps.

Post-encoding/pre-decoding processing steps can’t be polyfilled.

Platforms that do not support the feature will not be broken by attempts to invoke the new API; the request to insert processing steps will be ignored.


Debuggability


No special concerns identified; this may change based on learnings in development.


Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?


Yes.


Link to entry on the feature dashboard


https://www.chromestatus.com/feature/6321945865879552


Requesting approval to ship?


No. This feature needs to be proved before asking to ship.


Joe Medley

unread,
Oct 4, 2019, 1:43:41 PM10/4/19
to Harald Alvestrand, blink-dev
Do you have a tracking bug for this?
Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.


--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHU6OUgbuGo%2BWD4oNHTDVXn74KDjf9ADXO73PcfBOeu-Q%40mail.gmail.com.

Harald Alvestrand

unread,
Oct 4, 2019, 2:01:24 PM10/4/19
to Joe Medley, blink-dev
Nope. I'll create one.

Emily Stark

unread,
Oct 4, 2019, 6:23:14 PM10/4/19
to blink-dev, jme...@google.com
This sounds like it could have security/privacy considerations, but I don't understand it well enough to tell. Could you please add some security and privacy considerations to the explainer? (And make sure security and privacy sections exist in the spec when you have one -- and please make sure this feature goes through security review before it launches to beta.) Thanks!
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

Harald Alvestrand

unread,
Oct 5, 2019, 9:51:35 AM10/5/19
to Emily Stark, blink-dev, Joe Medley
There isn't any information revealed that isn't available already (I think). There are privacy considerations with media streams that are also valid for media streams processed in this fashion, but not so many new ones.
But yes, writing text that says that is important.

(One exception: the webrtc spec has a concept of "isolated media streams" - not implemented by Chrome yet. Isolated streams won't work with this interface, because there's no way to do this processing without breaking the isolation.)


To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bbc85674-9e69-499a-92c9-651c651840b1%40chromium.org.

Joe Medley

unread,
Oct 9, 2019, 1:59:13 PM10/9/19
to Harald Alvestrand, blink-dev
Please share the link when you do.

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195
If an API's not documented it doesn't exist.

Reply all
Reply to author
Forward
0 new messages