Intent to Implement: WebRTC Insertable Streams
Harald Alvestrand
Contact emails
h...@chromium.org, gui...@chromium.org
Explainer
https://github.com/alvestrand/webrtc-media-streams/blob/master/explainer.md
Design doc/Spec
In the process of being written - the explainer gives the basic design.
Summary
This API enables the insertion of hooks for user-defined processing steps in the encoding and decoding of a WebRTC MediaStreamTrack, supplementing or replacing the encoding process.
This is a very early Intent to Implement, the API specification is still in flux. We will ask for TAG review of the feature when a little more documentation work has been done.
Motivation
A number of applications for real time media on the Web (see, for instance, the webrtc-nv-use-cases document) require that functions not available in the browser are executed on the media stream, either before or after encoding for transmission or decoding on reception. Some of these cases are possible to do inefficiently today using the painting of media stream tracks to canvas and recapturing the track from the canvas; other applications are not possible.
This API aims to provide a minimally-disruptive means of making these use cases possible while maintaining compatibility with the existing RTCPeerConnection-based programming model.
Risks
This might not be possible to do in a performant manner. The basic requirement is that one should be able to reliably invoke Javascript-defined functionality on every video frame (cadence below 50 ms per frame), not only fast enough, but without glitches. The experiment will give some basis for evaluating whether this is possible or not.
Interoperability and Compatibility
Most browser engines use at least parts of the WebRTC project core libraries. Once it’s been proven that it is possible to do this, it should be more likely that others will follow suit.
Edge: No signals (but positive on WebCodecs)
Firefox: No signals (but positive on WebCodecs)
Safari: No signals
Web / Framework developers: No signals (but use cases contributed to nv-use-cases)
The proposal was presented at TPAC 2019.
Ergonomics
This API depends on the WebRTC RTCPeerConnection API, and on the WebCodec API.
Activation
It is possible to polyfill (to some degree) pre-encoding / post-decoding processing steps.
Post-encoding/pre-decoding processing steps can’t be polyfilled.
Platforms that do not support the feature will not be broken by attempts to invoke the new API; the request to insert processing steps will be ignored.
Debuggability
No special concerns identified; this may change based on learnings in development.
Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?
Yes.
Link to entry on the feature dashboard
https://www.chromestatus.com/feature/6321945865879552
Requesting approval to ship?
No. This feature needs to be proved before asking to ship.
Joe Medley
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHU6OUgbuGo%2BWD4oNHTDVXn74KDjf9ADXO73PcfBOeu-Q%40mail.gmail.com.
Harald Alvestrand
Emily Stark
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
Harald Alvestrand
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOqqYVHU6OUgbuGo%2BWD4oNHTDVXn74KDjf9ADXO73PcfBOeu-Q%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bbc85674-9e69-499a-92c9-651c651840b1%40chromium.org.