Intent to Ship: WebRTC Insertable Streams

556 views
Skip to first unread message

Guido Urdaneta

unread,
Aug 11, 2020, 10:58:26 AM8/11/20
to blink-dev

Contact emails

gui...@chromium.org, h...@chromium.org  


Explainer

https://github.com/w3c/webrtc-insertable-streams/blob/master/explainer.md


Spec

https://htmlpreview.github.io/?https://github.com/w3c/webrtc-insertable-streams/blob/master/index.html


Tag review: https://github.com/w3ctag/design-reviews/issues/531

Requested on 2020-06-29, but the TAG has not completed the review yet. 


Summary

This API allows RTCPeerConnections to expose the data flowing through it as streams (based on the WHATWG Streams API). This allows applications to insert custom processing of data via transform streams. 


Link to “Intent to Prototype” blink-dev discussion

https://groups.google.com/a/chromium.org/g/blink-dev/c/5UZuZNGvgwo


Link to Origin Trial feedback summary

Feedback from the trial has been positive, and several products have made public announcements of new features that rely on this API. More details at

https://docs.google.com/document/d/1QD_R8N7MXb_IvxEmGBqbpUunlBmJ6WAeEo8p0aTXpGM/edit?usp=sharing


Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.


Demo link

https://webrtc.github.io/samples/src/content/peerconnection/endtoend-encryption/


Risks

Interoperability and Compatibility

Since this is a new feature, the main risk is that other browsers decide not to implement it.


Signals from other implementations:

Gecko: Positive with comments. Official position requested at https://github.com/mozilla/standards-positions/issues/330, but no reply yet. During workgroup discussions they stated that they are OK with adopting this API, but they are opposed to recommending it for end-to-end encryption until a module for key management is part of the Web platform. Their view is that key management should not be done in JavaScript.

WebKit: Positive with comments. Like Gecko, their position (stated in workgroup discussions) is not to recommend this API for end-to-end encryption until a full solution is provided by the Web Platform. They see this API as an intermediate step in that direction and, in their view, shipping progressively is probably OK.


Web / Framework developers: Reception has been positive. Some articles published by various developers:

  1. http://webrtcbydralex.com/index.php/2020/03/30/secure-frames-sframes-end-to-end-media-encryption-with-webrtc-now-in-chrome/

  2. https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

  3. https://medium.com/@medooze/adding-metadata-to-media-frames-via-insertable-streams-api-for-webrtc-47f7a740e457

  4. https://www.meetecho.com/blog/janus-e2ee-sframe/

  5. https://connectycube.com/2020/04/23/the-current-state-of-e2ee-in-webrtc/

  6. https://www.lifesize.com/en/video-conferencing-blog/e2ee

  7. https://support.8x8.com/cloud-phone-service/meetings/8x8-video-meetings-virtual-office/are-my-8x8-video-meetings-encrypted

  8. https://thenextweb.com/apps/2020/05/21/a-look-at-how-jitsi-became-a-secure-open-source-alternative-to-zoom/

  9. https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-f9a83a997d6d

  10. https://twitter.com/BraveSampson/status/1270112148463554562


Most of the articles focus on the end-to-end encryption use case, but other use cases are also covered.

The Mozilla position request also contains some positive comments by developers.


Ergonomics

Are there any other platform APIs this feature will frequently be used in tandem with? 

This feature will be used frequently with other WebRTC-related APIs such as getUserMedia() and the rest of RTCPeerConnection.


Could the default usage of this API make it hard for Chrome to maintain good performance (i.e. synchronous return, must run on a certain thread, guaranteed return timing)?

No.


Activation

Will it be challenging for developers to take advantage of this feature immediately, as-is?

No. Several products based on this API have already been announced.


Would this feature benefit from having polyfills, significant documentation and outreach, and/or libraries built on top of it to make it easier to use?

No.


Is this feature fully tested by web-platform-tests

WPT suite (covers some tests).

Most of the tests are currently Blink Web Tests. Migration to WPT tracked at https://crbug.com/1114029. No issues expected in the migration.


Entry on the feature dashboard

https://www.chromestatus.com/feature/6321945865879552


Domenic Denicola

unread,
Aug 11, 2020, 11:43:05 AM8/11/20
to Guido Urdaneta, blink-dev
Very exciting to see this go out; I know developers have really enjoyed working with insertable streams!

From: Guido Urdaneta <gui...@chromium.org>

> https://htmlpreview.github.io/?https://github.com/w3c/webrtc-insertable-streams/blob/master/index.html

It would be nice if this were published somewhere as more than an Unofficial Draft (e.g., an Editor's Draft), and had the chance for some review to make sure it was specified in sufficient detail. E.g., has the Web RTC working group reviewed it? I notice at least an Apple engineer has reviewed it, and filed some issues; it might be worthwhile to address those: https://github.com/w3c/webrtc-insertable-streams/issues/created_by/youennf

I filed https://github.com/w3c/webrtc-insertable-streams/issues/40 in the meantime, which is a suggestion for a breaking API change, which I hope you can consider before shipping.

Guido Urdaneta

unread,
Aug 12, 2020, 7:03:14 AM8/12/20
to Domenic Denicola, Guido Urdaneta, blink-dev
Hi,

With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

Wrt to issues filed by Apple:
  • Privacy and Security reviews: they were performed internally at Google. See https://crbug.com/1059912 (restricted to Googlers)
  • Off-main-thread processing by default: Transferable Streams allow a very straightforward mechanism for processing off the main thread using Workers. The feedback from the origin trial in this regard has been very positive. There are some improvements that can be done (and are planned) in the Transferable Streams implementation in Chromium, but overall, based on the feedback received during the Origin Trial, I think this is sufficiently addressed by Transferable Streams and Workers. 
  • Ability to insert native source nodes in the pipeline: This refers to integrating browser-provided modules with Insertable Streams for tasks like key management for encryption. There is an interesting discussion in the issue. I see no impediment in the proposed Insertable Streams API to integrate with future mechanisms provided by the browser for key management, whether similar to the standard compression streams  or a module with a messaging interface running on a Worker, as skteched in https://github.com/w3c/webrtc-insertable-streams/issues/17#issuecomment-624615829.


Domenic Denicola

unread,
Aug 12, 2020, 10:17:39 AM8/12/20
to Guido Urdaneta, blink-dev
From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?

> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.

Guido Urdaneta

unread,
Aug 12, 2020, 1:20:59 PM8/12/20
to Domenic Denicola, Guido Urdaneta, blink-dev
On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:
From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?



 
> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.


I haven't closed the security and privacy review issues because we are still waiting for feedback from the TAG review, which includes a privacy/security questionnaire.
The internal Google security and privacy reviews should be enough to ship in Blink, though.

I closed the off-main thread issue because the model with transferable streams and workers has been proven to work well by the Origin Trial.
I haven't closed the issue about external modules, since there is a discussion going on that could result in ideas for a future module that would help complete a complete e2ee solution in the Web Platform (Apple's and Mozilla's main concern). Such a module would probably be defined in a separate spec, but maybe also as part of a future revision of Insertable Streams.
There is no impediment in the current spec to integrate with any such future modules, or present for that matter. For example, some origin trial participants have experimented with integration with WebCrypto. 

Yoav Weiss

unread,
Aug 13, 2020, 11:35:35 AM8/13/20
to Guido Urdaneta, Domenic Denicola, blink-dev
Yay to fake mustaches!!

On Wed, Aug 12, 2020 at 7:20 PM Guido Urdaneta <gui...@chromium.org> wrote:


On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:
From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?


I believe you need to change the status to "ED", so that the spec will no longer be marked as "Unofficial Draft".

Regarding the signal from WebKit, it's unclear which comment you were referring to in the WG discussion.
We typically require an official signal. Could you ask for one?



 
> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.


I haven't closed the security and privacy review issues because we are still waiting for feedback from the TAG review, which includes a privacy/security questionnaire.
The internal Google security and privacy reviews should be enough to ship in Blink, though.

I closed the off-main thread issue because the model with transferable streams and workers has been proven to work well by the Origin Trial.
I haven't closed the issue about external modules, since there is a discussion going on that could result in ideas for a future module that would help complete a complete e2ee solution in the Web Platform (Apple's and Mozilla's main concern). Such a module would probably be defined in a separate spec, but maybe also as part of a future revision of Insertable Streams.
There is no impediment in the current spec to integrate with any such future modules, or present for that matter. For example, some origin trial participants have experimented with integration with WebCrypto. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxZ-z5ijUY%2B-LiZ8xQ6fBDpsPg8d78Gkwq3MgpZxuTfS%3DA%40mail.gmail.com.

Guido Urdaneta

unread,
Aug 14, 2020, 1:30:37 PM8/14/20
to Yoav Weiss, Guido Urdaneta, Domenic Denicola, blink-dev
On Thu, Aug 13, 2020 at 5:35 PM Yoav Weiss <yo...@yoav.ws> wrote:
Yay to fake mustaches!!

On Wed, Aug 12, 2020 at 7:20 PM Guido Urdaneta <gui...@chromium.org> wrote:


On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:
From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?


I believe you need to change the status to "ED", so that the spec will no longer be marked as "Unofficial Draft".

Done.
 
Regarding the signal from WebKit, it's unclear which comment you were referring to in the WG discussion.

I was referring to this comment:
Youenn: I view this as a short-term solution. Long-term we should have E2E provided by the browser. But I think shipping progressively is probably OK.
This is the video: https://youtu.be/YGYlzlKFpXA?t=1758

We typically require an official signal. Could you ask for one?
Will update this thread when there's an official reply.

Yoav Weiss

unread,
Aug 18, 2020, 8:38:39 AM8/18/20
to Guido Urdaneta, Domenic Denicola, blink-dev
LGTM1

Guido Urdaneta

unread,
Aug 19, 2020, 5:02:56 PM8/19/20
to Yoav Weiss, blink-dev
Friendly ping, API Owners.

Chris Harrelson

unread,
Aug 19, 2020, 7:41:26 PM8/19/20
to Guido Urdaneta, Yoav Weiss, blink-dev
LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

Guido Urdaneta

unread,
Aug 20, 2020, 8:29:28 AM8/20/20
to Chris Harrelson, Guido Urdaneta, Yoav Weiss, blink-dev
The TAG completed the review. Their feedback is positive.
"Hi,
The TAG reviewed this specification this week and is happy with the way it is progressing.
Thank you for asking the TAG for review."
 

Philip Jägenstedt

unread,
Aug 20, 2020, 8:57:37 AM8/20/20
to Guido Urdaneta, Chris Harrelson, Yoav Weiss, blink-dev
Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

https://bugs.chromium.org/p/chromium/issues/detail?id=1114029 is also something that would be good to resolve or commit to before shipping, as it's very likely to end up backlogged otherwise. https://wpt.fyi/results/webrtc-insertable-streams has only two tests, and judging by the naming they're not testing that it works, but only that things fail in the right way. Upstreaming our own tests and adding minimal idlharness.js tests would be great. (https://github.com/web-platform-tests/wpt/pull/24135 is a start)

Guido Urdaneta

unread,
Aug 20, 2020, 10:33:52 AM8/20/20
to Philip Jägenstedt, Guido Urdaneta, Chris Harrelson, Yoav Weiss, blink-dev
On Thu, Aug 20, 2020 at 2:57 PM Philip Jägenstedt <foo...@chromium.org> wrote:
Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

Among the issues Domenic raised, I would say that the one with the most practical impact in terms of API is the readable/writable vs readableStream/writableStream naming. In this case we decided to go for the readable/writable naming proposed by Domenic.
Another open issue with potential API impact is an issue to add statistics accessible via getStats(). That one is still being discussed, but it does not affect the API covered by this I2S. It would affect the stats spec, for which there is a procedure to add new stats.
There are other issues that relate more to the longer term objective of adding a full E2EE solution to the Web Platform. While that will continue to be discussed, it is not part of this I2S.
Finally, there are some metadata discussions. Much of what is discussed also concerns a full E2EE solution, or future additions to be considered in a new revision of the spec (e.g., control channels for internal WebRTC components, etc.).
As far as I can tell, the main issue remaining with direct impact on the API was the naming of the stream pair, which has been resolved.
 

https://bugs.chromium.org/p/chromium/issues/detail?id=1114029 is also something that would be good to resolve or commit to before shipping, as it's very likely to end up backlogged otherwise. https://wpt.fyi/results/webrtc-insertable-streams has only two tests, and judging by the naming they're not testing that it works, but only that things fail in the right way. Upstreaming our own tests and adding minimal idlharness.js tests would be great. (https://github.com/web-platform-tests/wpt/pull/24135 is a start)

Philipp Hancke

unread,
Aug 20, 2020, 11:30:08 AM8/20/20
to Philip Jägenstedt, Guido Urdaneta, Chris Harrelson, Yoav Weiss, blink-dev
Am Do., 20. Aug. 2020 um 14:57 Uhr schrieb Philip Jägenstedt <foo...@chromium.org>:
Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

lets try to eliminate one of them :-)
Should we remove the Exposed=Window? I didn't see a point in exposing it on the worker but don't see one for exposing it on window either.
 
I agree with Guido, metadata and stats can be done later. I'm still quite happy with how this turned out!

Rick Byers

unread,
Aug 20, 2020, 2:54:57 PM8/20/20
to Philipp Hancke, Philip Jägenstedt, Guido Urdaneta, Chris Harrelson, Yoav Weiss, blink-dev

Philip Jägenstedt

unread,
Aug 21, 2020, 12:21:51 PM8/21/20
to Rick Byers, Chris Harrelson, Guido Urdaneta, Philipp Hancke, Yoav Weiss, blink-dev
LGTM4, thanks Guido for going through the open issues and upstreaming more tests!
Reply all
Reply to author
Forward
0 new messages