Intent to Ship: WebRTC Insertable Streams

[Guido Urdaneta]

Contact emails

gui...@chromium.org, h...@chromium.org  

Explainer

https://github.com/w3c/webrtc-insertable-streams/blob/master/explainer.md

Spec

https://htmlpreview.github.io/?https://github.com/w3c/webrtc-insertable-streams/blob/master/index.html

Tag review: https://github.com/w3ctag/design-reviews/issues/531. 

Requested on 2020-06-29, but the TAG has not completed the review yet. 

Summary

This API allows RTCPeerConnections to expose the data flowing through it as streams (based on the WHATWG Streams API). This allows applications to insert custom processing of data via transform streams. 

Link to “Intent to Prototype” blink-dev discussion

https://groups.google.com/a/chromium.org/g/blink-dev/c/5UZuZNGvgwo

Link to Origin Trial feedback summary

Feedback from the trial has been positive, and several products have made public announcements of new features that rely on this API. More details at

https://docs.google.com/document/d/1QD_R8N7MXb_IvxEmGBqbpUunlBmJ6WAeEo8p0aTXpGM/edit?usp=sharing

Is this feature supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Demo link

https://webrtc.github.io/samples/src/content/peerconnection/endtoend-encryption/

Risks

Interoperability and Compatibility

Since this is a new feature, the main risk is that other browsers decide not to implement it.

Signals from other implementations:

Gecko: Positive with comments. Official position requested at https://github.com/mozilla/standards-positions/issues/330, but no reply yet. During workgroup discussions they stated that they are OK with adopting this API, but they are opposed to recommending it for end-to-end encryption until a module for key management is part of the Web platform. Their view is that key management should not be done in JavaScript.

WebKit: Positive with comments. Like Gecko, their position (stated in workgroup discussions) is not to recommend this API for end-to-end encryption until a full solution is provided by the Web Platform. They see this API as an intermediate step in that direction and, in their view, shipping progressively is probably OK.

Web / Framework developers: Reception has been positive. Some articles published by various developers:

1. http://webrtcbydralex.com/index.php/2020/03/30/secure-frames-sframes-end-to-end-media-encryption-with-webrtc-now-in-chrome/

2. https://webrtchacks.com/true-end-to-end-encryption-with-webrtc-insertable-streams/

3. https://medium.com/@medooze/adding-metadata-to-media-frames-via-insertable-streams-api-for-webrtc-47f7a740e457

4. https://www.meetecho.com/blog/janus-e2ee-sframe/

5. https://connectycube.com/2020/04/23/the-current-state-of-e2ee-in-webrtc/

6. https://www.lifesize.com/en/video-conferencing-blog/e2ee

7. https://support.8x8.com/cloud-phone-service/meetings/8x8-video-meetings-virtual-office/are-my-8x8-video-meetings-encrypted

8. https://thenextweb.com/apps/2020/05/21/a-look-at-how-jitsi-became-a-secure-open-source-alternative-to-zoom/

9. https://medium.com/@medooze/sframe-js-end-to-end-encryption-for-webrtc-f9a83a997d6d

10. https://twitter.com/BraveSampson/status/1270112148463554562

Most of the articles focus on the end-to-end encryption use case, but other use cases are also covered.

The Mozilla position request also contains some positive comments by developers.

Ergonomics

Are there any other platform APIs this feature will frequently be used in tandem with? 

This feature will be used frequently with other WebRTC-related APIs such as getUserMedia() and the rest of RTCPeerConnection.

Could the default usage of this API make it hard for Chrome to maintain good performance (i.e. synchronous return, must run on a certain thread, guaranteed return timing)?

No.

Activation

Will it be challenging for developers to take advantage of this feature immediately, as-is?

No. Several products based on this API have already been announced.

Would this feature benefit from having polyfills, significant documentation and outreach, and/or libraries built on top of it to make it easier to use?

No.

Is this feature fully tested by web-platform-tests? 

WPT suite (covers some tests).

Most of the tests are currently Blink Web Tests. Migration to WPT tracked at https://crbug.com/1114029. No issues expected in the migration.

Entry on the feature dashboard

https://www.chromestatus.com/feature/6321945865879552

[Domenic Denicola]

Very exciting to see this go out; I know developers have really enjoyed working with insertable streams!

From: Guido Urdaneta <gui...@chromium.org>

> https://htmlpreview.github.io/?https://github.com/w3c/webrtc-insertable-streams/blob/master/index.html

It would be nice if this were published somewhere as more than an Unofficial Draft (e.g., an Editor's Draft), and had the chance for some review to make sure it was specified in sufficient detail. E.g., has the Web RTC working group reviewed it? I notice at least an Apple engineer has reviewed it, and filed some issues; it might be worthwhile to address those: https://github.com/w3c/webrtc-insertable-streams/issues/created_by/youennf

I filed https://github.com/w3c/webrtc-insertable-streams/issues/40 in the meantime, which is a suggestion for a breaking API change, which I hope you can consider before shipping.

[Guido Urdaneta]

Hi,

With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

Wrt to issues filed by Apple:

• Privacy and Security reviews: they were performed internally at Google. See https://crbug.com/1059912 (restricted to Googlers)
• Off-main-thread processing by default: Transferable Streams allow a very straightforward mechanism for processing off the main thread using Workers. The feedback from the origin trial in this regard has been very positive. There are some improvements that can be done (and are planned) in the Transferable Streams implementation in Chromium, but overall, based on the feedback received during the Origin Trial, I think this is sufficiently addressed by Transferable Streams and Workers. 
• Ability to insert native source nodes in the pipeline: This refers to integrating browser-provided modules with Insertable Streams for tasks like key management for encryption. There is an interesting discussion in the issue. I see no impediment in the proposed Insertable Streams API to integrate with future mechanisms provided by the browser for key management, whether similar to the standard compression streams  or a module with a messaging interface running on a Worker, as skteched in https://github.com/w3c/webrtc-insertable-streams/issues/17#issuecomment-624615829.

[Domenic Denicola]

From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?

> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.

[Guido Urdaneta]

On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:

From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?

This is the proper link: https://w3c.github.io/webrtc-insertable-streams/  

 

> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.

I haven't closed the security and privacy review issues because we are still waiting for feedback from the TAG review, which includes a privacy/security questionnaire.

The internal Google security and privacy reviews should be enough to ship in Blink, though.

I closed the off-main thread issue because the model with transferable streams and workers has been proven to work well by the Origin Trial.

I haven't closed the issue about external modules, since there is a discussion going on that could result in ideas for a future module that would help complete a complete e2ee solution in the Web Platform (Apple's and Mozilla's main concern). Such a module would probably be defined in a separate spec, but maybe also as part of a future revision of Insertable Streams.

There is no impediment in the current spec to integrate with any such future modules, or present for that matter. For example, some origin trial participants have experimented with integration with WebCrypto. 

[Yoav Weiss]

Yay to fake mustaches!!

On Wed, Aug 12, 2020 at 7:20 PM Guido Urdaneta <gui...@chromium.org> wrote:

On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:

From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?

This is the proper link: https://w3c.github.io/webrtc-insertable-streams/  

I believe you need to change the status to "ED", so that the spec will no longer be marked as "Unofficial Draft".

Regarding the signal from WebKit, it's unclear which comment you were referring to in the WG discussion.

We typically require an official signal. Could you ask for one?

 

> Wrt to issues filed by Apple:

It's great to hear that these are resolved in your view. It would be best if these resolutions were reflected in the specification and/or explainer, and then the issues could be closed so that folks could get a better sense of what the outstanding issues are with the proposal by looking at the issue tracker.

I haven't closed the security and privacy review issues because we are still waiting for feedback from the TAG review, which includes a privacy/security questionnaire.

The internal Google security and privacy reviews should be enough to ship in Blink, though.

I closed the off-main thread issue because the model with transferable streams and workers has been proven to work well by the Origin Trial.

I haven't closed the issue about external modules, since there is a discussion going on that could result in ideas for a future module that would help complete a complete e2ee solution in the Web Platform (Apple's and Mozilla's main concern). Such a module would probably be defined in a separate spec, but maybe also as part of a future revision of Insertable Streams.

There is no impediment in the current spec to integrate with any such future modules, or present for that matter. For example, some origin trial participants have experimented with integration with WebCrypto. 

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxZ-z5ijUY%2B-LiZ8xQ6fBDpsPg8d78Gkwq3MgpZxuTfS%3DA%40mail.gmail.com.

[Guido Urdaneta]

On Thu, Aug 13, 2020 at 5:35 PM Yoav Weiss <yo...@yoav.ws> wrote:

Yay to fake mustaches!!

On Wed, Aug 12, 2020 at 7:20 PM Guido Urdaneta <gui...@chromium.org> wrote:

On Wed, Aug 12, 2020 at 4:17 PM Domenic Denicola <d...@domenic.me> wrote:

From: Guido Urdaneta <gui...@chromium.org>

> With regards to the renaming change, I explained the rationale in the github issue. Still, we can do the rename you propose and include the old names as part of the legacy API to be covered by an extension of the Origin Trial (sent as a separate intent), so that current Origin Trial participants have time to migrate to the new names, but without exposing the old names to new users.

Great to hear; thank you!

> Wrt the question of this being reviewed by the WebRTC WG, yes, the spec has been reviewed by the WG and the document has been adopted as a WG document.

That's good to know. Is there a canonical spec URL then, that isn't hosted by htmlpreview.github.io? Similarly, is there a version of the specification that is published by the WG, instead of marked as "Unofficial Draft"?

This is the proper link: https://w3c.github.io/webrtc-insertable-streams/  

I believe you need to change the status to "ED", so that the spec will no longer be marked as "Unofficial Draft".

Done.

 

Regarding the signal from WebKit, it's unclear which comment you were referring to in the WG discussion.

I was referring to this comment:

Youenn: I view this as a short-term solution. Long-term we should have E2E provided by the browser. But I think shipping progressively is probably OK.

This is the video: https://youtu.be/YGYlzlKFpXA?t=1758

We typically require an official signal. Could you ask for one?

Done: https://lists.webkit.org/pipermail/webkit-dev/2020-August/031342.html

Will update this thread when there's an official reply.

[Yoav Weiss]

LGTM1

[Guido Urdaneta]

Friendly ping, API Owners.

[Chris Harrelson]

LGTM2

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAEhhuH4cJzLdrwZ8vwrxmnNpxJsjHKvA3p17_RHCY7yKSRQsvQ%40mail.gmail.com.

[Guido Urdaneta]

The TAG completed the review. Their feedback is positive.

https://github.com/w3ctag/design-reviews/issues/531#issuecomment-676645161

"Hi,
The TAG reviewed this specification this week and is happy with the way it is progressing.
Thank you for asking the TAG for review."

 

[Philip Jägenstedt]

Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

https://bugs.chromium.org/p/chromium/issues/detail?id=1114029 is also something that would be good to resolve or commit to before shipping, as it's very likely to end up backlogged otherwise. https://wpt.fyi/results/webrtc-insertable-streams has only two tests, and judging by the naming they're not testing that it works, but only that things fail in the right way. Upstreaming our own tests and adding minimal idlharness.js tests would be great. (https://github.com/web-platform-tests/wpt/pull/24135 is a start)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CA%2BBuZxbQ48j8dvSL56tFsw63B14bZ_OBtXSiRwTGzD%2Bch7%2BGpQ%40mail.gmail.com.

[Guido Urdaneta]

On Thu, Aug 20, 2020 at 2:57 PM Philip Jägenstedt <foo...@chromium.org> wrote:

Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

Among the issues Domenic raised, I would say that the one with the most practical impact in terms of API is the readable/writable vs readableStream/writableStream naming. In this case we decided to go for the readable/writable naming proposed by Domenic.

Another open issue with potential API impact is an issue to add statistics accessible via getStats(). That one is still being discussed, but it does not affect the API covered by this I2S. It would affect the stats spec, for which there is a procedure to add new stats.

There are other issues that relate more to the longer term objective of adding a full E2EE solution to the Web Platform. While that will continue to be discussed, it is not part of this I2S.

Finally, there are some metadata discussions. Much of what is discussed also concerns a full E2EE solution, or future additions to be considered in a new revision of the spec (e.g., control channels for internal WebRTC components, etc.).

As far as I can tell, the main issue remaining with direct impact on the API was the naming of the stream pair, which has been resolved.

 

https://bugs.chromium.org/p/chromium/issues/detail?id=1114029 is also something that would be good to resolve or commit to before shipping, as it's very likely to end up backlogged otherwise. https://wpt.fyi/results/webrtc-insertable-streams has only two tests, and judging by the naming they're not testing that it works, but only that things fail in the right way. Upstreaming our own tests and adding minimal idlharness.js tests would be great. (https://github.com/web-platform-tests/wpt/pull/24135 is a start)

We are upstreaming the tests here: https://chromium-review.googlesource.com/c/chromium/src/+/2367078

[Philipp Hancke]

Am Do., 20. Aug. 2020 um 14:57 Uhr schrieb Philip Jägenstedt <foo...@chromium.org>:

Domenic pointed out a few open issues, and I see that there are a bunch in https://github.com/w3c/webrtc-insertable-streams/issues. Have the open issues been triaged for whether they would change the API when addressed? Those issues would be important to make an explicit decision on before shipping.

lets try to eliminate one of them :-)

  https://github.com/w3c/webrtc-insertable-streams/issues/25

Should we remove the Exposed=Window? I didn't see a point in exposing it on the worker but don't see one for exposing it on window either.

 

I agree with Guido, metadata and stats can be done later. I'm still quite happy with how this turned out!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAARdPYcFMONkYbmqT4s7Lk397nZ6vjuiiAM0qi7F8Jj3h0Ku7g%40mail.gmail.com.

[Rick Byers]

LGTM3

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CADxkKiKjn4pggjf4E03%3DcaAd2C5hDy_v%3DhrW5WVVqVotFda1yA%40mail.gmail.com.

[Philip Jägenstedt]

LGTM4, thanks Guido for going through the open issues and upstreaming more tests!