Intent to Ship: TransformStream (part of the Streams Standard)

[Adam Rice]

Contact emails

ri...@chromium.org (spec and implementation), dom...@chromium.org (spec)

Explainer

Examples and motivation are included in the standard text: https://streams.spec.whatwg.org/#ts and https://streams.spec.whatwg.org/#example-ts-lipfuzz.

Design doc/Spec

Specification: https://streams.spec.whatwg.org/#ts

Tag review: https://github.com/w3ctag/design-reviews/issues/211

Summary

A TransformStream takes chunks from an input stream and performs arbitrary transformations on them before supplying them to an output stream. The pipeThrough API which shipped in M59 provided a way to pipe a stream through a {writable, readable} pair; TransformStream provides a helper to make creating such a pair much easier in the common case where it represents a transformation.

Motivation

TransformStream makes it easy to transform streaming data. Without it authoring a transformation involves a considerable amount of boilerplate. There is also a risk of developers creating transformations with non-standard APIs and losing the benefit of composition.

Risks

Interoperability and Compatibility

TransformStream is a new constructor in the global namespace. Its public API has been stable in the reference implementation since 2015, with only minor changes.

Edge: No signals

Firefox: Public support

Safari: No signals

Web developers: Positive

Edge and Safari have shipped parts of the Streams API, and Firefox has an implementation under development. Firefox has been active in development of the standard. Chrome will be the first browser to ship TransformStream. TransformStream is scheduled to be used in the Encoding and Fetch standards.

The reference implementation (a line-by-line transcription of the spec into JavaScript) has a full set of web platform tests (https://github.com/whatwg/streams/tree/master/reference-implementation/to-upstream-wpts/transform-streams) with 100% coverage of the reference implementation. These will be moved to the web-platform-tests repository as part of this implementation work.

Ergonomics

This is commonly used with response body streaming in the fetch API, as shown by some of the linked examples.

Stream APIs including TransformStream naturally function in an asynchronous chunked fashion which fits well into highly responsive web apps.

Activation

It will be immediately useful. Polyfills have existed for a while, and can be used to bridge the gap until all browsers reach feature parity.

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)?

Yes.

Link to entry on the feature dashboard

https://www.chromestatus.com/feature/5466425791610880

Requesting approval to ship?

Yes

[Boris Zbarsky]

On 3/15/18 8:46 AM, Adam Rice wrote:
> Firefox: Public support
> <https://github.com/mozilla/standards-positions/issues/70>

Uh... no. That issue is a _request_ for a Firefox position. There is
no Firefox position stated there.

The correct label for Firefox here is "No signals" as things stand.

In general, the right way to use the mozilla/standards-positions repo if
you're going to use it to make a claim about Mozilla's position is to
look at https://mozilla.github.io/standards-positions/

-Boris

[Rick Byers]

Adam,

Can you provide any more details on the positive web developer signals and "It will be immediately useful" bit?  I.e. are there existing deployed sites using the polyfill that will immediately start using our version when we ship?  Are there developers that have tried our implementation (behind a flag?) and have indicated that they would use it in production when we ship? 

I'm just trying to weigh the benefits of shipping now vs. the cost of waiting for some signals from other potential implementers.

Since this has landed in a WHATWG spec, that means there must be support from the design from someone outside of chromium, right?  Was it annevk@ who reviewed the design, or others?

Thanks,

   Rick

Thanks for the correction Boris.

 

-Boris

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/8ddff709-cbea-88f8-5346-5ee0a22a1361%40mit.edu.

[Adam Rice]

On 16 March 2018 at 01:41, Rick Byers <rby...@chromium.org> wrote:

Adam,

Can you provide any more details on the positive web developer signals and "It will be immediately useful" bit?  I.e. are there existing deployed sites using the polyfill that will immediately start using our version when we ship?  Are there developers that have tried our implementation (behind a flag?) and have indicated that they would use it in production when we ship? 

There are teams at Google and Facebook who are currently rolling their own transform streams and have expressed informal interest in using TransformStream once it has shipped.

As far I know, none of the widely-deployed polyfills have automatic fallback, so it won't be immediately used in that way.

 

I'm just trying to weigh the benefits of shipping now vs. the cost of waiting for some signals from other potential implementers.

I don't want developers to have to spend any more time rolling their own transforms. The longer we don't provide the well-lit path, the more developers will wander off into dark alleyways.

Since this has landed in a WHATWG spec, that means there must be support from the design from someone outside of chromium, right?  Was it annevk@ who reviewed the design, or others?

tschneidereit@ has been active in the repository and has contributed a number of fixes: https://github.com/whatwg/streams/commits?author=tschneidereit. Having said that, he hasn't made a specific statement regarding TransformStream.

Thanks,

Adam

[Rick Byers]

Thanks Adam.  Our job in intent to ship threads for new features is to try to balance "moving the web forward" with the interop risk that the API won't become broadly supported across browsers.

So I guess what I'm not yet understanding here is, once we ship this and a few people start using it, what reason will there be for any other browser to feel motivated to ship it also?  If we don't have any signal that any of them think it's a good idea, and expect developers to keep shipping libraries to do this job, then is there any reason to believe this is a feature that won't remain a Chrome-only API?

[Rick Byers]

I should add that I feel terrible making this argument because I personally really like TransferStreams.  Being able to easily pipe streams together seems like an important building block of any platform's standard library.  I'm just struggling to reconcile that with what I'm supposed to be doing as an API owner in ensuring that we're shipping features which are at least somewhat likely to make it into the web at large :-)

It's possible that this is an example of a class of "convenience features" which we should be thinking about differently from the new primitives we're normally talking about.  But I'm just not sure of the framework for reasoning about the shipping decision for such features.  This is as good of as a place as any to discuss the tradeoffs, so I welcome discussion from the community here :-)

Rick

[Rick Byers]

Yeah that's promising.  If Mozilla calls it "important" then I'd be happy to approve this intent.  Also the argument of being important for other platform features (like encoding) to build upon seems potentially compelling.

On Thu, Mar 22, 2018 at 11:21 AM, Yoav Weiss <yo...@yoav.ws> wrote:

It seems like the voices on Mozilla's issue are converging towards "important", so that's an encouraging sign that if shipped it will not be a Chrome-only API.  

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAFUtAY_2AHV2Mw_6a5%2BX%2BeRFpNtUfck6OChBgi8RcTt_H78kMA%40mail.gmail.com.

[Domenic Denicola]

From: Rick Byers <rby...@chromium.org>

> So I guess what I'm not yet understanding here is, once we ship this and a few people start using it, what reason will there be for any other browser to feel motivated to ship it also?  If we don't have any signal that any of them think it's a good idea, and expect developers to keep shipping libraries to do this job, then is there any reason to believe this is a feature that won't remain a Chrome-only API?

It's worth mentioning that in general with streams there has been a pretty healthy collaboration and shipping cadence among the browsers. We often ship first, which is understandable since the spec editors are all Googlers and so we attempt to put our implementation-money where our spec-mouth is to prove a new section of the spec (like TransformStreams) in implementations. But cross-browser participation in the issue tracker, and general support for the project, has been high.

It's not 100% clear that other browsers have had a chance to review everything about TransformStreams and enthusiastically agree to put them on their next-quarter's shipping plans. But as part of the general enterprise of streams that everyone seems bought into, I expect they'll also come along. It's also notable that for browsers already shipping writable streams (Edge and sorta-Safari), the delta to supporting a TransformStream---a specialized {readable, writable} pair---is small.

+WebKit and Edge folks in case they want to chime in.

[Adam Rice]

I feel there is a compelling interest in shipping this soon to provide a "well-lit path" to developers. Getting backpressure working correctly was one of the trickiest parts of designing TransformStream's algorithm. An ad-hoc transformation which doesn't get backpressure right will work in isolation but will fail to conserve resources in the way that is expected of streams. This problem is unlikely to become apparent until it is deployed at scale.

There is a transformation anti-pattern I have seen crop up in a couple of places where a ReadableStream's reader is wrapped in another ReadableStream. The resulting code is hard to understand and doesn't compose well.

I'd like to head off these problems before they become any more widespread.

I recently had a brainstorming session with domenic@ where we discussed how to make debugging streams easier. One thing that became apparent is that it will be easier to provide an easy-to-use debugging experience for pipes when they use the native TransformStream.

Some of the optimisations we are planning, such as short-circuiting identify transforms and hoisting platform transforms off the main thread, are contingent on using native TransformStream. The long-term goal is to do end-to-end pipe optimisation, which requires the browser to have visibility into how each { writable, readable } pair are linked together, ie. it has to be a native TransformStream.

TransformStream is a key part of the implementation of streaming TextEncoder / TextDecoder, which I hope to start soon.

Edge, Safari and Firefox have all implemented the Streams API to various degrees. It would be odd to implement the other parts and not implement TransformStream, particularly as it is the smallest part of the standard (Chrome's implementation is 449 lines including comments).

[Chris Harrelson]

Hi,

Based on what I heard in the thread so far, it sounds like (a) this is an incremental add-on to an existing feature-set, not a totally new concept and (b) there is significant developer interest in this. As for other browser interest, I see that development happened on the WHATWG streams repository. @Adam: I'm assuming you and others reached out to partners at other browsers to ask for their feedback on this feature? I also see there has been a TAG review pending (no reply yet of significance) since Nov 2 2017.

Assuming the above is accurate, I think this is enough to LGTM1 this intent.

As with any intent, I am happy to remove an LGTM before shipping if someone (e.g. folks from another browse vendor) reply with a significant unmet concern. As it stands this feature will not hit stable for about 10 weeks, which is some more time for any late-breaking concern.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAC_ixdyKC3hd0-QtAPpf0%2BdQzk1iOkV%3DyPDzUMtCDWKPP552Yg%40mail.gmail.com.

[Chris Harrelson]

On Fri, Mar 23, 2018 at 2:39 PM Chris Harrelson <chri...@chromium.org> wrote:

Hi,

Based on what I heard in the thread so far, it sounds like (a) this is an incremental add-on to an existing feature-set, not a totally new concept and (b) there is significant developer interest in this. As for other browser interest, I see that development happened on the WHATWG streams repository. @Adam: I'm assuming you and others reached out to partners at other browsers to ask for their feedback on this feature?

... And now I just noticed Domenic had cc'd folks and mentioned working together. So I think the answer to this is Yes. :)

[Domenic Denicola]

From: Yoav Weiss <yo...@yoav.ws>

> Domenic and Anne - how comfortable are you with the feature's design? I'm trying to estimate the level of confidence we have that the feature won't change significantly as more vendors adopt it, so would love to hear your opinion on the APIs maturity.

For my part, I'm very comfortable, given how we've taken a cautious and incremental approach to revving the feature behind a flag and in the spec. The basic design, of a linked { writable, readable } pair, has been known and described for a long time (years); Adam formally specified it in November, and since then we've been making minor tweaks to the spec as we gain experience. These minor tweaks were all around error cases, developer-facing in-spec documentation, increasing test coverage, and the interface with other standards.

I'd actually say that, of all the changes we've made since November's initial spec draft, none of them would have caused significant compatibility concern. (This is largely a consequence of the fact mentioned upthread, that once you have ReadableStream and WritableStream, TransformStream is a small incremental addition.) But, we wanted to get things into ship-shape before shipping, so here we are :). At this point I'm quite happy with the spec, and feel the design has proven itself pretty well in a series of demos and a large test suite.

[Yoav Weiss]

That's great to hear! LGTM2

[Rick Byers]

Ok, sounds like it'll probably turn out OK to me.  LGTM3

As with all features, if a year goes by with it being Chrome-only we should try to understand if there's anything to learn.  Please ensure there is a UseCounter so we can monitor usage (also helps other vendors with prioritization).

Rick

[Shuo Shen]

Thank you folks for moving forward with TranformStream support.

We are building video streaming at Facebook and would really love to see TransformStream natively supported by Browser.

Thanks,

Shuo