Intent to Implement: CompressStream

[向井かのん]

Contact emails canon...@google.com,ri...@chromium.org Explainer https://github.com/ricea/compressstream-explainer/blob/master/README.md Design docs/spec Not yet TAG review https://github.com/w3ctag/design-reviews/issues/410 Summary CompressStream is a JavaScript API for gzip compression using Streams. Motivation It is possible to compress stream data without this feature, but common libraries like zlib are complex to use. CompressStream makes it easy for developers to do this, and avoids the need to bundle a compressor with their application. Risks

Interoperability and Compatibility The main risk is that it fails to become an interoperable part of the web platform if other browsers do not implement it. Browsers may produce different compressed output for the same input. Firefox: No public signals Edge: No public signals Safari: No public signals Web developers: No signals Ergonomics The API is very minimal in this first version, which should make it easy to use. Developers familiar with the Streams API will be able to use it without learning anything new. The composability of Streams makes interoperability with other APIs such as TextEncoderStream easy. Activation CompressStream can be polyfilled using a JavaScript or wasm implementation of gzip. This should make it easy to adopt. Security CompressStream is agnostic to the data that passes through it, but users of the API may have to use caution if the data is to be sent or stored in such a way that an attacker might be able to observe its length. If most of the compressed data is known to an attacker, or parts of it can be controlled by an attacker, then they may be able to infer information about the unknown contents from its length. This is similar to the CRIME attack. This is not a problem with CompressStream itself, but fundamental to the way compression works, and it cannot be solved within the API.

Debuggability Normal DevTools JavaScript debugging will work. No special support is needed. Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, Chrome OS, Android, and Android WebView)? Yes Is this feature fully tested by web-platform-tests? No The feature will be tested in web platform tests. Link to entry on the Chrome Platform Status https://chromestatus.com/feature/5855937971617792

This intent message was generated by Chrome Platform Status.

[PhistucK]

I am interested in the web developer interest here. Is this something that is highly (or even moderately) needed?

I do not pretend to know a lot of projects (especially size-sensitive projects), but I do not remember this being an issue, or a missing feature, or even a need (at web development), ever.

If compressed uploads are the main problem this is trying to solve, the browser can just upload compressed data without exposing a stream API for it (maybe using a flag on fetch). I believe requests support compression just as much as responses, at the HTTP protocol level.

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAMp5r%2BzN2xkbJ%2B-_-0-L%2BNN17DUU6gZJkwwkMkQdsnSG2weNgg%40mail.gmail.com.

[Ilya Grigorik]

Really excited to see this! 

The need for this has come up many times in context of various WebPerf WG discussions. Use cases we heard and that immediately come to mind: analytics vendors asking for ability to compress on the client prior to sending a beacon, and optimizing use of local storage in SPAs, etc. Would love to have you join one of the upcoming group calls to discuss the design and use cases — might be a good TPAC topic, in fact.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BWkEvu9k7LmjEaFUjnWF4cGX38rnDwZ4_n5dmnsEzWCg%40mail.gmail.com.

[EricLaw-MSFT]

I'm excited to see this as well, and I'm glad to see that there's a corresponding DecompressStream. One immediate thought is that this would be a useful primitive for building JavaScript apps that manipulate files in the popular ZIP format (based on DEFLATE). I'd be very excited if brotli encoders and decoders were available, given the size/performance of polyfilling.

With regard to the Beacon use case, is the async nature of the API compatible with the new/upcoming restrictions on what sorts of code can run in beforeunload handlers?

[PhistucK]

Optimizing the local storage should be a browser feature - it has more content and more area to analyze and make the compression more effective.

Otherwise, applications would be forced to use less fields for carrying the information in order to make the compression effective, which could create awkward implementations, or greater data loss when data is being evicted by the browser.

Or am I missing something?

☆PhistucK

[Adam Rice]

With regard to the Beacon use case, is the async nature of the API compatible with the new/upcoming restrictions on what sorts of code can run in beforeunload handlers?

My understanding is that Promises in beforeunload handlers are not considered "async", and so if we specify it carefully, we should be able to make sure this works.

On Fri, 30 Aug 2019 at 10:27, 'EricLaw-MSFT' via blink-dev <blin...@chromium.org> wrote:

I'm excited to see this as well, and I'm glad to see that there's a corresponding DecompressStream. One immediate thought is that this would be a useful primitive for building JavaScript apps that manipulate files in the popular ZIP format (based on DEFLATE). I'd be very excited if brotli encoders and decoders were available, given the size/performance of polyfilling.

With regard to the Beacon use case, is the async nature of the API compatible with the new/upcoming restrictions on what sorts of code can run in beforeunload handlers?

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/de342790-a637-4208-b038-0886f4e69b6e%40chromium.org.

[Adam Rice]

>  I believe requests support compression just as much as responses, at the HTTP protocol level.

Unfortunately this is not the case. HTTP is asymmetrical in this respect: a client can send "accept-encoding" in the request to tell a server what encodings it can use in the response, but in the other direction there is a bootstrapping problem. There was a long effort to attempt to add compressed upload to the HTTP protocol, and many approaches were tried, but it was not successful because middleboxes kept misinterpreting the payload.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_%2BWkEvu9k7LmjEaFUjnWF4cGX38rnDwZ4_n5dmnsEzWCg%40mail.gmail.com.

[PhistucK]

That actually means that it is supported at the protocol level. Pre-advertising the feature is not supported.

(Middleboxes are broken in many other aspects anyway and are not part of the protocol)

What if individual fields are compressed rather than the entire payload (less efficient sometimes, but still may have some benefit)? Would this also incompatible with middleboxes? Or simply very inefficient?

☆PhistucK

[Adam Rice]

That actually means that it is supported at the protocol level. Pre-advertising the feature is not supported.
(Middleboxes are broken in many other aspects anyway and are not part of the protocol)

Yes. However, broken middleboxes is the environment that browsers have to function in.

What if individual fields are compressed rather than the entire payload (less efficient sometimes, but still may have some benefit)? Would this also incompatible with middleboxes? Or simply very inefficient?

This would work, but short fields would take more bytes rather than less. It's less flexible than CompressStream, as CompressStream can be used to implement it, but the converse is not true.

[nicj...@gmail.com]

CompressStream is something we would find useful for Boomerang, a Real User Monitoring (RUM) library: https://github.com/akamai/boomerang

In Boomerang we gather and beacon a lot of performance data from the browser, including NavigationTiming, ResourceTiming, UserTiming, etc.  Some of those interfaces can produce a lot of data (e.g. 10KB+), if we just serialize it straight to JSON and put it on the beacon.

For Boomerang we've jumped through a lot of hoops implementing custom compression schemes[1][2] for each set of data, which takes a lot of support code in the library[3], and is often the most expensive operation we execute on the page[4].  We also have to write code on the server to decode and convert it back into something usable.

Taking one example, on a popular media website, serializing all of the ResourceTiming entries to JSON is about 54 KB.

When we apply our compression techniques in Boomerang to it, it reduces in size to around 5.7 KB.

If we were to just simply gzip the original JSON, the size would be around 6.3 KB, only a smidge larger than our custom compression techniques.

If it were available in most browsers, we would probably skip all of that custom stuff and just gzip the JSON'd data.  That would save us complexity, library size, CPU time, maintainability, etc.

[1] https://nicj.net/compressing-resourcetiming/

[2] https://nicj.net/compressing-usertiming/

[3] https://github.com/akamai/boomerang/blob/master/plugins/restiming.js#L251

[4] https://nicj.net/an-audit-of-boomerangs-performance/#work-at-onload

[PhistucK]

Another solution for the compressed uploads could be a built in (but different, to avoid the current incompatibility with middleboxes) way to send compressed content in fetch, using a new HTTP header, like Request-Transfer-Encoding or something (since the compressed body is not the problematic part, otherwise the compression would not work anyway).

ZIP manipulation sounds like a niche use case that could be helped by Web Assembly and JavaScript engine optimizations. 

Can you list significant use cases for this feature other than uploading and local storage? I am struggling to find a real use case that should not be built in.

(I know the extensible web manifesto says browsers should have less magic and this is a nice goal, but I am not sure this is the right building block)

☆PhistucK

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/81531c85-b64d-450a-99fb-bd13ff71d717%40chromium.org.

[Adam Rice]

Can you list significant use cases for this feature other than uploading and local storage? I am struggling to find a real use case that should not be built in.

• WebRTC, WebTransport and other non-HTTP network APIs. 
• Lazy decompression of resources (rather than paying the cost up-front with HTTP).
• In-memory databases.
• Loading and saving gzipped files with the file system API.

But I think the most important use cases are the ones we haven't thought of yet.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CABc02_JRdAe0L8sFArXp3UGMTNnzoeYvEQCcg%3DSp9vAdtiEt_g%40mail.gmail.com.

[PhistucK]

Thank you!

It still feels like something that the browser should do on its own (by default in most cases) because it has a much broader perspective of the content and that can be compressed better.

But I realize that adding compression options (or even without options in some cases, just do it), even if this is far more efficient, to each and every API can feel like "the browser is doing much too much to please the developer" in a way.

Going this route feels like a micro-optimization.

☆PhistucK