Streaming support

[Marcus Cavanaugh]

In Gaia Email, we're working to add support for downloading large attachments without storing everything in memory (for all protocols, including POP3 and ActiveSync). Large attachments are a concern for us due to memory constraints on lower-end devices in particular. Right now, we either limit download size or risk running out of memory and crashing.

Goal:

We'd like to transition to a stream-based API to mitigate these memory issues and prepare us for the currently-in-standardization WHATWG Streams spec.

Options:

• Currently, the WHATWG Stream work remains in flux, and there aren't any polyfills that attempt to mimic the WHATWG's emerging API design. Ideally, we'd like to transition to this spec when it becomes stable and/or supported in Firefox.
• I'd like to avoid writing our own library, due to the complexity involved in building one that supports the conveniences of piping streams and the like; and for similar reasons, we'd like to find an abstraction that we can reuse throughout our protocol implementations.
• There are some existing stream libraries for JavaScript, such as Highland.js, which appears to be the most widely-used library focusing on streams; rx.js and bacon.js could also handle the task, but they are more targeted toward reactive programming than the streams abstraction specifically.

Implementation Notes:

We have our own POP3 library and ActiveSync implementation, which we can modify at will, and we already support sending streaming Blobs over SMTP. IMAP-wise, we may end up just chunking IMAP downloads with byte-range fetching rather than actually streaming directly from Browserbox for now, since we don't have a good way to control backpressure. If we do end up needing to modify Whiteout libs, we'd like to do in a way that makes sense given your direction/vision and ours.

tldr:

We're currently leaning toward using a library like Highland.js to provide a stream abstraction for downloading large attachments. Any thoughts on this direction?

[Marcus Cavanaugh]

Joshua Cranmer just pointed me to this, which may be more promising/desirable than Highland:

https://github.com/h13i32maru/eo.whatwg-streams

[Andris Reinman]

Hi,

We do have streaming support in the roadmap but it is not anything urgent and there’s also the workaround of using chunking, so right now it is buried over by issues with higher priorities. 

I have kept away from ES6 streams for now as the work on the specification has been too active but it seems that it has kind of slowed down, most new commits are about adding or fixing tests for the reference implementation, so it might be worth checking out eo.whatwg-streams but I have no idea if or when we would be actually using it. If you really need this feature then it would be probably faster if you’d update BrowserBox yourself and make a pull request than waiting for the update.

Best regards,

Andris

--
You received this message because you are subscribed to the Google Groups "Email.js Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emailjs+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Whiteout Networks GmbH c/o Werk1

Grafinger Str. 6

D-81671 München

Geschäftsführer: Oliver Gajek

RG München HRB 204479