Working with streams/requests/responses

[Domenic Denicola]

I'm starting to do Node for realz these days, and am trying to work with HTTP and promises. It's painful, so I feel either I must be doing HTTP and/or streams wrong, or this stuff is inherently painful and someone has probably wrapped them up in a nicer promise-based library.

Here is the kind of code I find myself writing: https://gist.github.com/2125448

Does anyone have any advice on how I could make this more straightforward? Perhaps the removeListener stuff isn't really necessary? Or better yet, has someone made a promise library for this sort of thing?

[Kris Kowal]

If you’re dropping the response object, you certainly don’t need to do
any explicit handle cleanup. That shortens the labor a bit.

I favor creating small, reusable wrappers and getting into promise
code as quickly as possible.

https://gist.github.com/2126149

Kris Kowal

[Domenic Denicola]

Heads up to any future passers-by: this approach unfortunately causes problems with anything other than a HEAD request, because every time you wait for a promise to resolve, you skip to the next tick of the event loop. But by then, you may have lost some of the response's data events.

Thus, using `promiseResponse` from Kris's gist, you cannot do:

promiseResponse(request).then(function (response) {
   response.pipe(anotherStream);
}).end();

because pipe() will miss the data events between ticks.

[Christoph Dorn]

Domenic Denicola wrote:

Heads up to any future passers-by: this approach unfortunately causes problems with anything other than a HEAD request, because every time you wait for a promise to resolve, you skip to the next tick of the event loop. But by then, you may have lost some of the response's data events.

Thus, using `promiseResponse` from Kris's gist, you cannot do:

promiseResponse(request).then(function (response) {
   response.pipe(anotherStream);
}).end();

because pipe() will miss the data events between ticks.

I am wondering if we can efficiently use a workaround such as this:

  http://www.senchalabs.org/connect/utils.html#exports.pause

Pause data and end events on the given obj. Middleware performing async tasks should utilize this utility (or similar), to re-emit data once the async operation has completed, otherwise these events may be lost.

var pause = utils.pause(req); fs.readFile(path, function(){ next(); pause.resume(); });


Christoph

[Domenic Denicola]

Went down that route, somewhat.

Sadly until Node 0.9, stream.pause is advisory for HTTP [1], so you will lose data trying to do this. (Perhaps Connect's pause is different than Node's built-in, though.) Plus, from what I can gather pausing involves sending some kind of TCP message to the client, as does resuming, which is expensive.

Another workaround is a BufferedStream [2], which will hold things in memory for at least one tick. It feels like bad practice, though: I think I'm supposed to be streaming directly from server to server (e.g. Amazon S3 to my web server's client), not storing things in memory.

[1]: https://groups.google.com/forum/#!topic/nodejs/yv6Dl-O-wYk/discussion
[2]: https://github.com/mjijackson/bufferedstream/

---

From: q-con...@googlegroups.com [q-con...@googlegroups.com] on behalf of Christoph Dorn [christ...@christophdorn.com]
Sent: Wednesday, April 04, 2012 15:20
To: q-con...@googlegroups.com
Subject: Re: [Q] Working with streams/requests/responses

[Kris Kowal]

On Tue, Apr 3, 2012 at 3:13 PM, Domenic Denicola
<dom...@domenicdenicola.com> wrote:
> Heads up to any future passers-by: this approach unfortunately causes
> problems with anything other than a HEAD request, because every time you
> wait for a promise to resolve, you skip to the next tick of the event loop.
> But by then, you may have lost some of the response's data events.
>
> Thus, using `promiseResponse` from Kris's gist, you cannot do:
>
> promiseResponse(request).then(function (response) {
>    response.pipe(anotherStream);
> }).end();
>
> because pipe() will miss the data events between ticks.

Ah, yeah. Node does not buffer input, just output. Such a pain.

Kris Kowal

[Kris Kowal]

It should have occurred to me to mention this earlier.

It is definitely necessary to attache event listeners on a stream in
the first available turn.

I do happen to have a library for this. I use it in Q-FS and Q-HTTP.
It is predictably called Q-IO and it is not documented anywhere.
(Sorry!)

https://github.com/kriskowal/q-io/blob/master/q-io.js

In any case, this is how you would use it in this example:

https://gist.github.com/2345090

I’m assuming the existence of a client.get method here, since you are
particularly looking for something that provides the stream as the
eventual fulfillment.

Kris Kowal

[Domenic Denicola]

This and similar solutions are pretty good; we're using Pipe from capsela [1]. The only problem is that they don't wrap a native HTTP request or response object; instead, you create a new Pipe or new q-io Reader/Writer that assimilates your existing request/response. In the process, you lose all of the extra capabilities, e.g. getting/setting headers.

I think this could be solved by moving toward a more wholesale promise-based request/response model, which from what I understand capsela provides. But that's a big jump. It might be nicer to have a simple PromisedHttp(Client|Server)(Request|Response) class that is initialized by giving it a corresponding Http(Client|Server)(Request|Response). Hmm, maybe a fun project.

[1]: https://github.com/capsela/capsela-util/blob/master/lib/Pipe.js

[Domenic Denicola]

We did end up using connect's `pause` for one instance. It turns out, looking at the source code, to be very similar to the capsela `Pipe` or q-io `Reader`/`Writer`, i.e. it buffers events in memory and re-emits them when you tell them it's OK to resume. Here's a quick rundown of the different approaches:

Most of the time we use capsela `Pipe`, when we are letting streams flow through the system (e.g. returning them as a promise's fulfillment value). E.g.:

```
var responsePromise = getPromiseForPausedResponseAsCapselaPipe();
return responsePromise.then(function (responsePromise) {
   responsePromise.pipe(fileStream); // add lots of error and end listeners + a deferred
   responsePromise.resume();
});
```

I believe q-io style would be something like

```
var responsePromise = getPromiseForQIOReader();
return responsePromise.read().then(function (responseString) {
   return fileStreamAsQIOWriter.write(responseString);
});
```

which loses the piping, but that's not a hard feature to add to Q-IO. Pull request time, I think.

---

We use connect's pause when streams are not flowing through the system, but instead directly recieved from a non-promisey part of the system, like middleware:

function middleware(request, response, next) {
    var pauser = connect.utils.pause(request);
    headerChecker.areHeadersOKAccordingToTheDatabase(request).then(function (ok) {
        if (ok) {
            next();
        } else {
            next(new Error("Bad headers"));
        }
    }).fin(pauser.resume.bind(pauser)).end();
}

---

BTW the BufferedStream I suggested upthread was buggy. It gets in to some kind of recursive nextTick calling of itself that pegs your CPU at 100%. Do not use! Use Capsela Pipe or Q-IO instead.

On Wednesday, April 4, 2012 3:25:00 PM UTC-4, Domenic Denicola wrote:

Went down that route, somewhat.

Sadly until Node 0.9, stream.pause is advisory for HTTP [1], so you will lose data trying to do this. (Perhaps Connect's pause is different than Node's built-in, though.) Plus, from what I can gather pausing involves sending some kind of TCP message to the client, as does resuming, which is expensive.

Another workaround is a BufferedStream [2], which will hold things in memory for at least one tick. It feels like bad practice, though: I think I'm supposed to be streaming directly from server to server (e.g. Amazon S3 to my web server's client), not storing things in memory.

[1]: https://groups.google.com/forum/#!topic/nodejs/yv6Dl-O-wYk/discussion
[2]: https://github.com/mjijackson/bufferedstream/

---

From: q-con...@googlegroups.com [q-con...@googlegroups.com] on behalf of Christoph Dorn [christoph-gg@christophdorn.com]

Sent: Wednesday, April 04, 2012 15:20
To: q-con...@googlegroups.com
Subject: Re: [Q] Working with streams/requests/responses

[Christoph Dorn]

Domenic Denicola 20 April, 2012 3:44 PM

It might be nicer to have a simple PromisedHttp(Client|Server)(Request|Response) class that is initialized by giving it a corresponding Http(Client|Server)(Request|Response). Hmm, maybe a fun project.

I have a hacked together interface between NodeJS connect and JSGI as a connect middleware that supports promises in some ways:

  https://github.com/pinf/server-js/blob/master/lib/vendor/connect/middleware/jsgi.js

Used here (no promises in example):

  https://github.com/pinf/server-js/tree/master/examples/demo/lib

I am just about ready to port the "pinf/server-js" to my latest codebase built on top of the sourcemint loader [1] and the sourcemint NodeJS platform [2].

This "PromisedHttp" stuff is pretty low-level for me and I was hoping to "find" :) something built by someone that I can take and build a bunch of examples for. This will shortly lead to having a toolset to easily work on and deploy Q promised-based systems that span server, browser, mobile and embedded. (The last remaining hurdle seems to be debugging Q promise-based code. I have a parse step so can manipulate the AST before loading into runtime but I need the hooks to inject (or equivalent) and the logic to re-assemble the call stack).

Christoph


[1] - https://github.com/sourcemint/loader-js
[2] - https://github.com/sourcemint/platform-nodejs