Re: [nodejs] Re: Evented JSGI

[Dean Landolt]

On Tue, Jan 26, 2010 at 12:00 PM, Kris Zyp <kri...@gmail.com> wrote:

On Jan 25, 12:41 am, "Isaac Z. Schlueter" <i...@foohack.com> wrote:
> Hey, what do you guys think of this?  A worthwhile direction to
> pursue?
>
> If a promise is like the async version of a function, then it seems a
> parallel metaphor to think of an event emitter as sorta like the async
> version of an object with specific fields.  To "read" the field, you
> attach a listener.  To have one object mirror another, you can just
> listen to the events you care about, and then re-emit them on the
> watcher object.

I think this looks like a great idea for handling streaming of the
body. Of course, I think it is pretty critical that there is support
for asynchronously returning the status and headers, especially for
use in Node. I didn't see any examples of that in your code (the
status and headers were always returned synchronously). Can I presume
that one would follow JSGI 0.3's use of promises for the response
object to defer the status and headers, and then use your streaming
body pattern for the sending the body? If so, that sounds like it
could be a cool combination (although the synchronous streaming
scenario seems like it could induce undesirable buffering).

This evented JSGI approach is very promising. If you combine this with the possibility to synchronously return the body as a forEachable like current JSGI you can get the best of both worlds (yes, I know this is irrelevant to node but I think there's general agreement among jsgites that the easy case should be easy -- this should always be a valid JSGI app: return {status:200,headers:{},body:["hello world"]).

But it's also important that the complex cases be possible. I'd like to propose we extend the current Stream defined in JSGI with this EventEmitter-style. Of course we'd have to specify streams without a node dependency (though it would be great if Ryan's stream concept was eventually proposed to CommonJS). But for the sake of JSGI, perhaps something as simple as:

A Stream MUST can contain a "write" method which, when called raises the "data" event. A Stream MUST contain a "close" method which when called raises the "eof" event...

For upload throttling the pause, resume and the drain events would be nice to spec, perhaps as optional. We may want to spec a blocking read for the sake of non-async JSGI apps that need to read the request body.

Interestingly I don't think this wouldn't actually screw up middleware as much as I'd initially thought. But in order to actually consume a request or response stream you would need the stream constructor available -- either through a require or (if we want to keep the dependency-free case possible) servers could include the constructor in the request (perhaps in request.jsgi.stream key?). In fact, as JSGI stands now you'd need to require a stream constructor if you want to modify request.input in middleware.

Having request and response body be some style of evented stream also alleviates issues around input stream buffering -- say you want to modify an incoming body in middleware -- it could look something like this:

var stream = request.body;
request.body = new request.jsgi.stream();
stream.addListener("data", function(data) {
    request.body.write(data)
});
return nextApp(request);

Modifying a response body isn't quite as easy of course -- we have to handle both the forEachable object and stream cases:


var transform = function(data) {
    return data + "..."
};
if (typeof response.body.forEach === "function") {
    var body = [];
    response.body.forEach(function(data) {
        body.push(transform(data));
    });
    response.body = body;
    return response;
}
else {
    var stream = response.body;
    response.body = request.jsgi.stream();
    stream.addListener("data", function(data) {
        response.body.write(transform(data));
    });
}


(BTW someone has to have a better way of manipulating a forEach with a supplied function -- it's shame it doesn't work like Array.map).

This isn't all that elegant, sure, but it could easily be generalized into a simple helper, so the middleware response clauses that want to modify a body could be as simple as:


require("jsgi/response").writeChunk(response.body, function(data) {
    return data + "..."
});


I'm still trying to flesh this out -- I'm working on converting Jack to the current JSGI 0.3 (sans streams), but once I get through that I'll try with async streams.

Thoughts?

[mob]

We've implemented something quite similar recently and it is working
well.

We made a single request object handle both request and response. It
is a stream with event emitter semantics. When a stream becomes
writable it issues a "writable" event and similarly for "readable".
For write, you want to know when the stream can accept more data
without blocking. Node buffers all write data, but I think a better
API is to get a writable event and to allow writes to return short.
This is more standard Posix semantics. Similarly for read.

What are you proposing to do?