Announcing Connect (middleware for node)

[Timothy Caswell]

Dear Noders,

Just a quick note to anyone who doesn't follow irc and doesn't follow me on twitter, but TJ Holowaychuk and I just released a middleware framework for node called "Connect".

It's on github under our employer's account. http://github.com/extjs/Connect

It's 100% free and MIT licensed. The goal is to provide a foundation for people to build from. We now need feedback from the framework developers in the node community to make this succeed.

There is a writeup on howtonode.org explaining it more.

-Tim Caswell

[Ryan Gahl]

Congrats and all, but it makes me curious why you don't think JSGI works well enough (the emerging standard for SSJS middleware that a bunch of projects have already adopted)...

(unless I'm missing the boat, of course)

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Timothy Caswell]

I love the work the CommonJS people are doing, I'm just not convinced it's a good idea to try to abstract away stuff like http middleware to the level where it works on both node's non-blocking style and everything else's blocking style.  I'm not a fan of over-abstracting.

I tried the whole abstract till everything looks the same with node-persistence where I was providing a layer that sat above both SQL based databases and NOSQL based datastores using the same API.  It worked, but hid and obscured a lot of the really low-level stuff that makes each platform uniquely useful.

If JSGI or isaac's EJSGI is a good solution then that's fine, I just thought we needed a good node-style option into the mix.  The end goal is really an easy way to share code and start writing real code.

I fully support CommonJS modules when it makes sense.  But having the same middleware spec for both blocking and non-blocking IO seems crazy to me.

[Daniel Smith]

On Mon, Jun 7, 2010 at 5:45 PM, Timothy Caswell <t...@creationix.com> wrote:

Dear Noders,

Just a quick note to anyone who doesn't follow irc and doesn't follow me on twitter, but TJ Holowaychuk and I just released a middleware framework for node called "Connect".

It's on github under our employer's account. http://github.com/extjs/Connect

May want to fix the references to Connect from the howtonode.org
blog entry.  They are going to http://github/com  :)

Daniel


--
Daniel Smith - Sonoma County, California
http://daniel.org/resume

[Timothy Caswell]

Thanks, it's fixed.

[Ryan Gahl]

On Mon, Jun 7, 2010 at 7:56 PM, Timothy Caswell <t...@creationix.com> wrote:

But having the same middleware spec for both blocking and non-blocking IO seems crazy to me.

Forgive me for prodding (just trying to learn as much as I can about various perspectives)... but when you look at this example from Pintura (http://github.com/kriszyp/pintura/blob/master/lib/pintura.js), does this offer any illustrations of what you're referring to? I.e. is there anything in Pintura's JSGI architecture that you'd point to and say "I feel that Connect does solves this better and here's why"?

(note, I'm not involved with Pintura actively so have no real bias; I know I reference it maybe more than any other project when I post to the group, but that's just because it's a current integration target of mine and I have an otherwise limited range of known node-friendly (JSGI) projects to reference)

[Ryan Gahl]

BTW, all my noob questioning aside... excellent work! It really does look like a nice intuitive interface. 

[Dean Landolt]

On Mon, Jun 7, 2010 at 8:56 PM, Timothy Caswell <t...@creationix.com> wrote:

I love the work the CommonJS people are doing, I'm just not convinced it's a good idea to try to abstract away stuff like http middleware to the level where it works on both node's non-blocking style and everything else's blocking style.  I'm not a fan of over-abstracting.

I tried the whole abstract till everything looks the same with node-persistence where I was providing a layer that sat above both SQL based databases and NOSQL based datastores using the same API.  It worked, but hid and obscured a lot of the really low-level stuff that makes each platform uniquely useful.

If JSGI or isaac's EJSGI is a good solution then that's fine, I just thought we needed a good node-style option into the mix.  The end goal is really an easy way to share code and start writing real code.

I fully support CommonJS modules when it makes sense.  But having the same middleware spec for both blocking and non-blocking IO seems crazy to me.

The way I see it, any abstraction like this is a good step forward. The more generalizable components that get built, be it for JSGI, EJSGI, (fab), Connect or some future framework, the better for all of us -- they would all be easy enough to port or even run through adapters on the fly. The win is that we start getting these building blocks in place -- the API details are irrelevant. I still think the JSGI interface is the most elegant (of course I'm biased) but that's neither here nor there -- the node community has been pretty slow to embrace it for one reason or another -- so if something closer to node's HTTP API gets more traction we all win.

But just to clarify this oft-abused misconception: the JSGI spec supports both sync and async -- but that doesn't mean you ever have to write code to accommodate sync. It's your server and your middleware stack. So if you like the general look and feel of the API but are scared off by the dreaded word "sync" fear not -- just Don't Do That. Ultimately, if you have something generalizable we can patch a line or two (seriously, promises are magical) and it'll handle sync and async and everyone's happy. All the while the node community never need be burdened by this. So give JSGI another look -- the request and response objects map elegantly to HTTP and there's all kinds of little symmetries and subtleties that make it a joy to work with.

[Mikeal Rogers]

Here is my perspective on Connect & JSGI.

JSGI is an attempt at a gateway interface like WSGI and Rake. Those
interfaces were initially developed to provide a sane interface
between the application layer and the server layer. The motivation
being that there are number of frameworks that should be able to run
on any compatible server. Since the gateway specification was the only
standard in the stack it was a good place to implement middleware that
could be shared between frameworks.

When i was yelling at isaacs about how eJSGI should work the first
time i met him (great introduction I'm sure) I think we both assumed
that there would be an interchangeable server component in the node
web framework stack.

Turns out, that's not the case. It's not conceivable at this time that
you would use a node web framework in production without the default
node http server, it's just too damn good.

Not needing an interchangeable server means that a gateway spec is a
bad place for interoperable middleware. I haven't spent enough time
with Connect to say that it's *the* middleware spec/implementation but
I do think that it's definitely in the right place. It's using the
standard http interfaces from node and abstracting the parts that need
to have interoperable plugins. It's a huge step in the right direction
and I'm all to happy to leave JSGI behind.

I'm sure JSGI will come back as a standard for an inter-platform web
gateway between node, narwhal, whatever other thing people use that
isn't node, but it's way too early to try and do that.

-Mikeal

[Dean Landolt]

On Mon, Jun 7, 2010 at 11:40 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

Here is my perspective on Connect & JSGI.

JSGI is an attempt at a gateway interface like WSGI and Rake. Those
interfaces were initially developed to provide a sane interface
between the application layer and the server layer. The motivation
being that there are number of frameworks that should be able to run
on any compatible server. Since the gateway specification was the only
standard in the stack it was a good place to implement middleware that
could be shared between frameworks.

When i was yelling at isaacs about how eJSGI should work the first
time i met him (great introduction I'm sure) I think we both assumed
that there would be an interchangeable server component in the node
web framework stack.

Turns out, that's not the case. It's not conceivable at this time that
you would use a node web framework in production without the default
node http server, it's just too damn good.

It's great in that it provides all the primitives necessary to interact with HTTP. XMLHttpRequest gives us (most of) the primitives we need for HTTP requests but that doesn't mean we want to write applications with raw xhr. Sure it's mostly a matter of taste, but node's HTTP API doesn't exactly lend itself to these kinds of abstractions and I think Tim's Connect examples bear this out [1] -- you've got to wrap a function just to get at the response headers? Really? I'd still like to know what's up with the next() call? The control flow is definitely non-obvious.

Here's an example of what I mean about symmetry: jsgi-client [2]. JSGI request and response objects map so well to HTTP that the same API makes a hell of a nice HTTP client. This of course also means that a proxy can be a few very obvious lines of code, among other things.

 

Not needing an interchangeable server means that a gateway spec is a
bad place for interoperable middleware. I haven't spent enough time
with Connect to say that it's *the* middleware spec/implementation but
I do think that it's definitely in the right place. It's using the
standard http interfaces from node and abstracting the parts that need
to have interoperable plugins. It's a huge step in the right direction
and I'm all to happy to leave JSGI behind.

I'm sure JSGI will come back as a standard for an inter-platform web
gateway between node, narwhal, whatever other thing people use that
isn't node, but it's way too early to try and do that.

To do what? Agree on a common higher-level HTTP interface? Why? It's not like there's not volumes of prior art and experience with this. I just don't buy this line of reasoning. If you don't like the particular API choices that is JSGI that's fine, there will be options and there will no doubt be adapters to mix and match the good stuff. But I can't believe the claim that node's low level HTTP API is the right place for interop, even amongst frameworks specifically targeting node.

[1] http://howtonode.org/connect-it/log-it.js

[2] http://github.com/kriszyp/commonjs-utils/blob/master/lib/jsgi-client.js

[Timothy Caswell]

Well said Mikeal, you understand what I'm trying to accomplish. Connect is NOT a finished project in it's current state, but rather a step in the direction I think we need to go.

Dean, I agree that jsgi is well thought out and have no clue why the node community never picked it up, but the fact is they havent, and so I'm trying this angle. Something about jsgi just never seemed right to me personally so I started from scratch assuming nothing but node's http api.

I think Connect after some molding and prodding from the community will work great as a node only middleware stack. I don't think there is anything stopping you from using jsgi and connect at the same time either, just not sure if that's a good idea.

[Marco Rogers]

I kind of agree with Dean that if interoperability was high on the
list for node, it would be happening. But it reminds me of the quote
from ry a while back when explaining his misinterpreted comments about
commonjs. Paraphrasing it was something like, "Right now we should be
focusing on operability in our respective projects rather than
interoperability". I agree with that only because ssjs is so new and
will have enough of an uphill battle to adoption without worrying
about the api churn usually brought on by integrating different
systems. And more so, there definitely isn't a lot of prior art in
developing an abstraction that will allow for both sync and async
styles in an elegant manner.

I have some issues with the way Connect is implemented, but I think it
works well for node right now. And node is currently focused on
getting things working well. Once we've got some more real world case
studies on what works and what doesn't, it'll make things that much
better when it's time to think about interoperability.

That being said, I really like the pintura example (gross nesting/
indention not-withstanding). And honestly I think that type of
wrapping paradigm could also make a good node framework. More
importantly, maybe this middleware issue shouldn't be such a big
problem for reuse of code across systems. That's what the module/
package spec is for. The middleware should be thin wrappers around a
reusable lib. That would make it trivial to port middleware for any
framework. So instead of downloading "gzip middleware for narwhal"
and "gzip middleware for node", you would download one gzip package
that includes the following modules:

1. a module for gzipping content with a sensible api
2. a lightweight middleware conforming to the node spec and using #1
3. a lightweight middleware conforming to the jsgi specs and using #1

You can use any combination of these that makes sense. The
middlewares would be 90% similar except for the http interface
adapter. Mikeal wasn't it you that convinced me of this the other
day?

I'd really like to see someone take a whack at that. It's currently
3rd on my list of things that'll never get done outside of my day
job :)

:Marco

On Jun 8, 12:26 am, Dean Landolt <d...@deanlandolt.com> wrote:

> On Mon, Jun 7, 2010 at 11:40 PM, Mikeal Rogers <mikeal.rog...@gmail.com>wrote:
>
>
>
>
>
> > Here is my perspective on Connect & JSGI.
>
> > JSGI is an attempt at a gateway interface like WSGI and Rake. Those
> > interfaces were initially developed to provide a sane interface
> > between the application layer and the server layer. The motivation
> > being that there are number of frameworks that should be able to run
> > on any compatible server. Since the gateway specification was the only
> > standard in the stack it was a good place to implement middleware that
> > could be shared between frameworks.
>
> > When i was yelling at isaacs about how eJSGI should work the first
> > time i met him (great introduction I'm sure) I think we both assumed
> > that there would be an interchangeable server component in the node
> > web framework stack.
>
> > Turns out, that's not the case. It's not conceivable at this time that
> > you would use a node web framework in production without the default
> > node http server, it's just too damn good.
>
> It's great in that it provides all the primitives necessary to interact with
> HTTP. XMLHttpRequest gives us (most of) the primitives we need for HTTP

> requests but that doesn't mean we *want* to write applications with raw xhr.

> Sure it's mostly a matter of taste, but node's HTTP API doesn't exactly lend
> itself to these kinds of abstractions and I think Tim's Connect examples
> bear this out [1] -- you've got to wrap a function just to get at the
> response headers? Really? I'd still like to know what's up with the next()
> call? The control flow is definitely non-obvious.
>
> Here's an example of what I mean about symmetry: jsgi-client [2]. JSGI
> request and response objects map so well to HTTP that the same API makes a
> hell of a nice HTTP client. This of course also means that a proxy can be a
> few very obvious lines of code, among other things.
>
>
>
> > Not needing an interchangeable server means that a gateway spec is a
> > bad place for interoperable middleware. I haven't spent enough time
> > with Connect to say that it's *the* middleware spec/implementation but
> > I do think that it's definitely in the right place. It's using the
> > standard http interfaces from node and abstracting the parts that need
> > to have interoperable plugins. It's a huge step in the right direction
> > and I'm all to happy to leave JSGI behind.
>
> > I'm sure JSGI will come back as a standard for an inter-platform web
> > gateway between node, narwhal, whatever other thing people use that
> > isn't node, but it's way too early to try and do that.
>
> To do what? Agree on a common higher-level HTTP interface? Why? It's not

> like there's not *volumes* of prior art and experience with this. I just

> don't buy this line of reasoning. If you don't like the particular API
> choices that is JSGI that's fine, there will be options and there will no
> doubt be adapters to mix and match the good stuff. But I can't believe the

> claim that node's low level HTTP API is the *right place* for interop, even

[caster]

With the history of extjs, you wonder how it might turn to GPL
afterwards when the community expands.

On Jun 8, 8:45 am, Timothy Caswell <t...@creationix.com> wrote:
> Dear Noders,
>
> Just a quick note to anyone who doesn't follow irc and doesn't follow me on twitter, but TJ Holowaychuk and I just released a middleware framework for node called "Connect".
>

> It's on github under our employer's account.http://github.com/extjs/Connect

[Timothy Caswell]

On Jun 7, 2010, at 11:15 PM, caster wrote:

> With the history of extjs, you wonder how it might turn to GPL
> afterwards when the community expands.

Seriously, what would be the incentive to change the license on Connect? This is a middleware framework, it's target audience is primarily framework and middleware developers. It's the api interface and the community support that has value, not the code itself. I wouldn't worry about this happening. Most of the people at the office now weren't around when the previous change happened anyway.

> On Jun 8, 8:45 am, Timothy Caswell <t...@creationix.com> wrote:
>> Dear Noders,
>>
>> Just a quick note to anyone who doesn't follow irc and doesn't follow me on twitter, but TJ Holowaychuk and I just released a middleware framework for node called "Connect".
>>
>> It's on github under our employer's account.http://github.com/extjs/Connect
>>
>> It's 100% free and MIT licensed. The goal is to provide a foundation for people to build from. We now need feedback from the framework developers in the node community to make this succeed.
>>
>> There is a writeup on howtonode.org explaining it more.
>>
>> -Tim Caswell
>

[Timothy Caswell]

Yes, there is nothing stopping people from writing middleware that works on everything. If/when there are multiple middleware frameworks that are all popular then we can do this.

I wouldn't worry about it for now and would like to focus on just making code that works and works well.

I'm still looking for feedback on Connect itself. What works, and what doesn't?

-Tim Caswell

[Dean Landolt]

On Tue, Jun 8, 2010 at 10:26 AM, Timothy Caswell <t...@creationix.com> wrote:

Yes, there is nothing stopping people from writing middleware that works on everything.  If/when there are multiple middleware frameworks that are all popular then we can do this.

I wouldn't worry about it for now and would like to focus on just making code that works and works well.

I'm still looking for feedback on Connect itself.  What works, and what doesn't?

Well said Tim.

[Ryan Gahl]

On Tue, Jun 8, 2010 at 9:26 AM, Timothy Caswell <t...@creationix.com> wrote:

I'm still looking for feedback on Connect itself.  What works, and what doesn't?

I'm planning to include a Connect-able interface in my WIP framework port... so when I get that far I'll let you know. I can say just from looking at it that I find the notion of .next() interesting, and may actually fork the code to see if that can be carried farther (using an FSM structure to route traffic to various layers of the app-stack as needed).

[Dean Landolt]

I'm also curious about the .next() thing. To me it's a little surprising -- why is it there? What's the distinction between calling it and just returning undefined? It seems like there has to be some kind of manager shuttling your request objects to the various components (otherwise you'd be calling .next(request) like in JSGI). So what's it's function.

Also, what's the best forum for this discussion? Should we be pounding the node list with Connect-specific questions?

[Timothy Caswell]

On Jun 8, 2010, at 9:16 AM, Dean Landolt wrote:

On Tue, Jun 8, 2010 at 11:52 AM, Ryan Gahl <ryan...@gmail.com> wrote:

On Tue, Jun 8, 2010 at 9:26 AM, Timothy Caswell <t...@creationix.com> wrote:

I'm still looking for feedback on Connect itself.  What works, and what doesn't?

I'm planning to include a Connect-able interface in my WIP framework port... so when I get that far I'll let you know. I can say just from looking at it that I find the notion of .next() interesting, and may actually fork the code to see if that can be carried farther (using an FSM structure to route traffic to various layers of the app-stack as needed).

I'm also curious about the .next() thing. To me it's a little surprising -- why is it there? What's the distinction between calling it and just returning undefined? It seems like there has to be some kind of manager shuttling your request objects to the various components (otherwise you'd be calling .next(request) like in JSGI). So what's it's function.

"next" is a callback to the next layer in the chain.  It's a callback for the cases where you don't want to call the layer right away.  Look at the static provider for an example.  It only calls next after hitting a 404 locally and that can't be determined till after spinning up the disk.  The reason it doesn't require any callbacks is because we made them optional for the common case of just re-passing through the same values.

I'm hoping that we can come up with something more elegant, especially in the area of error handling.

Also, what's the best forum for this discussion? Should we be pounding the node list with Connect-specific questions?

I think the node list is the right place for now, at least till it has more momentum.  Ryan, how do you feel about this?  You've said in the past that this is a good place for library related mail.

[Ciaran]

On Tue, Jun 8, 2010 at 3:22 PM, Timothy Caswell <t...@creationix.com> wrote:
>
>
> On Jun 7, 2010, at 11:15 PM, caster wrote:
>
>> With the history of extjs, you wonder how it might turn to GPL
>> afterwards when the community expands.
>

A *lot* of people got burned (rightly or wrongly) by that particular
switcharoo, and people have long memories :( It definitely is
something that puts me off about connect ... which is a shame as
technically it looks cool ! :)
-cj.

[Mikeal Rogers]

So, what I said, and I still think this is true, no middelware/plugin
system should buffer the data from any stream and should encourage the
use of "chained" streams.

An example use case is:

GET request -> Database query -> HTML template system -> gzip -> Response.

In the above system what you really want is a chain of streams so that
the database query returns some event emitter for "row" which makes a
call to the template engine to render another for that row which emits
a "data" event of rendered HTML which the gzipper compresses and emits
"data" which is written to the response object.

When the response write returns "false" you also want a pause event to
work it's way back up to the database query, same with the drain
events -> resume().

It doesn't matter how fast node is if we continue to wait on entire
remote IO queries to finish before we start returning them to the
Browser page load times will still bottleneck on waiting for the
queries to finish and buffering all of this in to memory means we
can't have nearly as many concurrent connections open before we run
out of memory. The above example should start rendering in the browser
before a large database query is even finished.

-Mikeal

[Ed Spencer]

Hey everyone,

I work with Tim and TJ at Ext JS and want to share my thoughts on the
licensing of Connect.

Connect is open source under the MIT license. That will never change.
Feel free to quote me on that. It is true that in the past a license
change was handled very badly - we all agree on that and it left a bad
taste in the mouths of some people.

Since then the company has moved on; the individual who made that
licensing change has moved onto new things and the team is focused on
creating great software like Connect. Those of us working at Ext JS
now do so because we love the products we get to work on, including
those projects that we get to open up to the community. I hope you're
not put off by what happened a long time ago as we'd love to get the
community standardising around this, that said we're not forcing this
on anyone and competition is good so if you want to release a project
that addresses the same need we'd be glad to check it out too.

Ed Spencer
Ext JS

[Dean Landolt]

On Tue, Jun 8, 2010 at 2:05 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

So, what I said, and I still think this is true, no middelware/plugin
system should buffer the data from any stream and should encourage the
use of "chained" streams.

An example use case is:

GET request -> Database query -> HTML template system -> gzip -> Response.

In the above system what you really want is a chain of streams so that
the database query returns some event emitter for "row" which makes a
call to the template engine to render another for that row which emits
a "data" event of rendered HTML which the gzipper compresses and emits
"data" which is written to the response object.

When the response write returns "false" you also want a pause event to
work it's way back up to the database query, same with the drain
events -> resume().

It doesn't matter how fast node is if we continue to wait on entire
remote IO queries to finish before we start returning them to the
Browser page load times will still bottleneck on waiting for the
queries to finish and buffering all of this in to memory means we
can't have nearly as many concurrent connections open before we run
out of memory. The above example should start rendering in the browser
before a large database query is even finished.

I completely agree. And this is exactly how async JSGI works (with the exception of throttling). Bodies (request and response) are provided as asynchronous streams where you can work a chunk at a time.

The streaming part is done with an object that has a forEach function on it -- interesting, this little es5 array paradigm gives us a stream-like paradigm for free -- and even provides the Array.prototype primitive to make it so that we can return raw arrays without any work... return {body: ["I look ","and act"," like ","a stream!"]}

Sometimes you just want to return a simple body -- this lets you do that without any problems.

The asynchronous bit of this comes into focus when you look at how Array.prototype.forEach works -- when it's finished it just returns undefined. So we just emulate that behavior with promises -- your custom forEach function can return a promise you resolve (to undefined, the value's unimporant) when you're finished writing out your body chunks -- it's just a signal to say that you're done. It really is that simple -- it lets you write code that's very clean and very javascripty and it doesn't come with any significant per-request penalties.

I should add that you can also push objects through your forEach stream handling. JSGI defines these forEachables as yielding objects that respond to toByteString, so if you want to be totally legal and pedantic just tack a toByteString function on your object that would serialize it to a string or buffer. Of course you don't have to be totally conformant if you're know you don't have any foreign middleware that does anything to your bodies before you decide to turn them into strings or buffers yourself.

The only missing piece of the puzzle is throttling. I personally think it could be a whole lot easier than the very low level node exposes. The JSGI request object is shared all the way down the chain -- it seems like the perfect semaphore to communicate this kind of information. It would be nice to standardize on something though, so any ideas would be greatly appreciated!

[Matt Ranney]

On Tue, Jun 8, 2010 at 11:35 AM, Ed Spencer <edw...@domine.co.uk> wrote:

I hope you're
not put off by what happened a long time ago

I think all you have to do is buy everybody dinner and drinks at Cheescake Factory in Palo Alto.  It worked for me.  I forgave Ext immediately.

[Mikeal Rogers]

I just don't think it's enough to only have a forEach.

This is basically how WSGI works and it's terrible. For one, if you
don't have data to return yet you have to check some status and return
'' everytime the iterator is called. There isn't any "drain'
equivalent that can propagate up the stack for slow clients, which
means you're still going to be buffering the writes.

We have a way to deal with all this in node, Streams. We should use them.

-Mikeal

[Timothy Caswell]

I agree, I'll see if I can work in some sort of stream interface. My current method of simply wrapping the "write" and "end" calls works sort of like a stream, but it doesn't support throttling and it a little hard to use.

I'm really hesitant to add any abstraction above node itself unless it makes a huge difference. I think streams might be a good place where it's worth it though.

[r...@tinyclouds.org]

On Tue, Jun 8, 2010 at 10:01 AM, Timothy Caswell <t...@creationix.com> wrote:
> I think the node list is the right place for now, at least till it has more
> momentum.  Ryan, how do you feel about this?  You've said in the past that
> this is a good place for library related mail.

Yes, please keep the discussion here. I'd prefer we not get fractured,
this is all interrelated.

[Dean Landolt]

On Tue, Jun 8, 2010 at 2:44 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

I just don't think it's enough to only have a forEach.

This is basically how WSGI works and it's terrible. For one, if you
don't have data to return yet you have to check some status and return
'' everytime the iterator is called. There isn't any "drain'
equivalent that can propagate up the stack for slow clients, which
means you're still going to be buffering the writes.

You may be misunderstanding javascript's forEach -- this is nothing like how WSGI works. You never have to return '' because you supply forEach with a handler function that you write to. If you have nothing to write, don't write anything -- and can buffer in a closure until you have something to write. This has nothing to do with the speed of incoming data, though -- it's just a framing problem. Sometimes the chunks your handler receives aren't enough to pass along, but throttling does not solve this problem either.

The function you provide forEach is like a filter that can operate on explicit chunks of data -- if, for instance, it's the result of a database lookup, each explicit chunk of data could be an object representing an individual row. In this the (very) common pattern you shouldn't have to worry about framing. If you want to do something to the chunk object you receive you provide a handler to do it -- they just operate one row object at a time and template into html or atom...JSON.stringify...whatever...

Throttling is irrelevant in this context, but still, I concede that it's important for other use cases, like Felix's. You probably wouldn't want your tubes clogged by a bunch of simultaneous large uploads -- having the freedom to pause/resume or apply the backoff algorithm of your choice should be possible, but I don't think it needs to be part of a stream interface. Functional handlers that handle a chunk at a time is a Good Thing.

 

We have a way to deal with all this in node, Streams. We should use them.

Just because they're there doesn't mean they're the right abstraction. They're certainly useful for library developers writing abstractions for application developers. I can't say async forEachables are, but I'm convinced Streams aren't.

[Mikeal Rogers]

> You may be misunderstanding javascript's forEach -- this is nothing like how
> WSGI works. You never have to return '' because you supply forEach with a
> handler function that you write to. If you have nothing to write, don't
> write anything -- and can buffer in a closure until you have something to
> write. This has nothing to do with the speed of incoming data, though --
> it's just a framing problem. Sometimes the chunks your handler receives
> aren't enough to pass along, but throttling does not solve this problem
> either.
> The function you provide forEach is like a filter that can operate on
> explicit chunks of data -- if, for instance, it's the result of a database
> lookup, each explicit chunk of data could be an object representing an
> individual row. In this the (very) common pattern you shouldn't have to
> worry about framing. If you want to do something to the chunk object you
> receive you provide a handler to do it -- they just operate one row object
> at a time and template into html or atom...JSON.stringify...whatever...
> Throttling is irrelevant in this context, but still, I concede that it's
> important for other use cases, like Felix's. You probably wouldn't want your
> tubes clogged by a bunch of simultaneous large uploads -- having the freedom
> to pause/resume or apply the backoff algorithm of your choice should be
> possible, but I don't think it needs to be part of a stream interface.
> Functional handlers that handle a chunk at a time is a Good Thing.

Why shouldn't pause/resume/drain be part of the standard interface? If
it's not then the interface encourages buffering.

I do understand what you *can* do with forEach but it seems like the
JSGI usage of forEach is bending over backwards to provide for a
simpler blocking interface without requiring blocking by allowing a
forEachable object. All the forEach voodoo aside, actually building
stuff on this this level of abstraction is a pain in the ass and the
harder it is the more people will do things the *easier* way (which is
to buffer).

In my opinion, we need to create a good, if not great, plugin
interface that discourages blocking and buffering. We can worry about
interop with other platforms later once we have some solid for node.
There is nothing in the Stream interface that can't be implemented in
other platforms like narwhal it just *doesn't* allow for a blocking
interface.

-Mikeal

[Dean Landolt]

forEachables are built for streams -- it's rather convenient that you can also return a direct type (an array) and have it work...but that's a sideshow. Don't let it distract you...

 

All the forEach voodoo aside, actually building
stuff on this this level of abstraction is a pain in the ass and the
harder it is the more people will do things the *easier* way (which is
to buffer).

Building stuff at this level without buffering is insanely easy -- it's reasonable to do without any helpers but throw in some a sugary library and you're in heaven. Here's one such library:

http://github.com/kriszyp/commonjs-utils/blob/master/lib/lazy-array.js

The idea of a lazy array-like responses isn't bending over backward for sync -- it's amazing in its own right. It allows us to treat our response bodies as array-like things. Imagine being able to run wu.js over your database results, record by record. That's effectively what you get with this forEachable response thing. It's array-like -- you get all the es5 array extras and you can use all of them indiscriminately (except reverse...don't do that).

How is that a pain in the ass?

 

In my opinion, we need to create a good, if not great, plugin
interface that discourages blocking and buffering. We can worry about
interop with other platforms later once we have some solid for node.
There is nothing in the Stream interface that can't be implemented in
other platforms like narwhal it just *doesn't* allow for a blocking
interface.

The community discourages blocking and buffering...isn't that enough? 

 

-Mikeal

[Marco Rogers]

Dean I remember lurking on the commonjs mailing list when JSGI was being solidified, and I followed all of the logic that led to forEachables.  But at that time it was still pretty abstract in my head, particularly in the case of using it in an async fashion.  Now that I have a more concrete mental model of asynchronous http because of node, I want to revisit it.  Your explanation is intriguing. Thanks for the run down.

I'm of two minds about this in terms of the API though.  One the one hand it is nice that JSGI requires only javascript constructs and you can work with streams as if they were arrays.  Node requires you to wrap your head around the stream api and learn to use it effectively.  It becomes a critical extension to the language/platform when you're using node.  But that's not different than what frameworks like django makes you learn in order to use them.  Nobody really deals with raw WSGI so I can't say what the strictly analogous comparison is.

On the other hand, I think the Stream api gives you more flexibility for creating interfaces that are more descriptive.  If you're just doing body.forEach, there's really no context for what's coming out of there.  It could be random chunks, it could be whole db rows, it could be json objects, it could be one item with an entire response body.  

But you can set your stream emitter interface to make the output explicit:

    var resultStream = db.query(...);

    resultStream.addListener('result_row', function(row) { ...});

This is pretty straight forward and this code can be used by anything from a template engine to a hand-rolled output script.  If feels to me like JSGI is nice for http round trips but using it in other contexts will be less clear cut. And in fact it wasn't developed to be used in other contexts.  Now that doesn't mean using Streams for http is Right with a capital R, just that it works for well for http and other async contexts.  And it cuts down on the different mental models you have to maintain.  And all of that is important for node.

I'm prepared to be wrong about any of this stuff.  The debate is very helpful for those of us looking at developing frameworks and libraries.

:Marco

--
Marco Rogers
marco....@gmail.com

Life is ten percent what happens to you and ninety percent how you respond to it.
- Lou Holtz

[Dean Landolt]

On Tue, Jun 8, 2010 at 6:02 PM, Marco Rogers <marco....@gmail.com> wrote:

Dean I remember lurking on the commonjs mailing list when JSGI was being solidified, and I followed all of the logic that led to forEachables.  But at that time it was still pretty abstract in my head, particularly in the case of using it in an async fashion.  Now that I have a more concrete mental model of asynchronous http because of node, I want to revisit it.  Your explanation is intriguing. Thanks for the run down.

I'm of two minds about this in terms of the API though.  One the one hand it is nice that JSGI requires only javascript constructs and you can work with streams as if they were arrays.  Node requires you to wrap your head around the stream api and learn to use it effectively.  It becomes a critical extension to the language/platform when you're using node.  But that's not different than what frameworks like django makes you learn in order to use them.  Nobody really deals with raw WSGI so I can't say what the strictly analogous comparison is.

The same people that would use JSGI would use WSGI -- library authors. Someone has to -- and then others get to assemble the pieces together. It's nice, but I agree, not the only way.

 

On the other hand, I think the Stream api gives you more flexibility for creating interfaces that are more descriptive.  If you're just doing body.forEach, there's really no context for what's coming out of there.  It could be random chunks, it could be whole db rows, it could be json objects, it could be one item with an entire response body.  

But you can set your stream emitter interface to make the output explicit:

    var resultStream = db.query(...);

    resultStream.addListener('result_row', function(row) { ...});

I love explicit! Try this on and see how it fits:

return require("commonjs-utils/lazy-array").LazyArray({

    some: function(write){

        var resultStream = db.query(...);

        resultStream.addListener('result_row', function(row) {

            write(row);

        });

    },

    totalCount: totalCount,

    length: length

});

That's how responses are returned from perstore. The totalCount and length props are optional but are useful for setting the Content-Range header in an HTTP response. Of course, perstore doesn't care about HTTP, it just cares about returning a lazy-array as a recordset. This is how these kind of streams are useful in a non-node context. Forget JSGI -- you can use perstore from node, or narwhal, or ringo, or flusspferd, or whatever (eventually, your browser) and you get this nice stream interface for free. This isn't necessarily about HTTP -- this is an interface to lazily stream objects...that's all. It's mighty convenient (and just damn good engineering) that this works as a JSGI response body (or request body if you have parsed objects you want to push down into your app).

[Timothy Caswell]

After meeting with several node people in San Francisco today, I'm convinced that I need to abstract a little more than I currently do in connect.  I'm about to go and start implementing it, but thought I'd throw out the idea to the mailing list for further feedback.  The gzip filter in the new format would look something like this:

exports.onResponse = function (obj, in, out) {

  var accept = obj.req.headers["accept-encoding"];

  var type = obj.res.headers["Content-Type"];

  

  if (!(obj.res.code === 200 && accept && accept.indexOf('gzip') >= 0

        && type && (/(text|javascript)/).test(type))) {

      var gzip = child_process.spawn("gzip", ["-9"]);

      pump(in, gzip.stdin);

      pump(gzip.stdout, out);

      

  } else {

    pump(in, out);

  }

};

Of course "in" is a reserved keyword and all, but it's the general idea of the code.  There is a combined req and res object in obj.  There are two hooks to get at the stream, they are onResponse and onRequest.

Instead of having to manually wrap the "write" and "end" calls to modify the stream, you just plug streams together using nice utility functions like pump.  I resisted abstracting the streams, but I think it's the only sane way to do it.

I'll probably modify this a little as I implement it, but this is the initial idea.  Any comments or questions?

-Tim Caswell

[Phill Sparks]

On Thu, Jun 10, 2010 at 05:49, Timothy Caswell <t...@creationix.com> wrote: 

exports.onResponse = function (obj, in, out) {

  var accept = obj.req.headers["accept-encoding"];

  var type = obj.res.headers["Content-Type"];

  

  if (!(obj.res.code === 200 && accept && accept.indexOf('gzip') >= 0

        && type && (/(text|javascript)/).test(type))) {

      var gzip = child_process.spawn("gzip", ["-9"]);

      pump(in, gzip.stdin);

      pump(gzip.stdout, out);

      

  } else {

    pump(in, out);

  }

};

I'll be curious to watch how you access the response headers.  AFAICS the statusCode and headers are not currently stored on the response (except as text on _header).  When calling writeHead headers may be passed as an object (key + value) or as an array of arrays, so there's no guaranteed object access.  And to even further complicate matters, the header keys are case insensitive so lazy developers could be writing 'content-type'.

Is there a response layer in Connect that I've missed?  I've not had a chance to go through the docs or code in any detail yet.

[Timothy Caswell]

So the assumption is that connect would wrap the node request and response a little and provides you with a slightly abstracted version of this.  The header casing problem is still unsolved.  Keep in mind that this alternative API is still pseudocode and not implemented.  Connect's current code is a very simple layering engine directly on top of node.

[Kris Zyp]

Based on the examples, it looks Connect is more about wiring
middleware together than trying invent yet another new form of
middleware, which I think is a useful niche. What would be awesome is
if you could use Connect to write middleware stacks like:

require('./lib/connect').createServer([
{ filter: 'log', type: 'connect' }, // use a connect middleware
app
{ filter: 'compress', type: 'jsgi' }, // use a jsgi middleware app
{ module: fab.stringify, type: 'fab' } // use a fab app without
parenthesis :)
]);

I think that would be incredibly useful and very beneficial to the
ecosystem of NOde middleware already available (http://wiki.github.com/
ry/node/modules#middleware) for JSGI and fab rather than focusing on
making even more difficult to combine middleware with yet more new
interfaces. I'd be glad to create a JSGI<->node adapters that could be
used in connect, although unfortunately I don't think I could directly
contribute to this project due to the licensing liability of
association with extjs.

Thanks,
Kris