My Humble Co-routine Proposal

566 views
Skip to first unread message

Tim Caswell

unread,
Nov 9, 2011, 8:34:32 AM11/9/11
to nod...@googlegroups.com
After endless debates on the best way to "fix" callbacks with solutions ranging from code transformers to fibers to real threads, I've had some time to think about this problem myself.

As an experiment, I "ported" (more like re-implemented) node to the lua language as Luvit < https://github.com/creationix/luvit>.  I learned three very important things in this experiment.

  1. Lua is not JavaScript.  It's pretty darn close and I like both for unique reasons, but JavaScript is *far* more popular due to this thing we call the internet.
  2. V8's JS to C++ conversion is really slow compared to luajit's Lua to C layer.  Also V8 uses a lot more ram and starts up much slower.
  3. Co-routines in Lua are really nice and with very little extra sugar make a simple, obvious, and effective way to write sync style code while living in an async world.

Because of discovery #1, I don't think Luvit will ever become more popular than nodeJS, especially for anything web related.  I see Luvit being used for things like mobile devices where performance *really* matters, JIT can be disabled (think iOS), and the built-in FFI makes SDL and openGL possible without bindings.

I do think that we can take what I learned from Lua's co-routines and bring it to node with minimal changes to the language.  This proposal needs two things:

  1. A new co-routine primitive as either a node add-on or an addition to the language.
  2. De-structuring assignment to keep the syntax sane since async functions tend to "return" more than one value.

In this proposal you create a co-routine explicitly outside of the main stack.  (The main stack is *never* suspended).  Within that new stack, you can `wait()` and `resume()`.  This makes it possible to use any normal non-co-routine-aware async function and using function composition, it's trivial to make your co-routine an async function with a callback as well.

This integrates well with existing node code and doesn't introduce any new hazards that aren't already there.  No new hazards is a hard requirement for anything like this.  Programming in node is hard enough already, let's not make it any harder.

For an example, see this: <https://gist.github.com/1349451>

Axel Kittenberger

unread,
Nov 9, 2011, 10:14:10 AM11/9/11
to nod...@googlegroups.com
I'm watching with great interest how Luvit develops. Considering it
great what you are doing there! Currently I have another node
unrelated project (Lsyncd http://code.google.com/p/lsyncd/ ) where I
ended up writing my own event machine core with Lua scripting for its
specific use case (file event monitoring). Might be very worthwhile to
transfer it to Luvit once it reaches adolescence. For some more funky
set ups I did started to code also unblocking network transfer in
core, but going into a shared environment with that things might be
very worthwhile, since there people do this things that know better
what they are doing.

Yes coroutines. true non-threaded pure cooperative in scripting engine
processing is something very nice that is missing in Javascript
language spec. But wouldn't this be more of a V8 thing to support or a
heavy patch to it than a node thing?

I considered also moving Lsyncd to node instead and recoding it from
Lua over to Javascript, but JS is missing few features it relies upon.
For me most important, true associative arrays (that is table keys can
be anything not just integers and strings) and weak link tables
(garbage collector can remove it, if an object is only referenced by
such tables) So I rather wait for Luvit for this :-) On the other
hand, I like how JS offers me well established conveniences like
"switch", "continue", ++ operators, ?: operator, where Lua-ism can be
a bummer.

I currently work on another web project that will surely be fixed onto
node, because javascript server and client side gives here some
interesting synergy, because it shares quite some bunch of code
between server and client (and also for some people having to
work/learn with 1 language only is also a bon).

Javascript and Lua. I always think as analogy C and Pascal a decade or
two ago. The former is the more industrial language, the later the
more academic. While the later is actually the tad "cleaner" language
(whatever that means) on the large scale the formers industry power
just outruns. That doesn't remove interesting niches for the later
tough.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Axel Kittenberger

unread,
Nov 9, 2011, 10:21:03 AM11/9/11
to nod...@googlegroups.com
> For an example, see this: <https://gist.github.com/1349451>

BTW: I recently coded up something very similar if I see this correctly:

https://github.com/axkibe/node-green-light

Didn't echo on node mailinglist a lot. Got just told to use
streamline. Fiber-anything got a massive negative touch to it. (And
yes while I also dislike pthreads a great deal, there is nothing wrong
with true (green) coroutines). IMHO if there would be true V8 support
for coroutines, instead of hacking pthreads with global locks, things
might look different, would they?

Jann Horn

unread,
Nov 9, 2011, 11:24:59 AM11/9/11
to nod...@googlegroups.com
2011/11/9 Axel Kittenberger <axk...@gmail.com>:

> with true (green) coroutines). IMHO if there would be true V8 support
> for coroutines, instead of hacking pthreads with global locks, things
> might look different, would they?

Nah, it'd still be confusing (if I understand you correctly). Example:

function User(name) {
...
}

User.prototype.setPassword = function(newPassword) {
var salt = makeRandomChars(16)
this.store('salt', salt);
var passwordHash = hash(newPassword, salt);
this.store('hash', passwordHash);
}

As soon as "store" becomes a blocking function, executing this
function twice might well cause you to have corrupt data in your DB.
Great. In streamlinejs, I'd be able to see the underscore and could
figure out that some caution is needed here.
What's wrong with streamlinejs? It seems to work, it has useful stack
traces, it doesn't really introduce large run-time performance
penalties... and you can even use it in the browser. If you have a
tool that works, why make a magic wand? Sure, it looks impressing and
you can do some weird magic with it that nobody can master without
studying gray magic for some time, but that's not what I expect a good
developer to want. I expect him to want something unambiguous that
just works.

PS: Yes, I do realize that that example would create inconsistent
state if the program fails at some point between those store calls.
Let's just assume it's an in-memory DB that gets cleared after each
restart so that the example becomes even more contrived.

Axel Kittenberger

unread,
Nov 9, 2011, 11:57:01 AM11/9/11
to nod...@googlegroups.com
> Nah, it'd still be confusing (if I understand you correctly). Example:

IMHO you didn't understand coroutines. This doesn't just doesn't mean
that anything can suddendly block/reenter.

As Tim wrote

"In this proposal you create a co-routine explicitly outside of the
main stack. (The main stack is *never* suspended). Within that new
stack, you can `wait()` and `resume()`. This makes it possible to use
any normal non-co-routine-aware async function and using function
composition, it's trivial to make your co-routine an async function
with a callback as well."

No function will wait() resume() which did not create the coroutine in
first place. In this proposal (and with greenlight) it even cannot,
since it misses the relevant handlers. If the "magic" handler is
called '_' or something else is a matter of taste. Its imho this basic
misunderstanding that causes the promp refusal. Think of a coroutines
basically just as one routine.

You might actually follow one of the links to see the proposals.

Tim Caswell

unread,
Nov 9, 2011, 12:07:37 PM11/9/11
to nod...@googlegroups.com
Anything that looks like blocking code without the user having to use
slightly special syntax is evil. Callbacks are great and viral in
this regard. They are just difficult for certain things and debugging
is tricky with the short stack traces. My proposed coroutines are
good because you have to explicitly call wait() to suspend your
co-routine and you can only do this within the body of a
Fiber(function () { ...}) block. The call to Fiber is non-blocking
itself so it's caller isn't affected at all. This explicit and
obvious place where state can change out from under you is the same as
callbacks and thus acceptable. The underscore in streamline is like
that too, but code transformation seems a little too much for this.
But that's just my personal taste.

Also I personally usually just vanilla callbacks for everything
because I'm used to it. I have some background in functional
programming and it's natural to me. But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.

Dean Landolt

unread,
Nov 9, 2011, 12:30:40 PM11/9/11
to nod...@googlegroups.com
On Wed, Nov 9, 2011 at 12:07 PM, Tim Caswell <t...@creationix.com> wrote:
Anything that looks like blocking code without the user having to use
slightly special syntax is evil.  Callbacks are great and viral in
this regard. They are just difficult for certain things and debugging
is tricky with the short stack traces.  My proposed coroutines are
good because you have to explicitly call wait() to suspend your
co-routine and you can only do this within the body of a
Fiber(function () { ...}) block.  The call to Fiber is non-blocking
itself so it's caller isn't affected at all.  This explicit and
obvious place where state can change out from under you is the same as
callbacks and thus acceptable.  The underscore in streamline is like
that too, but code transformation seems a little too much for this.
But that's just my personal taste.

Also I personally usually just vanilla callbacks for everything
because I'm used to it.  I have some background in functional
programming and it's natural to me.  But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.


This is a nice idea, but ISTM people that prefer a synchronous style will just end up pushing their whole app into the body of a fiber to get at the wait/resume functionality. This doesn't actually fix the interleaving hazards, it just pushes them down a level. The only way to fix them is with single-frame continuations, and for that you need syntax-level support. Conveniently, this is already being added to the language by way of generators :D

Sure, generators alone won't get us all the way to your nice, neat Fiber blocks -- we'll still need library support. Whether that ends up in core (or even as part of the language as Kris Zyp recently proposed on es-discuss), it's too early to tell. But Dave Herman's taskjs[1] has the notion of Task blocks that are an awful lot like your Fiber blocks, but without the hazards. Would that get us the rest of the way there? If so, I wonder if time may be better spent patching up the generator bits in traceur as a stopgap until v8 gets around to giving us native support.

Ted Young

unread,
Nov 9, 2011, 12:33:15 PM11/9/11
to nod...@googlegroups.com
What would the pattern for doing several async operations in parallel
and waiting for them to finish look like?

Ted

Axel Kittenberger

unread,
Nov 9, 2011, 1:05:52 PM11/9/11
to nod...@googlegroups.com

a(resume);
b(resume);
c(resume);
d(resume);
wait();
wait();
wait();
wait();

Looked it up, generators are effectively indeed the same thing as coroutines - as far I see it. Suppose then we just have to wait until ECMA 1.7 comes to V8


For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

Tim Caswell

unread,
Nov 9, 2011, 2:18:23 PM11/9/11
to nod...@googlegroups.com
@Ted, I've never thought that faux blocking was good for parallel
tasks. I would just use vanilla callbacks (or if you prefer promises)
for that, but as Axel pointed out, it's almost possible with my
proposal, the tricky part is you don't know which resume happens first
so getting any interesting data out of the resume callback is
impossible. The first wait will suspend the co-routine, but the first
resume called (which is unknown since the I/O waits are parallel and
a,b,c, and d are async) will resume it and pass the results to the
first wait, then it will suspend on the second wait and so forth. The
last wait won't resume till all four resumes have been called in any
order.

@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need. Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has? Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows. From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard. This includes promises and CPS style callbacks
as well as my proposal. The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.

>>>> nodejs+un...@googlegroups.com


>>>> For more options, visit this group at
>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>
>>>
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to nod...@googlegroups.com
>>> To unsubscribe from this group, send email to

>>> nodejs+un...@googlegroups.com


>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nod...@googlegroups.com
>> To unsubscribe from this group, send email to

>> nodejs+un...@googlegroups.com


>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com

Mikeal Rogers

unread,
Nov 9, 2011, 2:39:46 PM11/9/11
to nod...@googlegroups.com
On Nov 9, 2011, at November 9, 20115:34 AM, Tim Caswell wrote:

After endless debates on the best way to "fix" callbacks with solutions ranging from code transformers to fibers to real threads, I've had some time to think about this problem myself.

Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily handled with named functions that I start to grit my tether whenever someone makes a comment like this.

As an experiment, I "ported" (more like re-implemented) node to the lua language as Luvit < https://github.com/creationix/luvit>.  I learned three very important things in this experiment.

  1. Lua is not JavaScript.  It's pretty darn close and I like both for unique reasons, but JavaScript is *far* more popular due to this thing we call the internet.
  2. V8's JS to C++ conversion is really slow compared to luajit's Lua to C layer.  Also V8 uses a lot more ram and starts up much slower.
  3. Co-routines in Lua are really nice and with very little extra sugar make a simple, obvious, and effective way to write sync style code while living in an async world.

Because of discovery #1, I don't think Luvit will ever become more popular than nodeJS, especially for anything web related.  I see Luvit being used for things like mobile devices where performance *really* matters, JIT can be disabled (think iOS), and the built-in FFI makes SDL and openGL possible without bindings.

I do think that we can take what I learned from Lua's co-routines and bring it to node with minimal changes to the language.  This proposal needs two things:

  1. A new co-routine primitive as either a node add-on or an addition to the language.
  2. De-structuring assignment to keep the syntax sane since async functions tend to "return" more than one value.

There is a very good destructuring proposal that has made it in to ES.next. I'm known for being anti new features in JavaScript and even I like the proposal, it's very clean and the syntax makes the semantics very obvious.


In this proposal you create a co-routine explicitly outside of the main stack.  (The main stack is *never* suspended).  Within that new stack, you can `wait()` and `resume()`.  This makes it possible to use any normal non-co-routine-aware async function and using function composition, it's trivial to make your co-routine an async function with a callback as well.

So you're using a v8 isolate per actor?

That is what common-node does. It's slow, we've considered it before in node.js and it's too slow.


This integrates well with existing node code and doesn't introduce any new hazards that aren't already there.  No new hazards is a hard requirement for anything like this.  Programming in node is hard enough already, let's not make it any harder.

For an example, see this: <https://gist.github.com/1349451>


Dean Landolt

unread,
Nov 9, 2011, 2:41:43 PM11/9/11
to nod...@googlegroups.com
On Wed, Nov 9, 2011 at 2:18 PM, Tim Caswell <t...@creationix.com> wrote:
@Ted, I've never thought that faux blocking was good for parallel
tasks.  I would just use vanilla callbacks (or if you prefer promises)
for that, but as Axel pointed out, it's almost possible with my
proposal, the tricky part is you don't know which resume happens first
so getting any interesting data out of the resume callback is
impossible. The first wait will suspend the co-routine, but the first
resume called (which is unknown since the I/O waits are parallel and
a,b,c, and d are async) will resume it and pass the results to the
first wait, then it will suspend on the second wait and so forth. The
last wait won't resume till all four resumes have been called in any
order.

@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need.  Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has?  Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows.  From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard.  This includes promises and CPS style callbacks
as well as my proposal.  The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.


Indeed -- making the preemption points explicit, as a convention, goes a long way toward solving the problem. But making it impossible to preempt without some kind of explicit sigil is the only real solution. We all cheat; if cheating with a wait can be done, it will.

But actually I was under the impression that with your proposal you could still bury a wait call deep in the call stack of a Fiber. Looking closer, it appears that you may have sidestepped this problem with a clever API. Am I right in assuming you have to pass the wait handle down through the call stack in order to preempt? If so that's an interesting improvement on typical fibers -- the very act of passing around the wait handle is your sigil that a call may be preempted.

I'm not sure I understand your concern about argument passing in generators but I suspect the `send` API surface should cure what ails ya.

Ted Young

unread,
Nov 9, 2011, 3:35:50 PM11/9/11
to nod...@googlegroups.com

On Nov 9, 2011, at 11:18 AM, Tim Caswell wrote:

> @Ted, I've never thought that faux blocking was good for parallel
> tasks. I would just use vanilla callbacks (or if you prefer promises)
> for that, but as Axel pointed out, it's almost possible with my
> proposal, the tricky part is you don't know which resume happens first
> so getting any interesting data out of the resume callback is
> impossible. The first wait will suspend the co-routine, but the first
> resume called (which is unknown since the I/O waits are parallel and
> a,b,c, and d are async) will resume it and pass the results to the
> first wait, then it will suspend on the second wait and so forth. The
> last wait won't resume till all four resumes have been called in any
> order.

Have you seen tamejs? What do you think of their solution to this
problem:

var res1, res2;
await {
doOneThing(defer(res1));
andAnother(defer(res2));
}
thenDoSomethingWith(res1, res2);


Axel Kittenberger

unread,
Nov 9, 2011, 5:10:06 PM11/9/11
to nod...@googlegroups.com
On Wed, Nov 9, 2011 at 8:18 PM, Tim Caswell <t...@creationix.com> wrote:
> @Ted, I've never thought that faux blocking was good for parallel
> tasks.  I would just use vanilla callbacks (or if you prefer promises)
> for that, but as Axel pointed out, it's almost possible with my
> proposal, the tricky part is you don't know which resume happens first
> so getting any interesting data out of the resume callback is
> impossible. The first wait will suspend the co-routine, but the first
> resume called (which is unknown since the I/O waits are parallel and
> a,b,c, and d are async) will resume it and pass the results to the
> first wait, then it will suspend on the second wait and so forth. The
> last wait won't resume till all four resumes have been called in any
> order.

Certainly not impossible. Just have to wrap the callback to wrap the
answer where it came from.

a( function(err, asw) { resume(['a', arguments]); }
b( function(err, asw) { resume(['b', arguments]); }
c( function(err, asw) { resume(['c', arguments]); }
d( function(err, asw) { resume(['d', arguments]); }
results = {};
for (var i = 0; i < 4; i++) {
var asw = wait();
results[asw[0]] = results[asw[1]];
}

Well looks a tad more bloated already. Certainly not simpler than the
async version would be. Albeit as Tim said, this is not the strong end
of the concept, but it still works. Might also be offloaded to a
convenience library, like async stuff does anyway.

> Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily
> handled with named functions that I start to grit my tether whenever someone makes a comment like this.

It depends on the usecase, when logic gets more complex with classic
callbacks I often see myself writing statemachines. While
statemachines are great and efficient once they work right, humans (or
at least I) are not great in getting them proper or debugging them.
For small tasks and handlers I do classic callbacks, for more complex
logic I now use streamline, like that database data analyzer I
recently worked. Streamline does quite well in moving some of the
implementation details out of the way, so I can work on a more logical
level. I haven't yet figured out how to interprete its linenumbers in
backtraces. I honestly don't care about microseconds in my use case.
However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.

Axel Kittenberger

unread,
Nov 9, 2011, 5:20:15 PM11/9/11
to nod...@googlegroups.com
> Indeed -- making the preemption points explicit, as a convention, goes a
> long way toward solving the problem. But making it impossible to preempt
> without some kind of explicit sigil is the only real solution. We all cheat;
> if cheating with a wait can be done, it will.
> But actually I was under the impression that with your proposal you could
> still bury a wait call deep in the call stack of a Fiber. Looking closer, it
> appears that you may have sidestepped this problem with a clever API. Am I
> right in assuming you have to pass the wait handle down through the call
> stack in order to preempt? If so that's an interesting improvement on
> typical fibers -- the very act of passing around the wait handle is your
> sigil that a call may be preempted.

Correctly. For the implementation with fibers at least as long you do
not out trick by requiring the fibers modules directly. With that sort
of API Tim and I suppose have done more or less the same with
greenlight. if you did not get the wait handler down the stack for the
current context (or red sign as I called it) you cannot yield.

Mikeal Rogers

unread,
Nov 9, 2011, 5:32:07 PM11/9/11
to nod...@googlegroups.com

On Nov 9, 2011, at November 9, 20112:10 PM, Axel Kittenberger wrote:

However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.

I can't overstate how optimized closures are in modern javascript vm's. They are crazy, crazy, crazy fast.

In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).

A language designed from the ground up to support them, like lua, might be able to do a better job optimizing. But if you look at the overhead of coroutines in erlang it's much more than callbacks in node.js using closures.

You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.

I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.

In order to keep this thread on track can we please refrain from making any assumptions or claims about potential optimizations at the VM level.

node.js does not modify the vm, that's been a rule since day 1. if fibers work without vm modifications i'm sure Tim's proposal could be implemented without modifications to the vm as well. 

Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.

-Mikeal

Marak Squires

unread,
Nov 9, 2011, 5:39:07 PM11/9/11
to nod...@googlegroups.com
Tim -

Ignoring any of the technical merits, this seems to me that it will cause a fracture in node.js module development.

My personal opinion is that introducing and/or encouraging co-routines will just cause more mental points of failure for people starting out with node. 

Even if it makes sense on paper, adding more complexity to a system will only result in more confusion.

Learning async control flow can be daunting for new developers. If we have people going off and building co-routine based modules, new node developers will now have to learn two new things. 

I'd be interested to hear your thoughts on my perspective. 


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en



--
-- 
Marak Squires
Co-founder and Chief Evangelist
Nodejitsu, Inc.

_doq

unread,
Nov 9, 2011, 5:57:08 PM11/9/11
to nod...@googlegroups.com
Rant warning!

I came to this group based off a comment from polotek about:
 1. There is a group, and the devs are super involved in it!
 2. There were a few uneducated programmers attempting to change the whole of nodeJs via combating what they think is a problem with callbacks.

At first I was like, hmmm don't seem to be that much static out here... boy was I wrong. I am glad you like lua, but the issue with callbacks in every case I have seen is user error, as in they really don't get it. Maybe there is a really big learning curve I am not seeing, but all I can see is a bunch of stubborn coders trying to convert others to there way of thinking rather getting on board with the coop para-dime, this reminds me of the transition from procedural programming to oop.

Stuff like this makes me not want to even look at your code, "Lua is not JavaScript.  It's pretty darn close," yeah... it is just as close as C is to VB. I am really surprised though all your attempts you didn't just learn to use callback more elegantly rather then trying to re-core the language.

Rant End!

Jorge

unread,
Nov 9, 2011, 6:09:44 PM11/9/11
to nod...@googlegroups.com
On 09/11/2011, at 23:32, Mikeal Rogers wrote:
> (snip)

>

> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).


Isolated CO-routines ? Care to explain how would that work or what do you mean ?

> (snip)

>
> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.

What overhead?

The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.

Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.

> (snip)


>
> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.


See below.

> (snip)

>

> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.

Yeah. You first, please.
--
Jorge.

Isaac Schlueter

unread,
Nov 9, 2011, 6:22:02 PM11/9/11
to nod...@googlegroups.com
I think that some time this month, we should have some kind of a
gathering of all the people with a vested interest in cooperative
multithreading approaches in node, to form a committee to come up with
a solid proposal. We'll call it the November New Node Co-Routine
Proposal Committee. We can meet somewhere around the lower west side
in Manhattan, NYC.

NoNewNoCoRoProCo, SoHo.

Mikeal Rogers

unread,
Nov 9, 2011, 6:23:59 PM11/9/11
to nod...@googlegroups.com

On Nov 9, 2011, at November 9, 20113:09 PM, Jorge wrote:

> On 09/11/2011, at 23:32, Mikeal Rogers wrote:
>> (snip)
>
>>
>
>> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).
>
>
> Isolated CO-routines ? Care to explain how would that work or what do you mean ?

In javascript you avoid many side effects by having a "lock" on the whole vm while your code runs. You "block" the entire VM and state can only mutate in between callbacks. So if i do something like:

var blah = 'asdf'

doSomething(function (text) {
blah = text
require('module').doSomethingelse()
write(blah)
})

I **know** that i'm writing the value of text that was passed to MY FUNCTION, even tho this variable can be mutated from outside of my scope because I only call a function that doesn't have access to that scope. I avoid side effects from another doSomething callback that might be running in "parallel". This is a common problem in threaded programming.

Let's say this callback is run many times in parallel, and that doSomethingelse() is a fiber that can go off and do IO. Other doSomethign callbacks are going to fire in parallel and mutate blah. The way to avoid this is to create an isolate that copies the state from the parent isolate and any mutations are copy-on-write so i can't mutate the state of the other isolates.

If you don't do that you get all the same horrible side effects that plague threaded programming.

>
>> (snip)
>
>>
>> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.
>
> What overhead?
>
> The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.

No, with fibers the stack is "yeilded" and stays around, the new stack goes off and does IO and pauses the parent stack until it returns. With callbacks you exit out of the event loop and the stack goes away, the function that is referenced in the event system stays around because there is a reference to it.

>
> Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.
>
>> (snip)
>>
>> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.
>
>
> See below.
>
>> (snip)
>
>>
>
>> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.
>
> Yeah. You first, please.
> --
> Jorge.
>

Matt

unread,
Nov 9, 2011, 6:36:04 PM11/9/11
to nod...@googlegroups.com
Please don't forget about the W16 project too - that idea shows some real promise IMHO: https://github.com/sheremetyev/w16

Jorge

unread,
Nov 9, 2011, 6:55:46 PM11/9/11
to nod...@googlegroups.com
On 10/11/2011, at 00:36, Matt wrote:

Please don't forget about the W16 project too - that idea shows some real promise IMHO: https://github.com/sheremetyev/w16

Promising. Somebody said here not too long ago that "Threads do not scale on cores as people might think they do", buuut, this is entirely based on threads and shared memory, not on separate processes.
-- 
Jorge.

Tim Caswell

unread,
Nov 9, 2011, 7:23:47 PM11/9/11
to nod...@googlegroups.com
Mikeal, I'm not sure if you're talking about my proposal. I promise
it's no more dangrous than async callbacks aready are. In fact, any
function you call that has a co-routine somewhere in it is still a
normal law abiding non-blocking function call itself. It won't pause
you and allow things to change out from under you any more than a call
to setTimeout would. It doesn't matter what's in the body of setTimeout
because the call to setTimeout returns right away and you can be sure
your local state wasn't modified. Don't confuse this with the evil that
was promise.wait in the early days of node. That was truly dangerous
and I'm glad it was removed early on.

There is no need for isolates here. There is still only only one active
stack at a time and the main "thread" (as lua calls it) or stack can
never be suspended. Since the main "thead" can never be suspended,
there is no interleaving hazard except when the main "thread" explicitly
yields by letting the call stack unwind.

As far as implementation, I'm sure that's possible. node-fibers does
much of what I need and also the proposed new features Dean mentions
also seem to do what I need. The thing I want to propose is a sane
solution that's not any more dangerous than callbacks already are and is
as minimal as possible.

Going back to the hazard example:

var blah = 'asdf'

doSomething(function (text) {
blah = text
require('module').doSomethingelse()
write(blah)
})

This is actually not safe even in a syncronous world without callbacks
or co-routines if the module has a reference to the doSomething function
(IE it re-enters). But assuming you don't make that simple mistake, yes
it's safe. Adding async calls into doSomethingelse() or my proposed
co-coroutine API doesn't change this in any way. Unless one of your
direct descendents in the call stack re-calls doSomething before it
returns, you can assume that blah still holds the value you put in it.

This is not true of all co-routine implementations which is why I'm
making this distinction.

Tim Caswell

unread,
Nov 9, 2011, 7:27:38 PM11/9/11
to nod...@googlegroups.com
On 11/09/2011 04:39 PM, Marak Squires wrote:
> Tim -
>
> Ignoring any of the technical merits, this seems to me that it will
> cause a fracture in node.js module development.
>
> My personal opinion is that introducing and/or encouraging co-routines
> will just cause more mental points of failure for people starting out
> with node.
>
> Even if it makes sense on paper, adding more complexity to a system
> will only result in more confusion.
>
> Learning async control flow can be daunting for new developers. If we
> have people going off and building co-routine based modules, new node
> developers will now have to learn two new things.
>
> I'd be interested to hear your thoughts on my perspective.

Marak, I completely agree with you. That's why my proposal is so
minimal and integrates cleanly with async callbacks. From the outside
it's no different than using Step. The caller doesn't have to worry
that there might be some bad magic inside there it has to worry about.
The functions you call from within the co-routine are also normal async
functions that take a callbacks.

While it does add more complexity to the system, it's 100% optional
complexity and you don't have to worry about it if you don't want to.
You don't even have to worry about it if the libraries you use make use
of it. There is no concept of "fiber-aware" functions. Think of it as
a slightly more elegant version of step.

Bradley Meck

unread,
Nov 10, 2011, 12:43:38 AM11/10/11
to nod...@googlegroups.com
That quote was probably me. Scaling to multiple cores via threads is something that means the threads must be native OS threads and the OS must be recent enough to have the multi-core scheduling be enabled (which is true for any modern OS). Even if this is achieved conflict resolution is costly when concurrency increases beyond a small amount (lock thrashing or stm rewriting). Concurrent actions that share memory that may have conflicts will react very different from isolates that aggregate data. Even once all of these line up, we must ensure that all of our environments do not use a global interpreter lock (luckily we don't have this in v8) or similar lest we be left with the problem of one thing acting on a single thread and that providing sufficient bottleneck to ruin all the concurrency. Either way, take things with a grain of salt on both ends and just code for whatever makes you get the product done :).

Bradley Meck

unread,
Nov 10, 2011, 12:46:28 AM11/10/11
to nod...@googlegroups.com
Errata : when saying we don't have the interpreter lock in v8 I am talking about isolates and not the use of lockers.

Isaac Schlueter

unread,
Nov 10, 2011, 1:32:22 AM11/10/11
to nod...@googlegroups.com
It seems highly odd to me that you pass "resume" as the callback, then
call "wait()" to get the results. Also, it looks synchronous, but
it's not, and that confuses and scares me.

Despite an admirable effort, this is still magical and highly
complicated for anyone who doesn't grok it deeply.

It's too hard. I'm gonna stick with my hand-rolled flow control tricks.

On Wed, Nov 9, 2011 at 21:46, Bradley Meck <bradle...@gmail.com> wrote:
> Errata : when saying we don't have the interpreter lock in v8 I am talking
> about isolates and not the use of lockers.
>

Marcel Laverdet

unread,
Nov 10, 2011, 9:02:30 AM11/10/11
to nod...@googlegroups.com
I'll post my thoughts on this issue..

First a boilerplate rant on my node-fibers though. I never intended fibers to be used directly by clients, but instead to be used as a tool to create an ecosystem of cool new ways handle async. If the identifier "Fiber" or "yield" appear more than once or twice in your application you've probably done something terribly wrong. node-fibers is intended to be a very powerful (and dangerous!) tool used to create safer tools. You can see some of the projects enabled by fibers on the wiki: https://github.com/laverdet/node-fibers/wiki . Case in point you could implement your proposal to a T in like 10 lines using node-fibers.

I feel that an explicit "this call may block" sigil is usually important but that's not my call to make for your application. If it's important to you then enforce it with a library or coding standards. I was particularly impressed with the simplicity of node-green-light and how it handles the passing of the blocking sigil. Yeah you could cheat by calling `Fiber.yield()` directly but if that happens you have only yourself to blame. You could also just `delete Fiber.yield` after node-green-lights gets a reference to it if you feel like you can't control yourself.

This is a pillar of my defense of node-fibers but like no matter how many times I bring it up it seems to be ignored the next time a discussion of how coroutines are dangerous comes up. Arguing about how hazardous `Fiber.yield` is is a strawman.

Anyway I don't think we need any kind of coroutines in nodejs core, ever. All we need is a common idiom for asynchronous operations and we've already got that with `function(err,val)`. After that it's up to you to write an adapter to and from this common idiom. streamlinejs completely nails this with the single character adapter (underscore). All the other node-centric pure-JS solutions use this idiom as their common bridge. All external modules should (and do!) export async calls with this interface. **It doesn't matter if a module uses fibers or callbacks or Step internally as long as they're accepting a callback for blocking calls**. Then you just use whatever async library you want in your client code and you're done.

Specifically on the topic of your proposal I think you're missing one of the most compelling features of existing fibers & streamlinejs: fixing try\catch. function(err,val) isn't just a "sometimes" thing, it's the law. If an error is passed your `wait` should throw it. How many times can you write `if (err) { next(err); return; }` without wanting to quit your job?

@mikeal:
> Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily handled with named functions that I start to grit my tether whenever someone makes a comment like this.

Do you remember the rimraf challenge saga a few months ago where converting a very real application to use fibers resulted in a 40% decrease in lines of code? That wasn't a contrived example and it demonstrated a measurable reduction in code complexity. There's a small upfront cost to learn a new library but after that you reap the benefits forever.

Tim Caswell

unread,
Nov 10, 2011, 9:17:48 AM11/10/11
to nod...@googlegroups.com
Wow, thanks for the rant. It seems I haven't been as involved on this
list as I should be for the last year due to working on webOS.


On 11/09/2011 04:57 PM, _doq wrote:
> Rant warning!
>
> I came to this group based off a comment from polotek about:
> 1. There is a group, and the devs are super involved in it!
> 2. There were a few uneducated programmers attempting to change the
> whole of nodeJs via combating what they think is a problem with callbacks.
>
> At first I was like, hmmm don't seem to be that much static out
> here... boy was I wrong. I am glad you like lua, but the issue with
> callbacks in every case I have seen is user error, as in they really
> don't get it. Maybe there is a really big learning curve I am not
> seeing, but all I can see is a bunch of stubborn coders trying to
> convert others to there way of thinking rather getting on board with
> the coop para-dime, this reminds me of the transition from procedural
> programming to oop.
>

I do feel there is a large misunderstanding of my goals here.

First and foremost, let's set the record straight. I'm a strong
advocate of callback based programming and have been since the early
days of node. I even fought against blocking require because async
require was more elegant to me and I didn't see the point of using
blocking require just to make it easier for people coming from other
systems. I celebrated when promise.wait was removed and again when
"promises" were removed all-together. I've spent *years* trying to help
newcomers to node understand callbacks and tried to convince them to not
use hacks and instead just learn it "right". I understand the community
has grown a lot and I've been rather inactive for the last year while
working at HP, but I an *not* trying to convince anyone that co-routines
are better than callbacks. In fact I personally rarely use co-routines
even in Lua. I am definitely in the callback camp on this issue.

Here are some of my articles and presentations on the topic. I won't
bother you by searching through the mailing list and posting those as well.

- http://howtonode.org/control-flow
- http://howtonode.org/control-flow-part-ii
- http://howtonode.org/control-flow-part-iii
- http://howtonode.org/do-it-fast
- http://howtonode.org/step-of-conductor
- http://jsconf.eu/2010/speaker/techniques_and_tools_for_tamin.html

However, despite my efforts and the efforts of others, many newcomers to
node constantly have a hard time with the callbacks. I don't believe
that I'm just smart and everyone else is dumb. Maybe it's just my
background in functional programming, but callbacks are very easy for me
and thus it's hard to have empathy for people who struggle with it. But
the fact is that it's been a constant struggle for new people ever since
node was announced. One summer it seemed to be all this list was about
and all conference presentations on node talked about.

This "humble proposal" as I put it is a compromise, a proposal, to help
those coming to node. I have very strict rules about it's impact on the
rest of the system before I even consider promoting it. It must not
cause any more hazard than is already there. It must integrate cleanly
with normal callback based async functions. It must be exposed
externally as a normal async function. It must use internally normal
async functions.

I am sad that my post to the mailing list got sidetracked. I do not want
to discuss all the alternative ways to avoid callbacks that have already
been hashes and re-hashed on this list. I simply want smart and
unbiased thoughts on my proposal. While working on luvit I found this
entire other culture that understand co-routines and they consider the
people on the node list who defend callbacks (myself included) as
arrogant and refusing to be open minded about the issue. I discussed
the issue with them on their territory where they aren't constantly told
to shut-up and go away and seen as a bother. I learned that I
misunderstood the idea of co-routines and many of my arguments against
them were false.

> Stuff like this makes me not want to even look at your code, "Lua is
> not JavaScript. It's pretty darn close," yeah... it is just as close
> as C is to VB. I am really surprised though all your attempts you
> didn't just learn to use callback more elegantly rather then trying to
> re-core the language.
>

I'm sorry my post offended you. I know JavaScript far more than I know
Lua, but in my limited experience with Lua (read a couple books and
wrote a few thousand lines of code), it's very much like JavaScript.
I'm not talking syntax but semantics. Functions are almost identical
(first class values), closures work the same. Tables are similar to
objects, metatables can emulate prototypes and are a lot like proxies.
There is type coercion with truthy and falsy values. They both can run
on JIT capable vms. Primitives are passed by value, but anything
mutable is passed by reference. I could go on, but this is not a lua
forum. My point here is that using libuv with luajit is about as close
to a nodejs experience as you're going to get in *any* non-javascript
environment. It was important for my experiment to use a language that
already has co-routines in the community instead of hacking them onto
v8. I'm expert in node and the node architecture, not in co-coutines.
If I was going to evaluate co-routines, I needed the help of a community
that understood co-routines and has used them for years in real projects.

> Rant End!
> --

As I said in my nodeconf talk. It's very important this community stay
friendly as it grows. The leadership and core community members need to
help this out by not being arrogant, but instead understand the problems
newcomers face. It's hard to understand someone else via email if we
don't see eachother face to face first, but remember there are real
humans there with real feelings and concerns.

-Tim Caswell

Axel Kittenberger

unread,
Nov 10, 2011, 9:54:42 AM11/10/11
to nod...@googlegroups.com
I posted this once and do it again. I wonder when something that is a
practical consideration like callbacks becomes suddenly a doctrinal
thing to some people where even thinking otherwise makes them send you
verbally to be killed in Downtown New York, call you stubborn, and
what else gives. Anyway Tim is right, coroutines are not replacement
or alternative for callbacks, they are simply another flow-control
options for the user. So what gives the hatred?

Personally I'm torn between Fibers, Streamline and classical
approaches and do something else with every subproject. As said, there
is some limit to me in coding logics where classical callbacks start
to be come very tiresome - that is loops for me. And nested loops is
where it really gets me down and a solution like streamline really
eases the day a lot. Maybe node is really more suited for the simpler
I/O stuff, and if you code too much logics you should have better
chosen something with classical CGI/processes from start? Anyway if
one looks through the node module list, theres a hugh load of "control
flow" modules - are they all "just stubborn", or is there in fact an
itch they try to scratch?

And yes, Lua and Javascript are surprisingly quite close cousins but
without any common history (except scheme long down the road), call it
convergent evolution, as it happens occasionally in nature.

To go back on your original proposal, aside coding it with fibers as
interlocked pthreads that are built in the V8 and accepting their
costs I see noway to make node/javascript do efficient cooperative
routines without altering the V8, right? I was wrong before when I
said, I suppose through generators they will come anyway, since
generators are Javascript 1.7 (Mozilla) not ECMAScript and Google
targets with V8 only ECMAScript and whatever Webkit implements. Thus
if, we should best convince webkit to make something coroutine like,
then Google will duplicate that, then it will automatically come to
node :-). Personally I consider the Javascript 1.7 take on coroutines
not too great, calling it a generator and the way it is presented is
as one possible use of coroutines than the other way around.

Marcel Laverdet

unread,
Nov 10, 2011, 11:36:38 AM11/10/11
to nod...@googlegroups.com
> To go back on your original proposal, aside coding it with fibers as
> interlocked pthreads that are built in the V8 and accepting their
> costs I see noway to make node/javascript do efficient cooperative
> routines without altering the V8, right?

??

node-fibers right now has coroutines in v8 without messing with v8 internals. They are real coroutines too; ucontext is used in Linux and setjmp\longjmp are used in OS X. Originally it was built manually with ucontext but I've switched to libcoro because ucontext is flakey in OS X.

Axel Kittenberger

unread,
Nov 10, 2011, 11:47:25 AM11/10/11
to nod...@googlegroups.com
> node-fibers right now has coroutines in v8 without messing with v8
> internals. They are real coroutines too; ucontext is used in Linux and
> setjmp\longjmp are used in OS X. Originally it was built manually with
> ucontext but I've switched to libcoro because ucontext is flakey in OS X.

Didn't know that, thank you for clearing that up.

Bruno Jouhier

unread,
Nov 10, 2011, 12:33:39 PM11/10/11
to nodejs
> Anyway I don't think we need any kind of coroutines in nodejs core, ever.
> All we need is a common idiom for asynchronous operations and we've already
> got that with `function(err,val)`. After that it's up to you to write an
> adapter to and from this common idiom. streamlinejs completely nails this
> with the single character adapter (underscore). All the other node-centric
> pure-JS solutions use this idiom as their common bridge. All external
> modules should (and do!) export async calls with this interface. **It
> doesn't matter if a module uses fibers or callbacks or Step internally as
> long as they're accepting a callback for blocking calls**. Then you just
> use whatever async library you want in your client code and you're done.

I think that this should be the golden rule for async APIs: "Thou
shall use the callback(err, result) idiom". If we all do this, people
will be able to pick and choose whatever tool/library they like best
when consuming a library written by someone else. And people who
implement modules will also be able to use whatever tool/library they
like best to implement their stuff and expose it to others.

What would be problematic would be libraries that expose their APIs in
a different style and force their consumer to use a specific async
library on the side (for example mainpulate Fibers explicitly, or
using a specific "wait" call) to do anything with it. Then we would
see fragmentation.

Solutions like node-fibers or streamline, and most lightweight async
libraries comply with this "API standard". So rather than go into wars
about the solutions that we have chosen to implement our own stuff, we
should acknowledge the importance of this API convention.

>
> Specifically on the topic of your proposal I think you're missing one of
> the most compelling features of existing fibers & streamlinejs: fixing
> try\catch. function(err,val) isn't just a "sometimes" thing, it's the law.
> If an error is passed your `wait` should throw it. How many times can you
> write `if (err) { next(err); return; }` without wanting to quit your job?
>
> @mikeal:> Can we please identify the problem with callbacks. I'm not saying there
>
> isn't one, but most "examples" i see are so contrived and easily handled
> with named functions that I start to grit my tether whenever someone makes
> a comment like this.
>
> Do you remember the rimraf challenge saga a few months ago where converting
> a very real application to use fibers resulted in a 40% decrease in lines
> of code? That wasn't a contrived example and it demonstrated a measurable
> reduction in code complexity. There's a small upfront cost to learn a new
> library but after that you reap the benefits forever.

I also find that we are usually presented with examples that are
really simplistic in comparison with what we are dealing with in real
life. Rimraf was more interesting. Here are two other examples from
recent discussion threads:

A DB collection fetch followed by a loop on a cursor:

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
/**
* do something with every obj here
* suppose for this example it needs total count and offset (oi).
*/
});

A simple file paths manipulation rule:

function _extendPath(_, path) {
if (!_exists(_, path + ".js") && _exists(_, path) &&
fs.stat(path, _).isDirectory()) {
if (_exists(_, path + "/main.js"))
return path + "/main";
else if (_exists(_, path + "/index.js"))
return path + "/index";
}
return path;
}

The important thing is that these pieces of code do something *very
simple* and there is no reason to over-engineer them. Maybe some of us
are writing sophisticated algorithms all day long and think that
pieces of code like these are just boring stuff that we should not be
writing anyway. But this is life, many applications have to deal with
little boring pieces of logic like these ones and someone has to write
the code, and the rules will change, and someone will have to maintain
the code. I feel that tools like streamline or node-fibers really help
express these pieces of logic in a simple and understandable form. And
so far, I have not seen a convincing w


>
>
>
>
>
>
>
> On Thu, Nov 10, 2011 at 3:32 PM, Isaac Schlueter <i...@izs.me> wrote:
> > It seems highly odd to me that you pass "resume" as the callback, then
> > call "wait()" to get the results.  Also, it looks synchronous, but
> > it's not, and that confuses and scares me.
>
> > Despite an admirable effort, this is still magical and highly
> > complicated for anyone who doesn't grok it deeply.
>
> > It's too hard.  I'm gonna stick with my hand-rolled flow control tricks.
>

Bruno Jouhier

unread,
Nov 10, 2011, 12:35:47 PM11/10/11
to nodejs
Oops, hit the wrong key. Was almost finished anyway. Last incomplete
sentence was just:

I have not seen a convincing way to express this kind of logic with
callbacks.

Bruno

Axel Kittenberger

unread,
Nov 10, 2011, 12:40:25 PM11/10/11
to nod...@googlegroups.com
I spent a few thoughts how to improve on the basic idea, maybe that
stop&go interfaces are a tad too verbose to be attrictive. How about
an interface like this. If I don't miss something it should be quite
doable with fibers.

// Lets choose _ as blocking highlighter since thats well established.
coroutine(function(_) {
// anything that does not have '_' cannot yield.

// this one runs one asyncFunc. It expects the last paramter not
given there to be the callback,
// which returns (err, asw). asw is returned, err is thrown.
var asw = _(asyncFunc, param1, param2);

// this place is reached after asyncFunc calls its callback.

// and since it has been requested this is how you could do two
things in parallel.
var asw1, asw2;
_(asyncFunc1, param1, param2, function(asw) { asw1 = asw },
_, asyncFunc2, param2, param2, function(asw) { asw2 = asw });
// the magic difference is _ appearing as its own paramter so here
it uses itself as
// seperator of two calls to be run simultanously. Dunno maybe it
would be simpler instead
// of adding callbacks to return an array?

// this place is reached after asyncFunc1 and asyncFunc2 call its callback
});

// this one immediatly returns into the main stack (or when coroutine
first runs into _ or returns, I don't know now)

Mark Hahn

unread,
Nov 10, 2011, 12:47:52 PM11/10/11
to nod...@googlegroups.com
I think that this should be the golden rule for async APIs: "Thou
shall use the callback(err, result) idiom". 

That ain't gonna happen.  Libraries like jQuery are too established and won't change.  I've had to reject any sync solution that assumes this because I am too lazy to write 100's of wrappers for the incompatible libraries.

Tim Caswell

unread,
Nov 10, 2011, 1:12:29 PM11/10/11
to nod...@googlegroups.com
I think the node idiom for callbacks is great for node, but I don't
want to say all javascript everywhere should use it for async. When I
made Step, I made it take advantage of this node idiom and assume it's
there. When I released Do before that, I wrote about the importance
of all node modules adhering to this idiom so we can use whatever
async library we prefer but still share programs.

That's why I was careful in this co-routine proposal to make sure it
still used and provided node style callbacks. In fact it's even more
agnostic than that, but that's just a side effect of it being so
minimal. The _(fn, param, param2) idea works too, but it assume node
style callbacks and is a little more abstraction.

I guess my proposal is more of a set of rules where coroutines can be
used safely within node. I think we all agree on that now.

- It should be efficient, not use threads and locking in the implementation
- it can be either a node addon like node-fibers or implemented using
generators from es.next, as long as the other rules are sastified
- There should be a clear sigil to know when the coroutine gets
suspended and other stuff can happen.
- The effect needs to be not viral. If a module I depend on uses
coroutines, I shouldn't have to be aware or change my usage in any
way.


-Tim Caswell

Isaac Schlueter

unread,
Nov 10, 2011, 1:27:17 PM11/10/11
to nod...@googlegroups.com
On Thu, Nov 10, 2011 at 06:54, Axel Kittenberger <axk...@gmail.com> wrote:
> I posted this once and do it again. I wonder when something that is a
> practical consideration like callbacks becomes suddenly a doctrinal
> thing to some people where even thinking otherwise makes them send you
> verbally to be killed in Downtown New York,

I don't want anything killed in SoHo, verbally or otherwise. That was
just some jokey wordplay.

> call you stubborn, and
> what else gives. Anyway Tim is right, coroutines are not replacement
> or alternative for callbacks, they are simply another flow-control
> options for the user. So what gives the hatred?

I started writing a reply, and it turned into a blog post.

http://blog.izs.me/post/12604303054/experts-idiots-and-taste

Glenn Block

unread,
Nov 10, 2011, 1:30:43 PM11/10/11
to Tim Caswell, nod...@googlegroups.com
Reading this thread. I really wish we had async/await type syntax in
the box. It greatly simplifies writing async code and improves
readability.

Sent from my Windows Phone
From: Tim Caswell
Sent: 11/10/2011 10:12 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal

Angel Java Lopez

unread,
Nov 10, 2011, 1:44:45 PM11/10/11
to nod...@googlegroups.com
Hi people!

Glenn, sometime ago I found:
https://github.com/koush/node/wiki/%22async%22-support-in-node.js
by @koush

But I don't know the status of such proposal.

Angel "Java" Lopez
http://ajlopez.wordpress.com
http://twitter.com/ajlopez

Diogo Resende

unread,
Nov 10, 2011, 2:08:37 PM11/10/11
to nod...@googlegroups.com
On Thu, 10 Nov 2011 09:35:47 -0800 (PST), Bruno Jouhier wrote:
> I have not seen a convincing way to express this kind of logic with
> callbacks.
>
> Bruno

I just had an idea :P

async function bar(a, b) {
doSomething()
return(err, results) // <-- "special return"
}
async function baz(a, b) {
doSomethingElse()
return(err, results)
}

{
err1, results1 = bar(a, b)
err2, results2 = baz(a, b)

doSomethingAfterBarAndBaz()
}
doSomethingNotBlocked() // <-- because it's outside that scope


---
Diogo R.

Bruno Jouhier

unread,
Nov 10, 2011, 2:09:03 PM11/10/11
to nodejs
It's a bit easy to just throw words in the air. Let us be down to
earth and take real code samples that real people will write. Again,
here are two examples that have not been invented on purpose:

A DB collection fetch followed by a loop on a cursor:

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
/**
* do something with every obj here
* suppose for this example it needs total count and offset (oi).
*/

});

A simple file paths manipulation rule:

function _extendPath(_, path) {
if (!_exists(_, path + ".js") && _exists(_, path) &&
fs.stat(path, _).isDirectory()) {
if (_exists(_, path + "/main.js"))
return path + "/main";
else if (_exists(_, path + "/index.js"))
return path + "/index";
}
return path;

}

If callbacks are the answer, please, callback experts, roll up you
sleaves, show us what your code will look like and tell us how long it
took you to write it!

And it would be nice if we could do labs, put different developers at
different skill levels in front of these different pieces of code, ask
them to tell us what the code does, ask them to add an extra rule,
measure how long it takes them to make the change, measure how many
errors they will make, etc.

I'm not talking about questions of taste here. One thing we can very
easily measure is the size of the code. And seeing the code will
probably give us some good hints of what the outcome of the lab might
be.

So, please, roll up your sleeves!


Bruno

Mark Hahn

unread,
Nov 10, 2011, 2:11:54 PM11/10/11
to nod...@googlegroups.com
I think the node idiom for callbacks is great for node, but I don't
want to say all javascript everywhere should use it for async. 

When I mentioned the jQuery example, I was thinking of node, not the browser.  I use jQuery and I use couch.js in node.  couch.js uses the signature of db.dosomething(opts, errCallback, successCallback).  couch.js has 30 or 40 funcs in its api and I am not going to wrap them all.

Limiting js to one signature for function calls is not acceptable to me in any environment.

Diogo Resende

unread,
Nov 10, 2011, 2:12:54 PM11/10/11
to nod...@googlegroups.com
On Thu, 10 Nov 2011 11:09:03 -0800 (PST), Bruno Jouhier wrote:
> So, please, roll up your sleeves!

+1

---
Diogo R.

Mark Hahn

unread,
Nov 10, 2011, 2:13:34 PM11/10/11
to nod...@googlegroups.com
I'm not talking about questions of taste here.

I care very much about taste.  My coding experience (hell) is not determined by lines of code written, but by the pain involved.  My fingers may be old but they still type fast.

Glenn Block

unread,
Nov 10, 2011, 3:00:04 PM11/10/11
to Diogo Resende, nod...@googlegroups.com
Async / await is close. The only diff is more explicitly saying I want
these two calls to execute in sequence via an await keyword before the
call to the async function.

Sent from my Windows Phone

From: Diogo Resende
Sent: 11/10/2011 11:08 AM


To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal


---
Diogo R.

--

Liam

unread,
Nov 10, 2011, 3:12:29 PM11/10/11
to nodejs
On Nov 10, 6:17 am, Tim Caswell <t...@creationix.com> wrote:
> However, despite my efforts and the efforts of others, many newcomers to
> node constantly have a hard time with the callbacks. I don't believe
> that I'm just smart and everyone else is dumb. Maybe it's just my
> background in functional programming, but callbacks are very easy for me
> and thus it's hard to have empathy for people who struggle with it. But
> the fact is that it's been a constant struggle for new people ever since
> node was announced.

Amen. Node aspires to "supplant PHP." It ain't gonna get there unless
those developers can get basic stuff running (loops!) without becoming
confused.

To date, I have declined to try any of the sync-style alternatives,
despite having a ton of deeply-nested database transaction code. But
I'd love to try a blessed alternative.

> I am sad that my post to the mailing list got sidetracked. I do not want
> to discuss all the alternative ways to avoid callbacks that have already
> been hashes and re-hashed on this list. I simply want smart and
> unbiased thoughts on my proposal.

Destructuring assignment just to obtain err is a bit tedious, as is
the need to test err instead of using a try block. Can you make wait()
throw?

> While working on luvit I found this
> entire other culture that understand co-routines and they consider the
> people on the node list who defend callbacks (myself included) as
> arrogant and refusing to be open minded about the issue. I discussed
> the issue with them on their territory where they aren't constantly told
> to shut-up and go away and seen as a bother. I learned that I
> misunderstood the idea of co-routines and many of my arguments against
> them were false.

Some folks object to the callback complaints and alternatives a bit
too sharply. If an issue generates this much smoke, there's probably a
fire.

> As I said in my nodeconf talk. It's very important this community stay
> friendly as it grows. The leadership and core community members need to
> help this out by not being arrogant, but instead understand the problems
> newcomers face.

Hear, hear.

john.tiger

unread,
Nov 10, 2011, 3:46:21 PM11/10/11
to nod...@googlegroups.com
On 11/10/11 11:27, Isaac Schlueter wrote:
> I don't want anything killed in SoHo, verbally or otherwise. That was
> just some jokey wordplay.
@Isaac
I thought it was hilarious - though you might be guilty of listening to
too much Car Talk!

Since Tim has made a lot of good contributions to the Node community
(his original simple framework is what got us using Node), I think his
ideas are worth strong consideration - it would be nice if somehow flow
control (aka sequential operation) gets standardized or included in
core. In many applications it is a pretty necessary component.


Mikeal Rogers

unread,
Nov 10, 2011, 3:55:14 PM11/10/11
to nod...@googlegroups.com
"I feel like I'm taking crazy pills." -- Mugatu

There are two entirely different problems being talked about and everyone who wants coroutines love to conflate them because it makes for a better example.

The problems are:
1) You have a question you want to ask and the answer to that question is necessary for your application logic. This requires a callback.
2) You have an operation, or set of operations, you want to do to move or mutate data between file descriptors. This requires a stream.

Everyone needs to stop pretending that coroutines are better than streams, it's insane. None of these examples are simpler than.

req.pipe(filed('/path/to/file')).pipe(res)

or

req.pipe(request('http://www.google.com')).pipe(res)

While streams are still changing, and improving, what we can already do with them is quite nice. It's also considerably simpler than any example anyone, ever, has posted with coroutines. Seriously, I feel like I'm taking crazy pills, people will post their contrived example which is a few lines shorter than another contrived example not using streams and claim that it's shorter/simpler, it's not, stop it, seriously.

For the other problem. Asking a question required by application logic, we currently use callbacks. A large amount of indentation occurs when, and only when, the final solution in your application logic requires data from the top of the closure before you asked these questions. That is the problem. It's solvable, I'm not convinced coroutines make it simpler, at least in JavaScript.

The largest problem I see is that the data at the top of the closure is usually a set of Stream objects, so doing that IO is going to cause you to lose data in the streams. This is solvable, we add buffering semantics to .pause().

But I digress. This thread is obsessed with the wrong problem. It's something that is less than obvious when you first use node, so it gets a lot of attention, but it's much less of a problem now than it used to be. The more work we do in modules the simpler this gets. I'm not sold on generic flow control libraries either, the best solutions I've seen are domain specific, but good things are happening.

There are a lot of people saying "this is too complicated, we need to simplify it" and then proposing solutions that have an incredibly high tolerance for new complexity and build solutions squarely outside of where we are actually making progress on solving the problem already.

This is not productive.

-Mikeal

Liam

unread,
Nov 10, 2011, 4:19:10 PM11/10/11
to nodejs
On Nov 10, 12:55 pm, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> For the other problem. Asking a question required by application logic, we currently use callbacks. A large amount of indentation occurs when, and only when, the final solution in your application logic requires data from the top of the closure before you asked these questions.

Indentation isn't the annoying part. It's these everywhere in lieu of
a single try/catch
if (err) throw err;

and these
if (x)
async(x, nextThing)
else
nextThing(null)
function nextThing(err, result) {
...
}

As for streams, maybe we need more teaching about how to both apply
and implement stream classes. It seems like you're suggesting most
loop constructs can be addressed with streams?

Mikeal Rogers

unread,
Nov 10, 2011, 4:47:54 PM11/10/11
to nod...@googlegroups.com
Again, contrived examples disuade us from the real problem.

You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

While it might be annoying to have all these if (err) blocks a much bigger problem occurs when calling a function that throws. Most of the time, we call functions we did not write ourselves, which may have bugs which means they may throw and we can't even wrap them reliably in a try/catch because the error may be in a future callback added to the event system by that code.

We have a proposed solution for this, domains. Domains would allow one single handler around any amount of arbitrary code and inside any of the callbacks it might add to the event system.

Domains are a much simpler solution than coroutines and handle far more cases, they are so simple in fact that advocates of more complex proposals go to great lengths to ignore the domains proposal, much like they ignore streams.

I'm starting to identify a divide here.

node.js prefers solutions to problems that lend themselves well to abstractions. streams are great example, they lend themselves to amazing higher level abstractions that are nearly transparent to the user. Domains are even better, they will mostly be used by frameworks and other libraries and be invisible to application code.

Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

This is not productive.

-Mikeal

Axel Kittenberger

unread,
Nov 10, 2011, 4:59:41 PM11/10/11
to nod...@googlegroups.com
> Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

Have you not seen Tims proposal then? I don't see how it pushes it's
API to the highest level. The coroutines stuff stays local to one
routine and it still communicates downward and upward with callbacks.

> This is not productive.

This is not productive.

Mikeal Rogers

unread,
Nov 10, 2011, 5:05:17 PM11/10/11
to nod...@googlegroups.com

Yes, I've read Tim's proposal.

It mostly competes with streams. Have you used streams?

>
>> This is not productive.
>
> This is not productive.
>

Axel Kittenberger

unread,
Nov 10, 2011, 5:12:52 PM11/10/11
to nod...@googlegroups.com
> It mostly competes with streams. Have you used streams?

Yes for streaming data. Duh. If it can be used for more, please answer
Liams question, are suggesting streams as solution to loops?

If so, can you please recode following example here done with
streamline with streams instead? (it is a snipped from a real
application)

Mikeal Rogers

unread,
Nov 10, 2011, 5:31:30 PM11/10/11
to nod...@googlegroups.com
var c = dbclient.collection('example').find(_)
c.on('obj', function (obj) { 
 c.totalcount
 c.offset
})
c.on('end', function () {})

Or if you only need the full result.

dbclient.collection('example').find(_, function (e, result) {
  result.rows
  result.rows[0].offset
  result.rows[0].value
  result.totalcount
})

Like i stated originally, the scope of this problem is complex logic that requires state from the closure, if you have a question you want to ask that needs little closure state and can itself expose it's newly formed state the abstraction is simple, it is after-all a state machine.

If your complaint is the complexity of building this abstraction then I'm afraid it's falling on deaf ears. The amount one must know about how database cursors makes understanding an asynchronous counter seem trivial by comparison and both must be understood to write any database abstraction effectively.

I will again restate that I don't find most generic solutions adequate and domain specific abstractions have been much more impressive so far.

-Mikeal

Bruno Jouhier

unread,
Nov 10, 2011, 5:34:19 PM11/10/11
to nodejs
@Isaac,

Just read your blog post again. You make a lot of assertions and
explain why you would use or not use solution X, Y or Z. Fine, but
this is a bit irrelevant AFAIC.

I did not write streamline for you, and I don't really care whether
you like it or not.

A bit of context: I chose node.js for a project, and got laughed at
initially (server-side Javascript, you must be joking!!). I had a
small team of very skilled engineers. We started to write code the
'callback way'. In a few months we had prototyped a number of things
and written about 10,000 lines of code, all in callback style.

We had been quite successful overall and the team was excited about
the technology but I had to make a hard call: are we on the right
platform? Will we be productive enough? Are we writing code that is
easy to maintain or not? Will we be able to grow the team and maintain
discipline? How much is this going to cost us (in training, in code
reviews, etc.)? Is this viable or are we going against a wall?

My assessment was that, despite the excitement, the early successes
and the fact that node was getting more and more attention, we were
writing code that was fragile and difficult to maintain. We could do
wonders with JS and some things that were really hard/painful before
(dynamic stuff, I/O) had become incredibly elegant and pleasant; but
on the flip side, things that were really easy before (like writing a
simple rule like "if path + '.js' does not exist and path is a
directory and it contains a file called 'index.js' do XYZ else ...")
had become a lot more difficult, painful to write, error prone, less
maintainable, etc.

In the kind of software we are writing, there is about 10% of clever
technical stuff and 90% of mundane not-very-technical logic. Even if
we were in better shape on the clever stuff, we were going against a
wall if we could not maintain our previous productivity and quality
standards on what makes up 90% of our code.

Fortunately, I had used the experimentation time to experiment with
flow control libraries and I had found a way to greatly simplify the
problem with a preprocessor.

So, streamline is not something that I did for myself, or to please
you. It is something that I did because we would have had to drop
node.js if we hadn't found a way to get back on a productive,
disciplined, maintainable track. And, if fibers or generators prove to
solve the problem better than a preprocessor we are going to
transition to them (and streamline is probably going to make this very
painless).

But our assessment was just that there were too many problems with
callback-style async programming, too much risk, unless we could find
a solution that would dramatically change the game. Fortunately we
found one, we are moving along at good speed on node, the team is very
enthusiastic and node is making the news everyday so I'm not getting
laughed at any more (at least not for choosing node).

All this to say that the two code samples that I posted before are not
irrelevant. They are very relevant to what a lot of people are doing.
So I strongly encourage you to write them in callback style and
compare the results with the code I gave: code size, number of
functions, complexity of the execution flow, distance between the
expression of the rules in plain English and their expression in the
code that you wrote, etc.

So, the streamline/fibers magic is probably not for you. But be open
and accept the fact that others may have different requirements than
you and that for them, a bit of magic can turn node.js from a geeky
platform that doesn't fit their bill into a very pleasant platform to
work with at all levels.

My 2 cents, FWIW.

Bruno

Liam

unread,
Nov 10, 2011, 5:35:29 PM11/10/11
to nodejs


On Nov 10, 1:47 pm, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> Again, contrived examples disuade us from the real problem.
>
> You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

I always throw for an unexpected error. You need an app-restarter
anyway, since something could segfault, etc.

And returning 500 is a UI style obscenity. :-)

Bruno Jouhier

unread,
Nov 10, 2011, 5:49:18 PM11/10/11
to nodejs
@Mikeal,

No, this is not at all a valid translation. The _ is a callback
placeholder so:

* collection is async
* find is async
* count is async
* nextObject is async

And, sorry but this is life, this is working againts a SQL DB (those
still exist) and there is no choice but iterate with a cursor if you
have a large table and you have to scan lots of records (sometimes you
just can't avoid it), simply because that's the way the driver works.

Q: how do you write this with callbacks? It is too easy to just change
the problem and assume that you are going to emit events for all the
selected records, and that the setup calls are just sync.

Note: writing it with streamline takes a few minutes at most and isn't
an headache by any means. The example is actual code that works, not
some kind of pseudo code (same with the other sample I gave).

Bruno

Axel Kittenberger

unread,
Nov 10, 2011, 5:50:27 PM11/10/11
to nod...@googlegroups.com
Your example is not removing the streamling callback _ magic? makes
quite some deliberate simplications, I suppose you assume a better
interface from the drivers than there is :-). collection() is a
callback, find() is a callback, count() is a callback (since it only
has to be calculated if you say you need it). etc. Yes its a quite a
bulk with classical async code. Anyway if I see it correctly, with
streams you will be at lastest be very stumped when needed nested
loops.

var c1 = dbclient.collection('example', _).find(_);for(var o1 =
c1.nextObject(_); o1; o1 = c1.nextObject(_) {
var c2 = dbclient.collection(o1.refer, _).find(_);
for(var o2 = c2.nextObject(_); o2; o2 = c2.nextObject(_) {
c2.set('value', 1, _);
}
}
console.log('finished');
Yes you could pause the c1 stream, start a c2 stream, if it finishes
resume the c1 stream for another object, pause it, starts. I don't buy
this being simpler.

Axel Kittenberger

unread,
Nov 10, 2011, 5:52:15 PM11/10/11
to nod...@googlegroups.com
> And, sorry but this is life, this is working againts a SQL DB (those
> still exist)

Dunno how similar the SQL interface is, but that one was originally
against a mongoDB; I try to be cool and modern :-)

Bruno Jouhier

unread,
Nov 10, 2011, 5:58:31 PM11/10/11
to nodejs
Take a look at https://github.com/bjouhier/node-db and
https://github.com/bjouhier/node-db-oracle

We forked it not long ago to add cursors so the cursor stuff is not
documented yet. I'm into it these days so I assumed your example was
SQL. But we're also working with mongoDB :-) Cool stuff.

Bruno

Mikeal Rogers

unread,
Nov 10, 2011, 6:02:41 PM11/10/11
to nod...@googlegroups.com
How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.

Axel Kittenberger

unread,
Nov 10, 2011, 6:12:44 PM11/10/11
to nod...@googlegroups.com
On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
> How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.

What the code above does is for every row of a potentially large
collection (or call it table in SQL), it reads a string which contains
the name for other potentially large collections (tables). For these
tables it sets the field 'value' for every row to 1.

Yes its a made up example, didn't had to do a nested loop in practice so far.

Marcel Laverdet

unread,
Nov 10, 2011, 9:25:44 PM11/10/11
to nod...@googlegroups.com
@tim
> - It should be efficient, not use threads and locking in the implementation
> - it can be either a node addon like node-fibers or implemented using
> generators from es.next, as long as the other rules are sastified
> - There should be a clear sigil to know when the coroutine gets
> suspended and other stuff can happen.
>  - The effect needs to be not viral.  If a module I depend on uses
> coroutines, I shouldn't have to be aware or change my usage in any
> way.

node-fibers/future, streamlinejs, and node-green-light all already meet these expectations.


@mark
> When I mentioned the jQuery example, I was thinking of node, not the browser.  I use jQuery and I use couch.js in node.  couch.js uses the signature of db.dosomething(opts, errCallback, successCallback).  couch.js has 30 or 40 funcs in its api and I am not going to wrap them all.

I disagree I feel that these are the exceptions. Just because there exists a few libraries today that break the rules or were around before the rules existed doesn't mean we should throw out the rules altogether. You can write an adapter once to fix all these functions at once if it's important to you.

function fixjQuery(fn) {
  return function() {
    var args = [].slice.call(arguments);
    var cb = args[args.length - 1];
    args[args.length - 1] = function(val) {
      cb(null, val);
    }:
    args[args.length] = cb;
    return fn.apply(this, args);
  }
}


@mikeal
> Everyone needs to stop pretending that coroutines are better than streams, it's insane.

I wasn't pretending that. I've said before that fibers aren't generally helpful with streams. I'm kind of on the fence with db results as to whether those are a stream or callback because they're kind of both for me. Personally my SQL library has a dual-mode steam and single-callback library. I just use whichever is more appropriate for the situation at hand.


> people will post their contrived example which is a few lines shorter than another contrived example not using streams and claim that it's shorter/simpler, it's not, stop it, seriously.

Dude you just posted two contrived examples. And to be honest I've needed `pipe` like twice in my life so far. When I use it, it was amazing but most of my work is not redirecting streams of google.com to response objects.


>You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

Why don't you ever throw? What's wrong with having a try\catch at the top of your incoming request and catching it there? If an uncaught exception is thrown you catch it in the handler and send down a 500 error page. Sometimes you can catch an error before it makes it that far and fail gracefully. Say, an advertisement fails to render you can catch that exception and just blank out that div instead of killing the whole page.

Throwing is a beautiful thing because it means you can't ignore errors. Errors pop up in unexpected places, and having to constantly check for errors and forward them along is silly. And you will absolutely miss errors.


> Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

Like.. did you even look at the rimraf thing or what? I didn't see you address it in any of your emails. It wasn't a contrived example and resulted in a measurable 40% decrease in code length. It was a reasonably complex program too, I think it was about 80 or so lines of real code.

Mikeal Rogers

unread,
Nov 10, 2011, 9:30:10 PM11/10/11
to nod...@googlegroups.com
serving a file an proxying an HTTP request are hardly contrived.



>You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

Why don't you ever throw? What's wrong with having a try\catch at the top of your incoming request and catching it there? If an uncaught exception is thrown you catch it in the handler and send down a 500 error page. Sometimes you can catch an error before it makes it that far and fail gracefully. Say, an advertisement fails to render you can catch that exception and just blank out that div instead of killing the whole page.

Throwing is a beautiful thing because it means you can't ignore errors. Errors pop up in unexpected places, and having to constantly check for errors and forward them along is silly. And you will absolutely miss errors.

you don't throw in node.js because there isn't way to catch it nicely in callbacks. domains may change this but I'm talking about node.js as it is right now.

Mikeal Rogers

unread,
Nov 10, 2011, 9:34:34 PM11/10/11
to nod...@googlegroups.com

On Nov 10, 2011, at November 10, 20113:12 PM, Axel Kittenberger wrote:

> On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
>> How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.
>
> What the code above does is for every row of a potentially large
> collection (or call it table in SQL), it reads a string which contains
> the name for other potentially large collections (tables). For these
> tables it sets the field 'value' for every row to 1.

So, you want to do an operation to every row in a table? Is there any conditional logic for setting or not setting that row and if so does it require information that might be in the closure?

-Mikeal

Marcel Laverdet

unread,
Nov 10, 2011, 11:05:01 PM11/10/11
to nod...@googlegroups.com
you don't throw in node.js because there isn't way to catch it nicely in callbacks. domains may change this but I'm talking about node.js as it is right now.

Yeah but if you're using fibers/future or streamlinejs throwing an exception will just call your callback with an err param. All I was saying is that these solutions are compelling because they fix try/catch. I can throw an exception from deep in a library function behind several async calls and catch it in my HTTP listener. I'm sure you can do this to an extent with pure-JS solutions as well, but all those solutions are essentially rebuilding the try/catch paradigm from scratch via callbacks. With fibers/futures and streamlinejs you get native throw/try/catch/finally support with no extra callbacks or chaining.

Mark Hahn

unread,
Nov 10, 2011, 11:14:16 PM11/10/11
to nod...@googlegroups.com
Domains are a much simpler solution than coroutines and handle far more cases, 

I have followed this mailing list for a year and have never heard of domains.  Where can I read about them?

Mikeal Rogers

unread,
Nov 10, 2011, 11:16:23 PM11/10/11
to nod...@googlegroups.com
I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.

Mikeal Rogers

unread,
Nov 10, 2011, 11:18:01 PM11/10/11
to nod...@googlegroups.com
They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list.

That aren't in core yet.

-Mikeal

On Nov 10, 2011, at November 10, 20118:14 PM, Mark Hahn wrote:

Domains are a much simpler solution than coroutines and handle far more cases, 

I have followed this mailing list for a year and have never heard of domains.  Where can I read about them?

Mark Hahn

unread,
Nov 10, 2011, 11:32:09 PM11/10/11
to nod...@googlegroups.com
They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list. 

I hope I'm not a pest, but I can't find anything on this and I'm really curious.  I searched the nodejs-dev list for "domain" and got nothing.  Would it be too much to ask for you to describe domains in 100 words or less?

Marcel Laverdet

unread,
Nov 10, 2011, 11:33:13 PM11/10/11
to nod...@googlegroups.com
I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.

Ok then don't use it. Just don't come into our thread and shit on everything we're doing. I'm saying these features are compelling to me and a non-insignificant group of node users. We don't need or want these features to be in node core. Actually Tim might be saying that, but it's not my position. If you want to solve the problem your own way then I applaud you because diversity is really healthy here. I just don't understand why you feel the need to bust into every thread about fibers and start yelling at everyone. At least Isaac is being constructive and open-minded, all you're doing is saying "fuck this and everyone involved!"

Mikeal Rogers

unread,
Nov 10, 2011, 11:42:52 PM11/10/11
to nod...@googlegroups.com
That's concerning. We should get something written up, or maybe just get Ryan to merge them in :)

The basic idea is this (although this is *not* the exact API):

var d = process.createDomain()
// arbitrary code
d.stop()
d.on('error', function (err) { })
d.on('end', function () {})

Any code that runs between creation and the stop call will be in the domain. Any callbacks added to the event system will also be in the domain. So if:

function a (cb) {
  setTimeout(cb, 100)
}

function b (cb) {
  setTimeout(function () {
    throw new Error('asdf')
  }, 100)
}

var d = process.createDomain()
a(function () {
 setTimeout(function () {
    b(function () {})
  }, 100)
})
d.stop()

So In this code, event tho all these functions are being called in the future the event loop they are all attached to this domain and the throw in b will still be trapped by the domain. Once there is nothing left in the event system attached to the domain it fires the end event.

Hope that explains it. There is still some talk about parent/child relationships of domains but that isn't ironed out yet.

-Mikeal

On Nov 10, 2011, at November 10, 20118:32 PM, Mark Hahn wrote:

They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list. 

I hope I'm not a pest, but I can't find anything on this and I'm really curious.  I searched the nodejs-dev list for "domain" and got nothing.  Would it be too much to ask for you to describe domains in 100 words or less?

Mikeal Rogers

unread,
Nov 10, 2011, 11:55:34 PM11/10/11
to nod...@googlegroups.com
On Nov 10, 2011, at November 10, 20118:33 PM, Marcel Laverdet wrote:

I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.

Ok then don't use it. Just don't come into our thread and shit on everything we're doing. I'm saying these features are compelling to me and a non-insignificant group of node users. We don't need or want these features to be in node core. Actually Tim might be saying that, but it's not my position. If you want to solve the problem your own way then I applaud you because diversity is really healthy here. I just don't understand why you feel the need to bust into every thread about fibers and start yelling at everyone. At least Isaac is being constructive and open-minded, all you're doing is saying "fuck this and everyone involved!"

You seem to take offense at the idea that we might want to keep the thread productive.

This thread has been examining and identify problems and use cases and evaluating solutions. Solutions that might be usable by most node.js developers.

Tim's proposal is an attempt to fix a problem he perceives and he would like it to be in core.

You interrupted the thread to talk about fibers, as you often do, even though it is out of the scope of this conversation.

Nobody went out of their way to shit on fibers, or shit on this thread, you're the one that brought it up and I stated, factually, that it is not a solution to this problem that can be used by most node.js developers.

I said nothing negative about fibers, I only said that fibers as a solution is not usable by the majority if node.js developers, an indisputable fact.

You are not helping your community (fibers) with this kind of behavior.

-Mikeal

Mark Hahn

unread,
Nov 10, 2011, 11:55:38 PM11/10/11
to nod...@googlegroups.com
Hope that explains it. 

Thanks.  That is very interesting.  I will have to think about what it would mean for my code.

Mikeal Rogers

unread,
Nov 10, 2011, 11:59:46 PM11/10/11
to nod...@googlegroups.com

On Nov 10, 2011, at November 10, 20118:55 PM, Mark Hahn wrote:

Hope that explains it. 

Thanks.  That is very interesting.  I will have to think about what it would mean for my code.

It would be a blessing for frameworks like express. express could wrap the body of it's http server handler function in a domain and associate any error created within that with the right request/response and do a nice 500 error.

Isaac Schlueter

unread,
Nov 11, 2011, 12:11:20 AM11/11/11
to nod...@googlegroups.com
On Thu, Nov 10, 2011 at 20:05, Marcel Laverdet <mar...@laverdet.com> wrote:
> With
> fibers/futures and streamlinejs you get native throw/try/catch/finally
> support with no extra callbacks or chaining.

I take issue with the word "native" in that sentence.

Isaac Schlueter

unread,
Nov 11, 2011, 12:31:39 AM11/11/11
to nod...@googlegroups.com
On Thu, Nov 10, 2011 at 14:34, Bruno Jouhier <bjou...@gmail.com> wrote:
> I did not write streamline for you, and I don't really care whether
> you like it or not.

Please read the last 3 paragraphs again, wherein I thoroughly explain
that I don't care either. (So, why can't we seem to stop talking
about it...?)

> So, the streamline/fibers magic is probably not for you.

The pope is probably not a martian.

Liam

unread,
Nov 11, 2011, 1:19:12 AM11/11/11
to nodejs
These threads always go down this bowl-shaped hole because there are
two camps:
A believes async logic forces awkward code style on users
B believes broadly-applicable solutions to these issues are worse
than the problems

I suspect the only way camp A is going to sway the core team is by
congregating on a particular solution and producing a body of work
(including numerous real-world use cases) establishing its value, and
not just for newbies.

The core team is working on solutions to some of the stated problems,
e.g. domains (that's a confusing name, BTW). So you can't win this
battle on the basis of one or two language features which async
breaks.

@Mikeal, when you perceive people barking up the wrong tree, just show
them the right tree, everyone will learn more and faster. And maybe be
open-minded about trees you haven't stared up into yet? @Others, don't
respond to perceived slights with counter-punches. Ask for
clarification, or ignore them.

Async-everywhere is a new concept, and it's going to take a lot of
discussion and trials to discover the best practices for it. We need
to stop getting angry and let that process play out. These fights make
the Node community look sophomoric.

Marcel Laverdet

unread,
Nov 11, 2011, 2:46:33 AM11/11/11
to nod...@googlegroups.com
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
> You interrupted the thread to talk about fibers, as you often do, even though it is out of the scope of this conversation.

This thread literally has "fibers" in the subject line. Talking about fibers in a thread about fibers is interrupting? I don't think it's that farfetched that the author of the only coroutine library for nodejs would want to chime in. Tim's proposal is not that far of a departure from what exists already in node-fibers. I don't see why we need to throw away all the lessons learned from node-fibers for what amounts to moving a few parameters around.

> I said nothing negative about fibers, I only said that fibers as a solution is not usable by the majority if node.js developers, an indisputable fact.

`npm install fibers`. It works on Linux, OS X, and Solaris. Windows support is feasible once node gets dynamic module loading working. The only thing stopping anyone is personal convictions.

> I take issue with the word "native" in that sentence.

Sorry I mean like.. not reconstructed in userland? try\catch\finally\throw are features of the language. A lot of async libraries essentially simulate that functionality, but you have to register the right callback, or call the right function. With streamline and futures you can just use the regular `throw` statement in JS and catch it somewhere else.

All I'm trying to say in this thread is the situation we have now, strictly in terms of code, is reasonable. I don't think we need node core to adopt any single flow control library. The common (err,val) idiom is enough. If everybody adheres to this standard we can all use whatever flow control library we want. It shouldn't matter if someone is using Step, Q, fibers/futures, or streamlinejs.

Marcel Laverdet

unread,
Nov 11, 2011, 2:47:51 AM11/11/11
to nod...@googlegroups.com
This thread literally has "fibers" in the subject line

Err sorry I meant "co-routines" :-p

Brandon Benvie

unread,
Nov 11, 2011, 3:15:14 AM11/11/11
to nod...@googlegroups.com
>>  e.g. domains (that's a confusing name, BTW).

Sidetrack, for the sake of my sanity and many others do not use that name if it ever gets implemented in core. That's nearly the most completely unsearchable keyword I can think of, even filtering it to Node or JS searches.


>> Windows support is feasible once node gets dynamic module loading working

Works fine since 0.5.10 (or .11). Only difference is compilation needs to be done using MSVC against node.lib which is produced alone with node.exe when building. I found a number of Windows optimized implementations for what would be needed to implement the C++ end of Fibers when I looked into it.

Ultimately I decided that it would be better to continue working on various patterns built on top of Harmony WeakMaps and Proxies and some other stuff to allow eventual/soon async promise type execution to be usable by implementation agnostic synchronous consumers of the code. I feel that's probably the best likely long term route for solving my specific goals.

Mikeal Rogers

unread,
Nov 11, 2011, 3:35:53 AM11/11/11
to nod...@googlegroups.com
On Nov 11, 2011, at November 11, 201112:15 AM, Brandon Benvie wrote:

>>  e.g. domains (that's a confusing name, BTW).

Sidetrack, for the sake of my sanity and many others do not use that name if it ever gets implemented in core. That's nearly the most completely unsearchable keyword I can think of, even filtering it to Node or JS searches.

unsearchable names are what all the cool kids are doing nowadays :)

unrelated note, PhoneGap project is now named Apache Callback. for serious.



>> Windows support is feasible once node gets dynamic module loading working

Works fine since 0.5.10 (or .11). Only difference is compilation needs to be done using MSVC against node.lib which is produced alone with node.exe when building. I found a number of Windows optimized implementations for what would be needed to implement the C++ end of Fibers when I looked into it.

Ultimately I decided that it would be better to continue working on various patterns built on top of Harmony WeakMaps and Proxies and some other stuff to allow eventual/soon async promise type execution to be usable by implementation agnostic synchronous consumers of the code. I feel that's probably the best likely long term route for solving my specific goals.

Mikeal Rogers

unread,
Nov 11, 2011, 3:39:00 AM11/11/11
to nod...@googlegroups.com

On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:

> These threads always go down this bowl-shaped hole because there are
> two camps:
> A believes async logic forces awkward code style on users
> B believes broadly-applicable solutions to these issues are worse
> than the problems

I actually believe A and B are both true and am in search of C

C: patterns in async that simplify and/or abstract the awkwardness of async

we've made huge leaps in C over the last few years but we're not done yet. streams is one of these patterns, domains is another.

Jann Horn

unread,
Nov 11, 2011, 4:28:20 AM11/11/11
to nod...@googlegroups.com

Would that also allow having stuff like the current http request in a global that's domain-bound?

> >  They were first proposed by R...

Posting guidelines: https://github.com/joyent/node/wiki/Maili...

Bruno Jouhier

unread,
Nov 11, 2011, 5:04:58 AM11/11/11
to nodejs
Lots of new posts during the night here. A few notes.

1) Domains look cool. When they pop up, I'll change the try/catch/
finally rewrite rules in streamline to use them instead of setting up
all the callback scaffolding that gets generated today. That will
reduce the bloat in the generated code and it will also make it more
efficient. Nice thing is that I should be able to make the switch with
a simple change in the preprocessor: no change in source code!
Generators look cool too, and they will also allow me to greatly
simplify streamline's rewrite rules (again, without having to change
our source code). Preprocessors may look clumsy but they have some
advantages, especially with moving targets.

2) Lots of talking but I still haven't seen any convincing callback
implementation of the two little code samples that I posted yesterday.
If writing code in callback style is not an issue (which is the claim
being made, isn't it?) why doesn't someone produce implementations of
these less-than-10-lines snippets? That's only a few lines so it
should not take more than a few minutes. And don't get it wrong: I did
my share: these snippets are *not* pseudo-code, they are *real code*
(I did not say real JS!) because there is a preprocessor that turns
them into pure callback JS code.

Bruno

On Nov 11, 6:11 am, Isaac Schlueter <i...@izs.me> wrote:
> On Thu, Nov 10, 2011 at 20:05, Marcel Laverdet <mar...@laverdet.com> wrote:
> > With
> > fibers/futures and streamlinejs you get native throw/try/catch/finally
> > support with no extra callbacks or chaining.
>
> I take issue with the word "native" in that sentence.
>
>
>
>
>
>
>
> > On Fri, Nov 11, 2011 at 11:34 AM, Mikeal Rogers <mikeal.rog...@gmail.com>

Bruno Jouhier

unread,
Nov 11, 2011, 7:12:56 AM11/11/11
to nodejs
Glenn,

Async/await syntax is exactly what streamline.js gives you. The syntax
is just a bit different:
* async function foo(a, b) --> function foo(a, b, _)
* await foo(v1, v2) --> foo(v1, v2, _)

I did not introduce new keywords and I used a "magic parameter"
instead for several reasons: compatibilty with JS editors,
compatibility with CoffeeScript, compatibility with existing callback
APIs (in which the callback is not always in the same position), ease
of implementation (no need to hack existing parsers), etc.
But the concept is fundamentally aligned.

And streamline adds a little very useful feature to the game: futures.
If you call foo as foo(v1, v2) instead of foo(v1, v2, _) you obtain a
future F on which you can wait later with F(_).

Bruno

On Nov 10, 7:30 pm, Glenn Block <glenn.bl...@gmail.com> wrote:
> Reading this thread. I really wish we had async/await type syntax in
> the box. It greatly simplifies writing async code and improves
> readability.
>
> Sent from my Windows Phone
> From: Tim Caswell
> Sent: 11/10/2011 10:12 AM
> To: nod...@googlegroups.com
> Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
> I think the node idiom for callbacks is great for node, but I don't
> want to say all javascript everywhere should use it for async.  When I
> made Step, I made it take advantage of this node idiom and assume it's
> there.  When I released Do before that, I wrote about the importance
> of all node modules adhering to this idiom so we can use whatever
> async library we prefer but still share programs.
>
> That's why I was careful in this co-routine proposal to make sure it
> still used and provided node style callbacks.  In fact it's even more
> agnostic than that, but that's just a side effect of it being so
> minimal.  The _(fn, param, param2) idea works too, but it assume node
> style callbacks and is a little more abstraction.
>
> I guess my proposal is more of a set of rules where coroutines can be
> used safely within node.  I think we all agree on that now.
>
>  - It should be efficient, not use threads and locking in the implementation
>  - it can be either a node addon like node-fibers or implemented using
> generators from es.next, as long as the other rules are sastified
>  - There should be a clear sigil to know when the coroutine gets
> suspended and other stuff can happen.
>  - The effect needs to be not viral.  If a module I depend on uses
> coroutines, I shouldn't have to be aware or change my usage in any
> way.
>
> -Tim Caswell
>
>
>
>
>
>
>
>
>
> On Thu, Nov 10, 2011 at 11:47 AM, Mark Hahn <m...@hahnca.com> wrote:
> >>  I think that this should be the golden rule for async APIs: "Thou
> > shall use the callback(err, result) idiom".
> > That ain't gonna happen.  Libraries like jQuery are too established and
> > won't change.  I've had to reject any sync solution that assumes this
> > because I am too lazy to write 100's of wrappers for the incompatible
> > libraries.

vincentcr

unread,
Nov 11, 2011, 9:48:18 AM11/11/11
to nod...@googlegroups.com
>> So, why can't we seem to stop talking about it...?)

Because people take issue with your assertion that 'callbacks are not that hard', which you just throw in the air without any justification, intermixed with calling people idiots.

Yes, in the trivial case, callbacks are indeed simple, and with anonymous functions are eminently readable, even expressive. Certainly, it's a lot simpler than dealing with threads.

However, I find it hard to believe that you haven't found that it takes an awful lot of repetitive verbiage to write an async loop, or even a simple if-then-else with a async call in the middle, or that you don't find dealing with errors, hmm, error-prone. These are the building blocks of programming, which become suddenly non-trivial, sometimes icky, when asyncronicity is involved.

I think that the fact that everybody decides to write his own async lib after 2 weeks of coding in node.js, and that we "can't stop talking about it", should be a hint that perhaps there is something lacking in the platform?

Co-routines - especially when implemented natively - are a proven, elegant solution to this problem. It's not some kind of crazy half-assed idea that just popped into Tim's head. 

t...@creationix.com

unread,
Nov 11, 2011, 10:15:49 AM11/11/11
to nod...@googlegroups.com
I hereby close this thread from my point of view.  Through all the unfortunate noise I've learned a few important things and am satisfied with what I learned.

 1. Streamline and node-fibers and friends are not evil and are perfectly welcome parts of the community as long as they don't become part of core and newcomers don't use them as a substitute to learning async idioms.

 2. There is a very strong emotional reaction to bringing this subject up (more than I thought).  Perhaps there should be a separate mailing list where such discussions can stay civil and those who don't want to hear about it don't have to.

 3. There are some really cool things coming to node core that promise to help with some of these concerns in a different way.

Existing coroutine implementations are more thought out than my proposal and there seems to be no desire from anyone on either side to put it in core. Consider my proposal retracted.

Lets get back to making cool stuff!

Tim Caswell 



-- Sent from my HP TouchPad

Bradley Meck

unread,
Nov 11, 2011, 11:12:27 AM11/11/11
to nod...@googlegroups.com
No need for anyone to get offended. We all write stuff for ourselves. Better to list the pros and cons as we see them, and address the cons as they are brought to our attention (which all sides have cons or we wouldn't be having threads like this); rather than trying to simply talk about opinion.

Axel Kittenberger

unread,
Nov 11, 2011, 11:57:34 AM11/11/11
to nod...@googlegroups.com
>  2. There is a very strong emotional reaction to bringing this subject up
> (more than I thought).  Perhaps there should be a separate mailing list
> where such discussions can stay civil and those who don't want to hear about
> it don't have to.

"The heretic must be cast out, not because of the probability that he
is wrong but because the possibility that he is right."

The strong emotions, the FUD that is spread by people who got only a
rudimentary understanding of coroutines at best. Otherwise I perceived
so far by large node and community always very practically oriented
instead of doctrinal, you do what works for you.

BTW: When I get some time, this will be my next approach to built
something stable&use upon fibers:

coroutine(function(_) {

/* single wait call, last parameter is callback
_ knows when it is waiting or when not, so it can be the same func for
yield and resume. */

var asw = _(func, a, b, c, _);

/* callback not at end, no problem */
var asw = _(setTimeout(_, 15));

/* non standard callbacks, just wrap them */
var asw = _(func, function(a1, a2, a3) { upvalue = a1+a2+a3, _(); };

/* an interface to non standard callbacks like couchedb is easy. */
asw = _(couchdbFunc, a, b, function success(asw) { _(null, asw)},
function fail(err) { _{err});

/* do two things parallel, the answer will be in an array[2].
_ sees that the first parameter is instanceof Array. */
asw = _([func1, a, b, c, _], [func2, a2, b2,c2, _]);

});

The possible downer I see to this is 'this' not being in the funcs
what you might expect? I honestly never truely undestood how
javascript handles this and just fix it when its wrong.

Glenn Block

unread,
Nov 11, 2011, 1:51:35 PM11/11/11
to Bruno Jouhier, nodejs
Cool, will check it out, thanks for the clarification.

Sent from my Windows Phone

From: Bruno Jouhier
Sent: 11/11/2011 4:13 AM
To: nodejs
Subject: [nodejs] Re: My Humble Co-routine Proposal
Glenn,

Bruno

Glenn Block

unread,
Nov 11, 2011, 2:05:48 PM11/11/11
to t...@creationix.com, nod...@googlegroups.com
Tim, I think this is a great discussion to have. As someone very new to node figuring out how to manage async patterns is one of the first big things you slam into. The next thing you slam into is there are ten different ways to do it.


Sent from my Windows Phone

From: t...@creationix.com
Sent: 11/11/2011 7:15 AM

To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal

I hereby close this thread from my point of view.  Through all the unfortunate noise I've learned a few important things and am satisfied with what I learned.

 1. Streamline and node-fibers and friends are not evil and are perfectly welcome parts of the community as long as they don't become part of core and newcomers don't use them as a substitute to learning async idioms.

 2. There is a very strong emotional reaction to bringing this subject up (more than I thought).  Perhaps there should be a separate mailing list where such discussions can stay civil and those who don't want to hear about it don't have to.

 3. There are some really cool things coming to node core that promise to help with some of these concerns in a different way.

Existing coroutine implementations are more thought out than my proposal and there seems to be no desire from anyone on either side to put it in core. Consider my proposal retracted.

Lets get back to making cool stuff!

Tim Caswell 



-- Sent from my HP TouchPad

On Nov 11, 2011 2:39 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:

On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:

> These threads always go down this bowl-shaped hole because there are
> two camps:
> A believes async logic forces awkward code style on users
> B believes broadly-applicable solutions to these issues are worse
> than the problems

I actually believe A and B are both true and am in search of C

C: patterns in async that simplify and/or abstract the awkwardness of async

we've made huge leaps in C over the last few years but we're not done yet. streams is one of these patterns, domains is another.

tjholowaychuk

unread,
Nov 11, 2011, 11:01:55 PM11/11/11
to nodejs
I definitely dont really want to get involved in the arguments, but my
perspective
as both an application developer and library developer is that
callbacks are 100%
fine for libraries. This is perhaps because libraries are generally
quite small & focused,
but you're writing applications with well over 100k of js callbacks
become a huge pain for several
reasons. One reason being that it's simply tough to look at, which is
pretty important when you're
writing "controllers" that are more declarative in nature, secondly
it's reasonably difficult (more so at least)
to write a robust service because node and related APIs have so many
points of failure. You
have to try/catch this and that for immediate exceptions, add an
"error" listener for
things here and there, let alone handle all the (err) arguments. It
can be done, sure, but
it's MUCH nicer at this level to have multiple stacks and simply:

try {

} catch (err) {

}

Other than memory constraints there's almost no argument,
callbacks are nice sometimes, coroutines are nice sometimes I
dont think they should eliminate each-other they're complimentary.

On Nov 11, 11:05 am, Glenn Block <glenn.bl...@gmail.com> wrote:
> Tim, I think this is a great discussion to have. As someone very new to
> node figuring out how to manage async patterns is one of the first big
> things you slam into. The next thing you slam into is there are ten
> different ways to do it.
>
> Sent from my Windows Phone
> ------------------------------
> From: t...@creationix.com
> Sent: 11/11/2011 7:15 AM
> To: nod...@googlegroups.com
> Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
>
> I hereby close this thread from my point of view.  Through all the
> unfortunate noise I've learned a few important things and am satisfied with
> what I learned.
>
>  1. Streamline and node-fibers and friends are not evil and are perfectly
> welcome parts of the community as long as they don't become part of core
> and newcomers don't use them as a substitute to learning async idioms.
>
>  2. There is a very strong emotional reaction to bringing this subject up
> (more than I thought).  Perhaps there should be a separate mailing list
> where such discussions can stay civil and those who don't want to hear
> about it don't have to.
>
>  3. There are some really cool things coming to node core that promise to
> help with some of these concerns in a different way.
>
> Existing coroutine implementations are more thought out than my proposal
> and there seems to be no desire from anyone on either side to put it in
> core. Consider my proposal retracted.
>
> Lets get back to making cool stuff!
>
> Tim Caswell
>
> -- Sent from my HP TouchPad
> ------------------------------
> On Nov 11, 2011 2:39 AM, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
>
> On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:
>
> > These threads always go down this bowl-shaped hole because there are
> > two camps:
> > A believes async logic forces awkward code style on users
> > B believes broadly-applicable solutions to these issues are worse
> > than the problems
>
> I actually believe A and B are both true and am in search of C
>
> C: patterns in async that simplify and/or abstract the awkwardness of async
>
> we've made huge leaps in C over the last few years but we're not done yet.
> streams is one of these patterns, domains is another.
>
> --
> Job Board:http://jobs.nodejs.org/
> Posting guidelines:https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group athttp://groups.google.com/group/nodejs?hl=en?hl=en

tjholowaychuk

unread,
Nov 11, 2011, 11:02:40 PM11/11/11
to nodejs
damn that formatted really ugly!

Marco Rogers

unread,
Nov 12, 2011, 5:40:24 AM11/12/11
to nod...@googlegroups.com
Hi all. I've been busy this week, so I just caught up with this thread. I'd like to start by being constructive and responding to Bruno's challenge.

https://gist.github.com/1360330

My solutions speak for themselves. I'm sure lots of folks will have both large and small problems with what I did. But I think it's a worthy response for a couple of reasons.

- It's pure js
- It uses available solutions to reduce callbacks in async programming
- It shows how piping can solve real problems that aren't just about data streams

The pipe example is a complete bastardization of fd-backed data streams as they exist right now. I'm taking the Stream api as it is and applying it to other types of steaming data. I've been playing with this type of thing in some other contexts as well. I'm not sure if it's generally a good idea yet, but I really like it. More on that in another thread. Perhaps this weekend.

You'll also notice that my solutions include some re-imagining of the requirements. And some assumptions. I hesitated to do this at first. But then I realized that I don't need to read things the way that they are dictated to me. My job as a programmer is to develop solutions to the problem given. Sometimes that means looking holistically and applying solutions at different levels rather than having some local piece of code be more complex than it needs to be. Sometimes the constraints we put on ourselves are of our own making (this is a truism I try to remember in all walks of life). My code does what's needed. It's not much longer than the original. It's not callback heavy. It's even a little more flexible. It does use libraries and abstractions. But they are no more (or less) readable or understandable than what Bruno has done with magic placeholders. Looking forward to some feedback. I already have some responses to expected retorts :)

:Marco

Marco Rogers

unread,
Nov 12, 2011, 6:09:06 AM11/12/11
to nod...@googlegroups.com
I'd also like to respond about measuring the complexity of code against productivity and developer skill. It took a little time to develop my solutions to your challenge examples. But not an unreasonable amount of time. Less than it took to write this post. I spend a lot of time thinking about and designing code before writing it anyway. If we're only measuring time pounding at the keyboard, then sure, you win. It's always going to be true that writing simple imperative code using the blocking flow control constructs that we all know and love is "easier". And async control flow with callbacks "hard". Not because the language makes it hard, but because humans don't really think that way when trying to solve tasks like this. I understand that you would like to try to soften this blow, and I'm not attacking your work. Also please try to take my next comments at face value because I mean no offense. I'm responding to your story because you seem to want feedback on our experience.

I'm not sure your arguments about how you ended up with streamline make sense to me. You claim that your team is highly skilled. But that your code ended up being fragile and difficult to maintain. So by that logic, it takes some kind of expert to write complex callback logic that isn't fragile? I don't buy it. If your team was having trouble with node, perhaps node isn't the solution you should've been using in the first place. Especially since you keep saying that 90% of the code you write is the mundane kind. Why wouldn't Python, PHP, or Ruby be sufficient for you? You could even use something like event machine or tornado for the parts that you needed to be more performant and then have the mundane parts stay *actually* sync.

Instead you took on node.js which is still young and still working on elegant solutions to these new paradigms. You even pushed for it against opposition in your organization if I remember correctly. You also took on an unnecessary build-time dependency for your project and for your team. You took on a personal maintenance project that you'll have to support for your team and anyone else who decides to use it (which may be a lot based on your hard advocacy here). You also added this onus to every developer who has to maintain your code after you. What they're going to do is say "I don't know what this streamline thing is but it's not just javascript. Just add more underscores until it works".

You wrote 10,000 lines of code *before* feeling like you were going down the wrong path. And your solution was to add an entire layer of leaky abstraction on top of that instead of just switching to a more traditional technology. In short, you trust your NIH solution more than you trusted your team to come through for you with the platform that *you* pushed them to use.

And you did all of this because of what you perceive to be a large cognitive price to using callbacks? I can't help but feel like you could've reduced that price almost as much just by building some nice abstractions in js. Streamline itself is surely a testament to what you can do in pure js. And you would've saved yourself many other potential headaches from these other decisions.

When I read back on this, it sounds harsh and I actually feel like I should apologize for the brutal honesty. But that's how I view your situation as you've outlined it. I've done a lot of personal work to stop panning these alternative solutions to async control flow. I've actually come to respect them a lot even while I don't personally see a reason to make the tradeoffs they require. I enjoy the debate about these things because I also feel that non of the current solutions are sufficiently elegant. But your personal experiences that you're using to push your point just don't hold water for me.

:Marco

On Thursday, November 10, 2011 2:34:19 PM UTC-8, Bruno Jouhier wrote:
@Isaac,

Just read your blog post again. You make a lot of assertions and
explain why you would use or not use solution X, Y or Z. Fine, but
this is a bit irrelevant AFAIC.

I did not write streamline for you, and I don't really care whether
you like it or not. 

A bit of context: I chose node.js for a project, and got laughed at
initially (server-side Javascript, you must be joking!!). I had a
small team of very skilled engineers. We started to write code the
'callback way'. In a few months we had prototyped a number of things
and written about 10,000 lines of code, all in callback style.

We had been quite successful overall and the team was excited about
the technology but I had to make a hard call: are we on the right
platform? Will we be productive enough? Are we writing code that is
easy to maintain or not? Will we be able to grow the team and maintain
discipline? How much is this going to cost us (in training, in code
reviews, etc.)? Is this viable or are we going against a wall?

My assessment was that, despite the excitement, the early successes
and the fact that node was getting more and more attention, we were
writing code that was fragile and difficult to maintain. We could do
wonders with JS and some things that were really hard/painful before
(dynamic stuff, I/O) had become incredibly elegant and pleasant; but
on the flip side, things that were really easy before (like writing a
simple rule like "if path + '.js' does not exist and path is a
directory and it contains a file called 'index.js' do XYZ else ...")
had become a lot more difficult, painful to write, error prone, less
maintainable, etc.

In the kind of software we are writing, there is about 10% of clever
technical stuff and 90% of mundane not-very-technical logic. Even if
we were in better shape on the clever stuff, we were going against a
wall if we could not maintain our previous productivity and quality
standards on what makes up 90% of our code.

Fortunately, I had used the experimentation time to experiment with
flow control libraries and I had found a way to greatly simplify the
problem with a preprocessor.

So, streamline is not something that I did for myself, or to please
you. It is something that I did because we would have had to drop
node.js if we hadn't found a way to get back on a productive,
disciplined, maintainable track. And, if fibers or generators prove to
solve the problem better than a preprocessor we are going to
transition to them (and streamline is probably going to make this very
painless).

But our assessment was just that there were too many problems with
callback-style async programming, too much risk, unless we could find
a solution that would dramatically change the game. Fortunately we
found one, we are moving along at good speed on node, the team is very
enthusiastic and node is making the news everyday so I'm not getting
laughed at any more (at least not for choosing node).

All this to say that the two code samples that I posted before are not
irrelevant. They are very relevant to what a lot of people are doing.
So I strongly encourage you to write them in callback style and
compare the results with the code I gave: code size, number of
functions, complexity of the execution flow, distance between the
expression of the rules in plain English and their expression in the
code that you wrote, etc.

So, the streamline/fibers magic is probably not for you. But be open
and accept the fact that others may have different requirements than
you and that for them, a bit of magic can turn node.js from a geeky
platform that doesn't fit their bill into a very pleasant platform to
work with at all levels.

My 2 cents, FWIW.

Bruno


On Nov 10, 7:27 pm, Isaac Schlueter <i....@izs.me> wrote:
> On Thu, Nov 10, 2011 at 06:54, Axel Kittenberger <axk...@gmail.com> wrote:
> > I posted this once and do it again. I wonder when something that is a
> > practical consideration like callbacks becomes suddenly a doctrinal
> > thing to some people where even thinking otherwise makes them send you
> > verbally to be killed in Downtown New York,
>
> I don't want anything killed in SoHo, verbally or otherwise.  That was
> just some jokey wordplay.
>
> > call you stubborn, and
> > what else gives. Anyway Tim is right, coroutines are not replacement
> > or alternative for callbacks, they are simply another flow-control
> > options for the user. So what gives the hatred?
>
> I started writing a reply, and it turned into a blog post.
>
> http://blog.izs.me/post/12604303054/experts-idiots-and-taste

Axel Kittenberger

unread,
Nov 12, 2011, 6:30:45 AM11/12/11
to nod...@googlegroups.com
Thats quite some aggressive questioning. I just say, I love how
streamline made one of my projects (heavily DB reliant) much easier
and am thankfull for that. Maybe the answer would be, yes shouldn't
have used node from start for this one - but its a pity, if something
like streamline extends its field of problems its suitable for.

The solution with streaming everything is that it doesn't go well into
logical boxes. A loop with an if condition. Nested loops, etc.

I never considered streams and coroutines to be opposites. Other
languages got both.. But if you want to take that perspective as
counterparts, I daresay coroutines is the more powerfull concept. You
can code quite easily a stream with a coroutine, but not vice versa.

A realization of a pump that does draining, buffer overflows, and all
that (Supposing out calls the callback when its kernel buffers are not
full)

function mypump(callback) {
coroutine(function(wait) {
try {
while(!in.endOfStream()) {
out.write(input.get(wait), wait);
}
} catch (err) { callback(err) };
callback(null, 'fin');
}
};

Bruno Jouhier

unread,
Nov 12, 2011, 7:23:48 AM11/12/11
to nodejs
Marco,

It's true that my sample was not very well written. It can
advantageously be rewritten with less _exist calls and without
checking that the directory exists (which is overkill if we are only
testing two files inside it but could make sense if there were more
things to do with the directory).

Here is something that seems to match your implementation:

function _extendPath(_, path) {
var exts = [".js", "/main.js", "/index.js"];
for (var i in exts) {
if (_exists(_, path + exts[i])) return path + ext[i];
}
return path;
}

BTW, took me 2 minutes to write it.

The challenge that I gave could be (approximately) expressed in plain
English as A:

if there's no file called path + ".js" and if path is a directory
then if there is a "main.js" or "index.js" file inside it return the
extended path
otherwise return path

The modified version corresponds to B:

try extending path with ".js", "/main.js" or "/index.js"
if one of these extended path exists return the first that does.
otherwise (all extended paths failed) return path.

My point is not about whether B is better than A. It is about how easy
it is to translate A or B (or lots of other rules like A and B) into
code? about the time it will take someone to understand the code
(signal-to-noise ratio is important here), about how difficult it will
be to change the code if the rules change, etc.

It only takes me a few minutes to translate a rule like A or B into
code, and my claim is that a Javascript programmer will "immediately"
understand the code that I have written.

Your callback implementation is not awful at all but the signal-to-
noise ratio is lower, the programmer needs to know caolan's async
library, etc.

And I still would like to see what A's control flow translates into
(even if it is less efficient than B). There are cases where the rule
is naturally expressed in something that resembles A and where you
cannot reformulate so easily.

Bruno
It is loading more messages.
0 new messages