My Humble Co-routine Proposal

Showing 1-208 of 208 messages
My Humble Co-routine Proposal Tim Caswell 11/9/11 5:34 AM
After endless debates on the best way to "fix" callbacks with solutions ranging from code transformers to fibers to real threads, I've had some time to think about this problem myself.

As an experiment, I "ported" (more like re-implemented) node to the lua language as Luvit < https://github.com/creationix/luvit>.  I learned three very important things in this experiment.

  1. Lua is not JavaScript.  It's pretty darn close and I like both for unique reasons, but JavaScript is *far* more popular due to this thing we call the internet.
  2. V8's JS to C++ conversion is really slow compared to luajit's Lua to C layer.  Also V8 uses a lot more ram and starts up much slower.
  3. Co-routines in Lua are really nice and with very little extra sugar make a simple, obvious, and effective way to write sync style code while living in an async world.

Because of discovery #1, I don't think Luvit will ever become more popular than nodeJS, especially for anything web related.  I see Luvit being used for things like mobile devices where performance *really* matters, JIT can be disabled (think iOS), and the built-in FFI makes SDL and openGL possible without bindings.

I do think that we can take what I learned from Lua's co-routines and bring it to node with minimal changes to the language.  This proposal needs two things:

  1. A new co-routine primitive as either a node add-on or an addition to the language.
  2. De-structuring assignment to keep the syntax sane since async functions tend to "return" more than one value.

In this proposal you create a co-routine explicitly outside of the main stack.  (The main stack is *never* suspended).  Within that new stack, you can `wait()` and `resume()`.  This makes it possible to use any normal non-co-routine-aware async function and using function composition, it's trivial to make your co-routine an async function with a callback as well.

This integrates well with existing node code and doesn't introduce any new hazards that aren't already there.  No new hazards is a hard requirement for anything like this.  Programming in node is hard enough already, let's not make it any harder.

For an example, see this: <https://gist.github.com/1349451>

Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 7:14 AM
I'm watching with great interest how Luvit develops. Considering it
great what you are doing there! Currently I have another node
unrelated project (Lsyncd http://code.google.com/p/lsyncd/ ) where I
ended up writing my own event machine core with Lua scripting for its
specific use case (file event monitoring). Might be very worthwhile to
transfer it to Luvit once it reaches adolescence. For some more funky
set ups I did started to code also unblocking network transfer in
core, but going into a shared environment with that things might be
very worthwhile, since there people do this things that know better
what they are doing.

Yes coroutines. true non-threaded pure cooperative in scripting engine
processing is something very nice that is missing in Javascript
language spec. But wouldn't this be more of a V8 thing to support or a
heavy patch to it than a node thing?

I considered also moving Lsyncd to node instead and recoding it from
Lua over to Javascript, but JS is missing few features it relies upon.
For me most important, true associative arrays (that is table keys can
be anything not just integers and strings) and weak link tables
(garbage collector can remove it, if an object is only referenced by
such tables) So I rather wait for Luvit for this :-) On the other
hand, I like how JS offers me well established conveniences like
"switch", "continue", ++ operators, ?: operator, where Lua-ism can be
a bummer.

I currently work on another web project that will surely be fixed onto
node, because javascript server and client side gives here some
interesting synergy, because it shares quite some bunch of code
between server and client (and also for some people having to
work/learn with 1 language only is also a bon).

Javascript and Lua. I always think as analogy C and Pascal a decade or
two ago. The former is the more industrial language, the later the
more academic. While the later is actually the tad "cleaner" language
(whatever that means) on the large scale the formers industry power
just outruns. That doesn't remove interesting niches for the later
tough.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 7:21 AM
> For an example, see this: <https://gist.github.com/1349451>

BTW: I recently coded up something very similar if I see this correctly:

https://github.com/axkibe/node-green-light

Didn't echo on node mailinglist a lot. Got just told to use
streamline. Fiber-anything got a massive negative touch to it. (And
yes while I also dislike pthreads a great deal, there is nothing wrong
with true (green) coroutines). IMHO if there would be true V8 support
for coroutines, instead of hacking pthreads with global locks, things
might look different, would they?

Re: [nodejs] My Humble Co-routine Proposal Jann Horn 11/9/11 8:24 AM
2011/11/9 Axel Kittenberger <axk...@gmail.com>:

> with true (green) coroutines). IMHO if there would be true V8 support
> for coroutines, instead of hacking pthreads with global locks, things
> might look different, would they?

Nah, it'd still be confusing (if I understand you correctly). Example:

function User(name) {
  ...
}

User.prototype.setPassword = function(newPassword) {
  var salt = makeRandomChars(16)
  this.store('salt', salt);
  var passwordHash = hash(newPassword, salt);
  this.store('hash', passwordHash);
}

As soon as "store" becomes a blocking function, executing this
function twice might well cause you to have corrupt data in your DB.
Great. In streamlinejs, I'd be able to see the underscore and could
figure out that some caution is needed here.
What's wrong with streamlinejs? It seems to work, it has useful stack
traces, it doesn't really introduce large run-time performance
penalties... and you can even use it in the browser. If you have a
tool that works, why make a magic wand? Sure, it looks impressing and
you can do some weird magic with it that nobody can master without
studying gray magic for some time, but that's not what I expect a good
developer to want. I expect him to want something unambiguous that
just works.

PS: Yes, I do realize that that example would create inconsistent
state if the program fails at some point between those store calls.
Let's just assume it's an in-memory DB that gets cleared after each
restart so that the example becomes even more contrived.

Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 8:57 AM
> Nah, it'd still be confusing (if I understand you correctly). Example:

IMHO you didn't understand coroutines. This doesn't just doesn't mean
that anything can suddendly block/reenter.

As Tim wrote

"In this proposal you create a co-routine explicitly outside of the
main stack.  (The main stack is *never* suspended).  Within that new
stack, you can `wait()` and `resume()`.  This makes it possible to use
any normal non-co-routine-aware async function and using function
composition, it's trivial to make your co-routine an async function
with a callback as well."

No function will wait() resume() which did not create the coroutine in
first place. In this proposal (and with greenlight) it even cannot,
since it misses the relevant handlers. If the "magic" handler is
called '_' or something else is a matter of taste. Its imho this basic
misunderstanding that causes the promp refusal. Think of a coroutines
basically just as one routine.

You might actually follow one of the links to see the proposals.

Re: [nodejs] My Humble Co-routine Proposal Tim Caswell 11/9/11 9:07 AM
Anything that looks like blocking code without the user having to use
slightly special syntax is evil.  Callbacks are great and viral in
this regard. They are just difficult for certain things and debugging
is tricky with the short stack traces.  My proposed coroutines are
good because you have to explicitly call wait() to suspend your
co-routine and you can only do this within the body of a
Fiber(function () { ...}) block.  The call to Fiber is non-blocking
itself so it's caller isn't affected at all.  This explicit and
obvious place where state can change out from under you is the same as
callbacks and thus acceptable.  The underscore in streamline is like
that too, but code transformation seems a little too much for this.
But that's just my personal taste.

Also I personally usually just vanilla callbacks for everything
because I'm used to it.  I have some background in functional
programming and it's natural to me.  But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] My Humble Co-routine Proposal Dean Landolt 11/9/11 9:30 AM


On Wed, Nov 9, 2011 at 12:07 PM, Tim Caswell <t...@creationix.com> wrote:
Anything that looks like blocking code without the user having to use
slightly special syntax is evil.  Callbacks are great and viral in
this regard. They are just difficult for certain things and debugging
is tricky with the short stack traces.  My proposed coroutines are
good because you have to explicitly call wait() to suspend your
co-routine and you can only do this within the body of a
Fiber(function () { ...}) block.  The call to Fiber is non-blocking
itself so it's caller isn't affected at all.  This explicit and
obvious place where state can change out from under you is the same as
callbacks and thus acceptable.  The underscore in streamline is like
that too, but code transformation seems a little too much for this.
But that's just my personal taste.

Also I personally usually just vanilla callbacks for everything
because I'm used to it.  I have some background in functional
programming and it's natural to me.  But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.


This is a nice idea, but ISTM people that prefer a synchronous style will just end up pushing their whole app into the body of a fiber to get at the wait/resume functionality. This doesn't actually fix the interleaving hazards, it just pushes them down a level. The only way to fix them is with single-frame continuations, and for that you need syntax-level support. Conveniently, this is already being added to the language by way of generators :D

Sure, generators alone won't get us all the way to your nice, neat Fiber blocks -- we'll still need library support. Whether that ends up in core (or even as part of the language as Kris Zyp recently proposed on es-discuss), it's too early to tell. But Dave Herman's taskjs[1] has the notion of Task blocks that are an awful lot like your Fiber blocks, but without the hazards. Would that get us the rest of the way there? If so, I wonder if time may be better spent patching up the generator bits in traceur as a stopgap until v8 gets around to giving us native support.

Re: [nodejs] My Humble Co-routine Proposal tedsuo 11/9/11 9:33 AM
What would the pattern for doing several async operations in parallel  
and waiting for them to finish look like?

Ted

Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 10:05 AM

a(resume);
b(resume);
c(resume);
d(resume);
wait();
wait();
wait();
wait();

Looked it up, generators are effectively indeed the same thing as coroutines - as far I see it. Suppose then we just have to wait until ECMA 1.7 comes to V8

Re: [nodejs] My Humble Co-routine Proposal Tim Caswell 11/9/11 11:18 AM
@Ted, I've never thought that faux blocking was good for parallel
tasks.  I would just use vanilla callbacks (or if you prefer promises)
for that, but as Axel pointed out, it's almost possible with my
proposal, the tricky part is you don't know which resume happens first
so getting any interesting data out of the resume callback is
impossible. The first wait will suspend the co-routine, but the first
resume called (which is unknown since the I/O waits are parallel and
a,b,c, and d are async) will resume it and pass the results to the
first wait, then it will suspend on the second wait and so forth. The
last wait won't resume till all four resumes have been called in any
order.

@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need.  Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has?  Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows.  From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard.  This includes promises and CPS style callbacks
as well as my proposal.  The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.

Re: [nodejs] My Humble Co-routine Proposal Mikeal Rogers 11/9/11 11:39 AM

On Nov 9, 2011, at November 9, 20115:34 AM, Tim Caswell wrote:

After endless debates on the best way to "fix" callbacks with solutions ranging from code transformers to fibers to real threads, I've had some time to think about this problem myself.

Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily handled with named functions that I start to grit my tether whenever someone makes a comment like this.

As an experiment, I "ported" (more like re-implemented) node to the lua language as Luvit < https://github.com/creationix/luvit>.  I learned three very important things in this experiment.

  1. Lua is not JavaScript.  It's pretty darn close and I like both for unique reasons, but JavaScript is *far* more popular due to this thing we call the internet.
  2. V8's JS to C++ conversion is really slow compared to luajit's Lua to C layer.  Also V8 uses a lot more ram and starts up much slower.
  3. Co-routines in Lua are really nice and with very little extra sugar make a simple, obvious, and effective way to write sync style code while living in an async world.

Because of discovery #1, I don't think Luvit will ever become more popular than nodeJS, especially for anything web related.  I see Luvit being used for things like mobile devices where performance *really* matters, JIT can be disabled (think iOS), and the built-in FFI makes SDL and openGL possible without bindings.

I do think that we can take what I learned from Lua's co-routines and bring it to node with minimal changes to the language.  This proposal needs two things:

  1. A new co-routine primitive as either a node add-on or an addition to the language.
  2. De-structuring assignment to keep the syntax sane since async functions tend to "return" more than one value.

There is a very good destructuring proposal that has made it in to ES.next. I'm known for being anti new features in JavaScript and even I like the proposal, it's very clean and the syntax makes the semantics very obvious.


In this proposal you create a co-routine explicitly outside of the main stack.  (The main stack is *never* suspended).  Within that new stack, you can `wait()` and `resume()`.  This makes it possible to use any normal non-co-routine-aware async function and using function composition, it's trivial to make your co-routine an async function with a callback as well.

So you're using a v8 isolate per actor?

That is what common-node does. It's slow, we've considered it before in node.js and it's too slow.


This integrates well with existing node code and doesn't introduce any new hazards that aren't already there.  No new hazards is a hard requirement for anything like this.  Programming in node is hard enough already, let's not make it any harder.

For an example, see this: <https://gist.github.com/1349451>


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] My Humble Co-routine Proposal Dean Landolt 11/9/11 11:41 AM


On Wed, Nov 9, 2011 at 2:18 PM, Tim Caswell <t...@creationix.com> wrote:
@Ted, I've never thought that faux blocking was good for parallel
tasks.  I would just use vanilla callbacks (or if you prefer promises)
for that, but as Axel pointed out, it's almost possible with my
proposal, the tricky part is you don't know which resume happens first
so getting any interesting data out of the resume callback is
impossible. The first wait will suspend the co-routine, but the first
resume called (which is unknown since the I/O waits are parallel and
a,b,c, and d are async) will resume it and pass the results to the
first wait, then it will suspend on the second wait and so forth. The
last wait won't resume till all four resumes have been called in any
order.

@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need.  Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has?  Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows.  From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard.  This includes promises and CPS style callbacks
as well as my proposal.  The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.


Indeed -- making the preemption points explicit, as a convention, goes a long way toward solving the problem. But making it impossible to preempt without some kind of explicit sigil is the only real solution. We all cheat; if cheating with a wait can be done, it will.

But actually I was under the impression that with your proposal you could still bury a wait call deep in the call stack of a Fiber. Looking closer, it appears that you may have sidestepped this problem with a clever API. Am I right in assuming you have to pass the wait handle down through the call stack in order to preempt? If so that's an interesting improvement on typical fibers -- the very act of passing around the wait handle is your sigil that a call may be preempted.

I'm not sure I understand your concern about argument passing in generators but I suspect the `send` API surface should cure what ails ya.
Re: [nodejs] My Humble Co-routine Proposal tedsuo 11/9/11 12:35 PM

On Nov 9, 2011, at 11:18 AM, Tim Caswell wrote:

> @Ted, I've never thought that faux blocking was good for parallel
> tasks.  I would just use vanilla callbacks (or if you prefer promises)
> for that, but as Axel pointed out, it's almost possible with my
> proposal, the tricky part is you don't know which resume happens first
> so getting any interesting data out of the resume callback is
> impossible. The first wait will suspend the co-routine, but the first
> resume called (which is unknown since the I/O waits are parallel and
> a,b,c, and d are async) will resume it and pass the results to the
> first wait, then it will suspend on the second wait and so forth. The
> last wait won't resume till all four resumes have been called in any
> order.

Have you seen tamejs?  What do you think of their solution to this  
problem:

var res1, res2;
await {
        doOneThing(defer(res1));
        andAnother(defer(res2));
}
thenDoSomethingWith(res1, res2);


Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 2:10 PM
On Wed, Nov 9, 2011 at 8:18 PM, Tim Caswell <t...@creationix.com> wrote:
> @Ted, I've never thought that faux blocking was good for parallel
> tasks.  I would just use vanilla callbacks (or if you prefer promises)
> for that, but as Axel pointed out, it's almost possible with my
> proposal, the tricky part is you don't know which resume happens first
> so getting any interesting data out of the resume callback is
> impossible. The first wait will suspend the co-routine, but the first
> resume called (which is unknown since the I/O waits are parallel and
> a,b,c, and d are async) will resume it and pass the results to the
> first wait, then it will suspend on the second wait and so forth. The
> last wait won't resume till all four resumes have been called in any
> order.

Certainly not impossible. Just have to wrap the callback to wrap the
answer where it came from.

a( function(err, asw) { resume(['a', arguments]); }
b( function(err, asw) { resume(['b', arguments]); }
c( function(err, asw) { resume(['c', arguments]); }
d( function(err, asw) { resume(['d', arguments]); }
results = {};
for (var i = 0; i < 4; i++) {
   var asw = wait();
   results[asw[0]] = results[asw[1]];
}

Well looks a tad more bloated already. Certainly not simpler than the
async version would be. Albeit as Tim said, this is not the strong end
of the concept, but it still works. Might also be offloaded to a
convenience library, like async stuff does anyway.

> Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily
> handled with named functions that I start to grit my tether whenever someone makes a comment like this.

It depends on the usecase, when logic gets more complex with classic
callbacks I often see myself writing statemachines. While
statemachines are great and efficient once they work right, humans (or
at least I) are not great in getting them proper or debugging them.
For small tasks and handlers I do classic callbacks, for more complex
logic I now use streamline, like that database data analyzer I
recently worked. Streamline does quite well in moving some of the
implementation details out of the way, so I can work on a more logical
level. I haven't yet figured out how to interprete its linenumbers in
backtraces. I honestly don't care about microseconds in my use case.
However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.

Re: [nodejs] My Humble Co-routine Proposal Axel Kittenberger 11/9/11 2:20 PM
> Indeed -- making the preemption points explicit, as a convention, goes a
> long way toward solving the problem. But making it impossible to preempt
> without some kind of explicit sigil is the only real solution. We all cheat;
> if cheating with a wait can be done, it will.
> But actually I was under the impression that with your proposal you could
> still bury a wait call deep in the call stack of a Fiber. Looking closer, it
> appears that you may have sidestepped this problem with a clever API. Am I
> right in assuming you have to pass the wait handle down through the call
> stack in order to preempt? If so that's an interesting improvement on
> typical fibers -- the very act of passing around the wait handle is your
> sigil that a call may be preempted.

Correctly. For the implementation with fibers at least as long you do
not out trick by requiring the fibers modules directly. With that sort
of API Tim and I suppose have done more or less the same with
greenlight. if you did not get the wait handler down the stack for the
current context (or red sign as I called it) you cannot yield.

Re: [nodejs] My Humble Co-routine Proposal Mikeal Rogers 11/9/11 2:32 PM

On Nov 9, 2011, at November 9, 20112:10 PM, Axel Kittenberger wrote:

However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.

I can't overstate how optimized closures are in modern javascript vm's. They are crazy, crazy, crazy fast.

In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).

A language designed from the ground up to support them, like lua, might be able to do a better job optimizing. But if you look at the overhead of coroutines in erlang it's much more than callbacks in node.js using closures.

You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.

I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.

In order to keep this thread on track can we please refrain from making any assumptions or claims about potential optimizations at the VM level.

node.js does not modify the vm, that's been a rule since day 1. if fibers work without vm modifications i'm sure Tim's proposal could be implemented without modifications to the vm as well. 

Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.

-Mikeal
Re: [nodejs] My Humble Co-routine Proposal Marak Squires 11/9/11 2:39 PM
Tim -

Ignoring any of the technical merits, this seems to me that it will cause a fracture in node.js module development.

My personal opinion is that introducing and/or encouraging co-routines will just cause more mental points of failure for people starting out with node. 

Even if it makes sense on paper, adding more complexity to a system will only result in more confusion.

Learning async control flow can be daunting for new developers. If we have people going off and building co-routine based modules, new node developers will now have to learn two new things. 

I'd be interested to hear your thoughts on my perspective. 


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en



--
-- 
Marak Squires
Co-founder and Chief Evangelist
Nodejitsu, Inc.

Re: My Humble Co-routine Proposal _doq 11/9/11 2:57 PM
Rant warning!

I came to this group based off a comment from polotek about:
 1. There is a group, and the devs are super involved in it!
 2. There were a few uneducated programmers attempting to change the whole of nodeJs via combating what they think is a problem with callbacks.

At first I was like, hmmm don't seem to be that much static out here... boy was I wrong. I am glad you like lua, but the issue with callbacks in every case I have seen is user error, as in they really don't get it. Maybe there is a really big learning curve I am not seeing, but all I can see is a bunch of stubborn coders trying to convert others to there way of thinking rather getting on board with the coop para-dime, this reminds me of the transition from procedural programming to oop.

Stuff like this makes me not want to even look at your code, "Lua is not JavaScript.  It's pretty darn close," yeah... it is just as close as C is to VB. I am really surprised though all your attempts you didn't just learn to use callback more elegantly rather then trying to re-core the language.

Rant End!
Re: [nodejs] My Humble Co-routine Proposal Jorge 11/9/11 3:09 PM
On 09/11/2011, at 23:32, Mikeal Rogers wrote:
> (snip)

>

> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).


Isolated CO-routines ? Care to explain how would that work or what do you mean ?

> (snip)

>
> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.

What overhead?

The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.

Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.

> (snip)


>
> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.


See below.

> (snip)

>

> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.

Yeah. You first, please.
--
Jorge.

Re: [nodejs] My Humble Co-routine Proposal Isaac Schlueter 11/9/11 3:22 PM
I think that some time this month, we should have some kind of a
gathering of all the people with a vested interest in cooperative
multithreading approaches in node, to form a committee to come up with
a solid proposal.  We'll call it the November New Node Co-Routine
Proposal Committee.  We can meet somewhere around the lower west side
in Manhattan, NYC.

NoNewNoCoRoProCo, SoHo.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] My Humble Co-routine Proposal Mikeal Rogers 11/9/11 3:23 PM

On Nov 9, 2011, at November 9, 20113:09 PM, Jorge wrote:

> On 09/11/2011, at 23:32, Mikeal Rogers wrote:
>> (snip)
>
>>
>
>> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).
>
>
> Isolated CO-routines ? Care to explain how would that work or what do you mean ?

In javascript you avoid many side effects by having a "lock" on the whole vm while your code runs. You "block" the entire VM and state can only mutate in between callbacks. So if i do something like:

var blah = 'asdf'

doSomething(function (text) {
  blah = text
  require('module').doSomethingelse()
  write(blah)
})

I **know** that i'm writing the value of text that was passed to MY FUNCTION, even tho this variable can be mutated from outside of my scope because I only call a function that doesn't have access to that scope. I avoid side effects from another doSomething callback that might be running in "parallel". This is a common problem in threaded programming.

Let's say this callback is run many times in parallel, and that doSomethingelse() is a fiber that can go off and do IO. Other doSomethign callbacks are going to fire in parallel and mutate blah. The way to avoid this is to create an isolate that copies the state from the parent isolate and any mutations are copy-on-write so i can't mutate the state of the other isolates.

If you don't do that you get all the same horrible side effects that plague threaded programming.

>
>> (snip)
>
>>
>> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.
>
> What overhead?
>
> The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.

No, with fibers the stack is "yeilded" and stays around, the new stack goes off and does IO and pauses the parent stack until it returns. With callbacks you exit out of the event loop and the stack goes away, the function that is referenced in the event system stays around because there is a reference to it.

>
> Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.
>
>> (snip)
>>
>> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.
>
>
> See below.
>
>> (snip)
>
>>
>
>> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.
>
> Yeah. You first, please.
> --
> Jorge.
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] My Humble Co-routine Proposal Matt Sergeant 11/9/11 3:36 PM
Please don't forget about the W16 project too - that idea shows some real promise IMHO: https://github.com/sheremetyev/w16
Re: [nodejs] My Humble Co-routine Proposal Jorge 11/9/11 3:55 PM
On 10/11/2011, at 00:36, Matt wrote:

Please don't forget about the W16 project too - that idea shows some real promise IMHO: https://github.com/sheremetyev/w16

Promising. Somebody said here not too long ago that "Threads do not scale on cores as people might think they do", buuut, this is entirely based on threads and shared memory, not on separate processes.
-- 
Jorge.
Re: [nodejs] My Humble Co-routine Proposal Tim Caswell 11/9/11 4:23 PM
Mikeal, I'm not sure if you're talking about my proposal.  I promise
it's no more dangrous than async callbacks aready are.  In fact, any
function you call that has a co-routine somewhere in it is still a
normal law abiding non-blocking function call itself.  It won't pause
you and allow things to change out from under you any more than a call
to setTimeout would.  It doesn't matter what's in the body of setTimeout
because the call to setTimeout returns right away and you can be sure
your local state wasn't modified.  Don't confuse this with the evil that
was promise.wait in the early days of node.  That was truly dangerous
and I'm glad it was removed early on.

There is no need for isolates here.  There is still only only one active
stack at a time and the main "thread" (as lua calls it) or stack can
never be suspended.  Since the main "thead" can never be suspended,
there is no interleaving hazard except when the main "thread" explicitly
yields by letting the call stack unwind.

As far as implementation, I'm sure that's possible.  node-fibers does
much of what I need and also the proposed new features Dean mentions
also seem to do what I need.  The thing I want to propose is a sane
solution that's not any more dangerous than callbacks already are and is
as minimal as possible.

Going back to the hazard example:

var blah = 'asdf'

doSomething(function (text) {
   blah = text
   require('module').doSomethingelse()
   write(blah)
})

This is actually not safe even in a syncronous world without callbacks
or co-routines if the module has a reference to the doSomething function
(IE it re-enters).  But assuming you don't make that simple mistake, yes
it's safe.  Adding async calls into doSomethingelse() or my proposed
co-coroutine API doesn't change this in any way.  Unless one of your
direct descendents in the call stack re-calls doSomething before it
returns, you can assume that blah still holds the value you put in it.

This is not true of all co-routine implementations which is why I'm
making this distinction.

Re: [nodejs] My Humble Co-routine Proposal Tim Caswell 11/9/11 4:27 PM
On 11/09/2011 04:39 PM, Marak Squires wrote:
> Tim -
>
> Ignoring any of the technical merits, this seems to me that it will
> cause a fracture in node.js module development.
>
> My personal opinion is that introducing and/or encouraging co-routines
> will just cause more mental points of failure for people starting out
> with node.
>
> Even if it makes sense on paper, adding more complexity to a system
> will only result in more confusion.
>
> Learning async control flow can be daunting for new developers. If we
> have people going off and building co-routine based modules, new node
> developers will now have to learn two new things.
>
> I'd be interested to hear your thoughts on my perspective.

Marak, I completely agree with you.  That's why my proposal is so
minimal and integrates cleanly with async callbacks.  From the outside
it's no different than using Step.  The caller doesn't have to worry
that there might be some bad magic inside there it has to worry about.  
The functions you call from within the co-routine are also normal async
functions that take a callbacks.

While it does add more complexity to the system, it's 100% optional
complexity and you don't have to worry about it if you don't want to.  
You don't even have to worry about it if the libraries you use make use
of it.  There is no concept of "fiber-aware" functions.  Think of it as
a slightly more elegant version of step.

Re: [nodejs] My Humble Co-routine Proposal Bradley Meck 11/9/11 9:43 PM
That quote was probably me. Scaling to multiple cores via threads is something that means the threads must be native OS threads and the OS must be recent enough to have the multi-core scheduling be enabled (which is true for any modern OS). Even if this is achieved conflict resolution is costly when concurrency increases beyond a small amount (lock thrashing or stm rewriting). Concurrent actions that share memory that may have conflicts will react very different from isolates that aggregate data. Even once all of these line up, we must ensure that all of our environments do not use a global interpreter lock (luckily we don't have this in v8) or similar lest we be left with the problem of one thing acting on a single thread and that providing sufficient bottleneck to ruin all the concurrency. Either way, take things with a grain of salt on both ends and just code for whatever makes you get the product done :).
Re: [nodejs] My Humble Co-routine Proposal Bradley Meck 11/9/11 9:46 PM
Errata : when saying we don't have the interpreter lock in v8 I am talking about isolates and not the use of lockers.
Re: [nodejs] My Humble Co-routine Proposal Isaac Schlueter 11/9/11 10:32 PM
It seems highly odd to me that you pass "resume" as the callback, then
call "wait()" to get the results.  Also, it looks synchronous, but
it's not, and that confuses and scares me.

Despite an admirable effort, this is still magical and highly
complicated for anyone who doesn't grok it deeply.

It's too hard.  I'm gonna stick with my hand-rolled flow control tricks.

On Wed, Nov 9, 2011 at 21:46, Bradley Meck <bradle...@gmail.com> wrote:
> Errata : when saying we don't have the interpreter lock in v8 I am talking
> about isolates and not the use of lockers.
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] My Humble Co-routine Proposal Marcel 11/10/11 6:02 AM
I'll post my thoughts on this issue..

First a boilerplate rant on my node-fibers though. I never intended fibers to be used directly by clients, but instead to be used as a tool to create an ecosystem of cool new ways handle async. If the identifier "Fiber" or "yield" appear more than once or twice in your application you've probably done something terribly wrong. node-fibers is intended to be a very powerful (and dangerous!) tool used to create safer tools. You can see some of the projects enabled by fibers on the wiki: https://github.com/laverdet/node-fibers/wiki . Case in point you could implement your proposal to a T in like 10 lines using node-fibers.

I feel that an explicit "this call may block" sigil is usually important but that's not my call to make for your application. If it's important to you then enforce it with a library or coding standards. I was particularly impressed with the simplicity of node-green-light and how it handles the passing of the blocking sigil. Yeah you could cheat by calling `Fiber.yield()` directly but if that happens you have only yourself to blame. You could also just `delete Fiber.yield` after node-green-lights gets a reference to it if you feel like you can't control yourself.

This is a pillar of my defense of node-fibers but like no matter how many times I bring it up it seems to be ignored the next time a discussion of how coroutines are dangerous comes up. Arguing about how hazardous `Fiber.yield` is is a strawman.

Anyway I don't think we need any kind of coroutines in nodejs core, ever. All we need is a common idiom for asynchronous operations and we've already got that with `function(err,val)`. After that it's up to you to write an adapter to and from this common idiom. streamlinejs completely nails this with the single character adapter (underscore). All the other node-centric pure-JS solutions use this idiom as their common bridge. All external modules should (and do!) export async calls with this interface. **It doesn't matter if a module uses fibers or callbacks or Step internally as long as they're accepting a callback for blocking calls**. Then you just use whatever async library you want in your client code and you're done.

Specifically on the topic of your proposal I think you're missing one of the most compelling features of existing fibers & streamlinejs: fixing try\catch. function(err,val) isn't just a "sometimes" thing, it's the law. If an error is passed your `wait` should throw it. How many times can you write `if (err) { next(err); return; }` without wanting to quit your job?

@mikeal:
> Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily handled with named functions that I start to grit my tether whenever someone makes a comment like this.

Do you remember the rimraf challenge saga a few months ago where converting a very real application to use fibers resulted in a 40% decrease in lines of code? That wasn't a contrived example and it demonstrated a measurable reduction in code complexity. There's a small upfront cost to learn a new library but after that you reap the benefits forever.
Re: [nodejs] Re: My Humble Co-routine Proposal Tim Caswell 11/10/11 6:17 AM
Wow, thanks for the rant.  It seems I haven't been as involved on this
list as I should be for the last year due to working on webOS.


On 11/09/2011 04:57 PM, _doq wrote:
> Rant warning!
>
> I came to this group based off a comment from polotek about:
>  1. There is a group, and the devs are super involved in it!
>  2. There were a few uneducated programmers attempting to change the
> whole of nodeJs via combating what they think is a problem with callbacks.
>
> At first I was like, hmmm don't seem to be that much static out
> here... boy was I wrong. I am glad you like lua, but the issue with
> callbacks in every case I have seen is user error, as in they really
> don't get it. Maybe there is a really big learning curve I am not
> seeing, but all I can see is a bunch of stubborn coders trying to
> convert others to there way of thinking rather getting on board with
> the coop para-dime, this reminds me of the transition from procedural
> programming to oop.
>

I do feel there is a large misunderstanding of my goals here.

First and foremost, let's set the record straight.  I'm a strong
advocate of callback based programming and have been since the early
days of node.  I even fought against blocking require because async
require was more elegant to me and I didn't see the point of using
blocking require just to make it easier for people coming from other
systems.  I celebrated when promise.wait was removed and again when
"promises" were removed all-together.  I've spent *years* trying to help
newcomers to node understand callbacks and tried to convince them to not
use hacks and instead just learn it "right".  I understand the community
has grown a lot and I've been rather inactive for the last year while
working at HP, but I an *not* trying to convince anyone that co-routines
are better than callbacks.  In fact I personally rarely use co-routines
even in Lua.  I am definitely in the callback camp on this issue.

Here are some of my articles and presentations on the topic.  I won't
bother you by searching through the mailing list and posting those as well.

  - http://howtonode.org/control-flow
  - http://howtonode.org/control-flow-part-ii
  - http://howtonode.org/control-flow-part-iii
  - http://howtonode.org/do-it-fast
  - http://howtonode.org/step-of-conductor
  - http://jsconf.eu/2010/speaker/techniques_and_tools_for_tamin.html

However, despite my efforts and the efforts of others, many newcomers to
node constantly have a hard time with the callbacks.  I don't believe
that I'm just smart and everyone else is dumb.  Maybe it's just my
background in functional programming, but callbacks are very easy for me
and thus it's hard to have empathy for people who struggle with it.  But
the fact is that it's been a constant struggle for new people ever since
node was announced.  One summer it seemed to be all this list was about
and all conference presentations on node talked about.

This "humble proposal" as I put it is a compromise, a proposal, to help
those coming to node.  I have very strict rules about it's impact on the
rest of the system before I even consider promoting it.  It must not
cause any more hazard than is already there.  It must integrate cleanly
with normal callback based async functions.  It must be exposed
externally as a normal async function.  It must use internally normal
async functions.

I am sad that my post to the mailing list got sidetracked. I do not want
to discuss all the alternative ways to avoid callbacks that have already
been hashes and re-hashed on this list.  I simply want smart and
unbiased thoughts on my proposal.  While working on luvit I found this
entire other culture that understand co-routines and they consider the
people on the node list who defend callbacks (myself included) as
arrogant and refusing to be open minded about the issue.  I discussed
the issue with them on their territory where they aren't constantly told
to shut-up and go away and seen as a bother.  I learned that I
misunderstood the idea of co-routines and many of my arguments against
them were false.

> Stuff like this makes me not want to even look at your code, "Lua is
> not JavaScript.  It's pretty darn close," yeah... it is just as close
> as C is to VB. I am really surprised though all your attempts you
> didn't just learn to use callback more elegantly rather then trying to
> re-core the language.
>
I'm sorry my post offended you.  I know JavaScript far more than I know
Lua, but in my limited experience with Lua (read a couple books and
wrote a few thousand lines of code), it's very much like JavaScript.  
I'm not talking syntax but semantics.  Functions are almost identical
(first class values), closures work the same.  Tables are similar to
objects, metatables can emulate prototypes and are a lot like proxies.  
There is type coercion with truthy and falsy values. They both can run
on JIT capable vms.  Primitives are passed by value, but anything
mutable is passed by reference.  I could go on, but this is not a lua
forum.  My point here is that using libuv with luajit is about as close
to a nodejs experience as you're going to get in *any* non-javascript
environment.  It was important for my experiment to use a language that
already has co-routines in the community instead of hacking them onto
v8.  I'm expert in node and the node architecture, not in co-coutines.  
If I was going to evaluate co-routines, I needed the help of a community
that understood co-routines and has used them for years in real projects.

> Rant End!
> --

As I said in my nodeconf talk.  It's very important this community stay
friendly as it grows.  The leadership and core community members need to
help this out by not being arrogant, but instead understand the problems
newcomers face.  It's hard to understand someone else via email if we
don't see eachother face to face first, but remember there are real
humans there with real feelings and concerns.

-Tim Caswell

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 6:54 AM
I posted this once and do it again. I wonder when something that is a
practical consideration like callbacks becomes suddenly a doctrinal
thing to some people where even thinking otherwise makes them send you
verbally to be killed in Downtown New York, call you stubborn, and
what else gives. Anyway Tim is right, coroutines are not replacement
or alternative for callbacks, they are simply another flow-control
options for the user. So what gives the hatred?

Personally I'm torn between Fibers, Streamline and classical
approaches and do something else with every subproject. As said, there
is some limit to me in coding logics where classical callbacks start
to be come very tiresome - that is loops for me. And nested loops is
where it really gets me down and a solution like streamline really
eases the day a lot. Maybe node is really more suited for the simpler
I/O stuff, and if you code too much logics you should have better
chosen something with classical CGI/processes from start? Anyway if
one looks through the node module list, theres a hugh load of "control
flow" modules - are they all "just stubborn", or is there in fact an
itch they try to scratch?

And yes, Lua and Javascript are surprisingly quite close cousins but
without any common history (except scheme long down the road), call it
convergent evolution, as it happens occasionally in nature.

To go back on your original proposal, aside coding it with fibers as
interlocked pthreads that are built in the V8 and accepting their
costs I see noway to make node/javascript do efficient cooperative
routines without altering the V8, right? I was wrong before when I
said, I suppose through generators they will come anyway, since
generators are Javascript 1.7 (Mozilla) not ECMAScript and Google
targets with V8 only ECMAScript and whatever Webkit implements. Thus
if, we should best convince webkit to make something coroutine like,
then Google will duplicate that, then it will automatically come to
node :-). Personally I consider the Javascript 1.7 take on coroutines
not too great, calling it a generator and the way it is presented is
as one possible use of coroutines than the other way around.

Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 8:36 AM
> To go back on your original proposal, aside coding it with fibers as
> interlocked pthreads that are built in the V8 and accepting their
> costs I see noway to make node/javascript do efficient cooperative
> routines without altering the V8, right?

??

node-fibers right now has coroutines in v8 without messing with v8 internals. They are real coroutines too; ucontext is used in Linux and setjmp\longjmp are used in OS X. Originally it was built manually with ucontext but I've switched to libcoro because ucontext is flakey in OS X.


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 8:47 AM
> node-fibers right now has coroutines in v8 without messing with v8
> internals. They are real coroutines too; ucontext is used in Linux and
> setjmp\longjmp are used in OS X. Originally it was built manually with
> ucontext but I've switched to libcoro because ucontext is flakey in OS X.

Didn't know that, thank you for clearing that up.

Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 9:33 AM
> Anyway I don't think we need any kind of coroutines in nodejs core, ever.
> All we need is a common idiom for asynchronous operations and we've already
> got that with `function(err,val)`. After that it's up to you to write an
> adapter to and from this common idiom. streamlinejs completely nails this
> with the single character adapter (underscore). All the other node-centric
> pure-JS solutions use this idiom as their common bridge. All external
> modules should (and do!) export async calls with this interface. **It
> doesn't matter if a module uses fibers or callbacks or Step internally as
> long as they're accepting a callback for blocking calls**. Then you just
> use whatever async library you want in your client code and you're done.

I think that this should be the golden rule for async APIs: "Thou
shall use the callback(err, result) idiom". If we all do this, people
will be able to pick and choose whatever tool/library they like best
when consuming a library written by someone else. And people who
implement modules will also be able to use whatever tool/library they
like best to implement their stuff and expose it to others.

What would be problematic would be libraries that expose their APIs in
a different style and force their consumer to use a specific async
library on the side (for example mainpulate Fibers explicitly, or
using a specific "wait" call) to do anything with it. Then we would
see fragmentation.

Solutions like node-fibers or streamline, and most lightweight async
libraries comply with this "API standard". So rather than go into wars
about the solutions that we have chosen to implement our own stuff, we
should acknowledge the importance of this API convention.

>
> Specifically on the topic of your proposal I think you're missing one of
> the most compelling features of existing fibers & streamlinejs: fixing
> try\catch. function(err,val) isn't just a "sometimes" thing, it's the law.
> If an error is passed your `wait` should throw it. How many times can you
> write `if (err) { next(err); return; }` without wanting to quit your job?
>
> @mikeal:> Can we please identify the problem with callbacks. I'm not saying there
>
> isn't one, but most "examples" i see are so contrived and easily handled
> with named functions that I start to grit my tether whenever someone makes
> a comment like this.
>
> Do you remember the rimraf challenge saga a few months ago where converting
> a very real application to use fibers resulted in a 40% decrease in lines
> of code? That wasn't a contrived example and it demonstrated a measurable
> reduction in code complexity. There's a small upfront cost to learn a new
> library but after that you reap the benefits forever.

I also find that we are usually presented with examples that are
really simplistic in comparison with what we are dealing with in real
life. Rimraf was more interesting. Here are two other examples from
recent discussion threads:

A DB collection fetch followed by a loop on a cursor:

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
    /**
     * do something with every obj here
     * suppose for this example it needs total count and offset (oi).
     */
});

A simple file paths manipulation rule:

function _extendPath(_, path) {
    if (!_exists(_, path + ".js") && _exists(_, path) &&
      fs.stat(path, _).isDirectory()) {
        if (_exists(_, path + "/main.js"))
            return path + "/main";
        else if (_exists(_, path + "/index.js"))
            return path + "/index";
    }
    return path;
}

The important thing is that these pieces of code do something *very
simple* and there is no reason to over-engineer them. Maybe some of us
are writing sophisticated algorithms all day long and think that
pieces of code like these are just boring stuff that we should not be
writing anyway. But this is life, many applications have to deal with
little boring pieces of logic like these ones and someone has to write
the code, and the rules will change, and someone will have to maintain
the code. I feel that tools like streamline or node-fibers really help
express these pieces of logic in a simple and understandable form. And
so far, I have not seen a convincing w


>
>
>
>
>
>
>
> On Thu, Nov 10, 2011 at 3:32 PM, Isaac Schlueter <i...@izs.me> wrote:
> > It seems highly odd to me that you pass "resume" as the callback, then
> > call "wait()" to get the results.  Also, it looks synchronous, but
> > it's not, and that confuses and scares me.
>
> > Despite an admirable effort, this is still magical and highly
> > complicated for anyone who doesn't grok it deeply.
>
> > It's too hard.  I'm gonna stick with my hand-rolled flow control tricks.
>
Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 9:35 AM
Oops, hit the wrong key. Was almost finished anyway. Last incomplete
sentence was just:

I have not seen a convincing way to express this kind of logic with
callbacks.

Bruno
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 9:40 AM
I spent a few thoughts how to improve on the basic idea, maybe that
stop&go interfaces are  a tad too verbose to be attrictive. How about
an interface like this. If I don't miss something it should be quite
doable with fibers.

// Lets choose _ as blocking highlighter since thats well established.
coroutine(function(_) {
  // anything that does not have '_' cannot yield.

  // this one runs one asyncFunc. It expects the last paramter not
given there to be the callback,
  // which returns (err, asw). asw is returned, err is thrown.
  var asw = _(asyncFunc, param1, param2);

  // this place is reached after asyncFunc calls its callback.

  // and since it has been requested this is how you could do two
things in parallel.
  var asw1, asw2;
  _(asyncFunc1, param1, param2,  function(asw) { asw1 = asw },
  _, asyncFunc2, param2, param2, function(asw) { asw2 = asw });
  // the magic difference is _ appearing as its own paramter so here
it uses itself as
  // seperator of two calls to be run simultanously.  Dunno maybe it
would be simpler instead
  // of adding callbacks to return an array?

  // this place is reached after asyncFunc1 and asyncFunc2 call its callback
});

// this one immediatly returns into the main stack (or when coroutine
first runs into _ or returns, I don't know now)

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 9:47 AM
I think that this should be the golden rule for async APIs: "Thou
shall use the callback(err, result) idiom". 

That ain't gonna happen.  Libraries like jQuery are too established and won't change.  I've had to reject any sync solution that assumes this because I am too lazy to write 100's of wrappers for the incompatible libraries.

Re: [nodejs] Re: My Humble Co-routine Proposal Tim Caswell 11/10/11 10:12 AM
I think the node idiom for callbacks is great for node, but I don't
want to say all javascript everywhere should use it for async.  When I
made Step, I made it take advantage of this node idiom and assume it's
there.  When I released Do before that, I wrote about the importance
of all node modules adhering to this idiom so we can use whatever
async library we prefer but still share programs.

That's why I was careful in this co-routine proposal to make sure it
still used and provided node style callbacks.  In fact it's even more
agnostic than that, but that's just a side effect of it being so
minimal.  The _(fn, param, param2) idea works too, but it assume node
style callbacks and is a little more abstraction.

I guess my proposal is more of a set of rules where coroutines can be
used safely within node.  I think we all agree on that now.

 - It should be efficient, not use threads and locking in the implementation
 - it can be either a node addon like node-fibers or implemented using
generators from es.next, as long as the other rules are sastified
 - There should be a clear sigil to know when the coroutine gets
suspended and other stuff can happen.
 - The effect needs to be not viral.  If a module I depend on uses
coroutines, I shouldn't have to be aware or change my usage in any
way.


-Tim Caswell

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/10/11 10:27 AM
On Thu, Nov 10, 2011 at 06:54, Axel Kittenberger <axk...@gmail.com> wrote:
> I posted this once and do it again. I wonder when something that is a
> practical consideration like callbacks becomes suddenly a doctrinal
> thing to some people where even thinking otherwise makes them send you
> verbally to be killed in Downtown New York,

I don't want anything killed in SoHo, verbally or otherwise.  That was
just some jokey wordplay.

> call you stubborn, and
> what else gives. Anyway Tim is right, coroutines are not replacement
> or alternative for callbacks, they are simply another flow-control
> options for the user. So what gives the hatred?

I started writing a reply, and it turned into a blog post.

http://blog.izs.me/post/12604303054/experts-idiots-and-taste

RE: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/10/11 10:30 AM
Reading this thread. I really wish we had async/await type syntax in
the box. It greatly simplifies writing async code and improves
readability.

Sent from my Windows Phone
From: Tim Caswell
Sent: 11/10/2011 10:12 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal

Re: [nodejs] Re: My Humble Co-routine Proposal ajlopez 11/10/11 10:44 AM
Hi people!

Glenn, sometime ago I found:
https://github.com/koush/node/wiki/%22async%22-support-in-node.js
by @koush

But I don't know the status of such proposal.

Angel "Java" Lopez
http://ajlopez.wordpress.com
http://twitter.com/ajlopez
Re: [nodejs] Re: My Humble Co-routine Proposal Diogo Resende 11/10/11 11:08 AM
On Thu, 10 Nov 2011 09:35:47 -0800 (PST), Bruno Jouhier wrote:
> I have not seen a convincing way to express this kind of logic with
> callbacks.
>
> Bruno

I just had an idea :P

async function bar(a, b) {
   doSomething()
   return(err, results) // <-- "special return"
}
async function baz(a, b) {
   doSomethingElse()
   return(err, results)
}

{
   err1, results1 = bar(a, b)
   err2, results2 = baz(a, b)

   doSomethingAfterBarAndBaz()
}
doSomethingNotBlocked() // <-- because it's outside that scope


---
Diogo R.

Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 11:09 AM
It's a bit easy to just throw words in the air. Let us be down to
earth and take real code samples that real people will write. Again,
here are two examples that have not been invented on purpose:

A DB collection fetch followed by a loop on a cursor:

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
    /**
     * do something with every obj here
     * suppose for this example it needs total count and offset (oi).
     */

});

A simple file paths manipulation rule:

function _extendPath(_, path) {
    if (!_exists(_, path + ".js") && _exists(_, path) &&
      fs.stat(path, _).isDirectory()) {
        if (_exists(_, path + "/main.js"))
            return path + "/main";
        else if (_exists(_, path + "/index.js"))
            return path + "/index";
    }
    return path;

}

If callbacks are the answer, please, callback experts, roll up you
sleaves, show us what your code will look like and tell us how long it
took you to write it!

And it would be nice if we could do labs, put different developers at
different skill levels in front of these different pieces of code, ask
them to tell us what the code does, ask them to add an extra rule,
measure how long it takes them to make the change, measure how many
errors they will make, etc.

I'm not talking about questions of taste here. One thing we can very
easily measure is the size of the code. And seeing the code will
probably give us some good hints of what the outcome of the lab might
be.

So, please, roll up your sleeves!


Bruno
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 11:11 AM
I think the node idiom for callbacks is great for node, but I don't
want to say all javascript everywhere should use it for async. 

When I mentioned the jQuery example, I was thinking of node, not the browser.  I use jQuery and I use couch.js in node.  couch.js uses the signature of db.dosomething(opts, errCallback, successCallback).  couch.js has 30 or 40 funcs in its api and I am not going to wrap them all.

Limiting js to one signature for function calls is not acceptable to me in any environment.
Re: [nodejs] Re: My Humble Co-routine Proposal Diogo Resende 11/10/11 11:12 AM
On Thu, 10 Nov 2011 11:09:03 -0800 (PST), Bruno Jouhier wrote:
> So, please, roll up your sleeves!

+1

---
Diogo R.

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 11:13 AM
I'm not talking about questions of taste here.

I care very much about taste.  My coding experience (hell) is not determined by lines of code written, but by the pain involved.  My fingers may be old but they still type fast.

RE: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/10/11 12:00 PM
Async / await is close. The only diff is more explicitly saying I want
these two calls to execute in sequence via an await keyword before the
call to the async function.

Sent from my Windows Phone
From: Diogo Resende
Sent: 11/10/2011 11:08 AM

To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal


---
Diogo R.

--

Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: My Humble Co-routine Proposal Liam 11/10/11 12:12 PM
On Nov 10, 6:17 am, Tim Caswell <t...@creationix.com> wrote:
> However, despite my efforts and the efforts of others, many newcomers to
> node constantly have a hard time with the callbacks.  I don't believe
> that I'm just smart and everyone else is dumb.  Maybe it's just my
> background in functional programming, but callbacks are very easy for me
> and thus it's hard to have empathy for people who struggle with it.  But
> the fact is that it's been a constant struggle for new people ever since
> node was announced.

Amen. Node aspires to "supplant PHP." It ain't gonna get there unless
those developers can get basic stuff running (loops!) without becoming
confused.

To date, I have declined to try any of the sync-style alternatives,
despite having a ton of deeply-nested database transaction code. But
I'd love to try a blessed alternative.

> I am sad that my post to the mailing list got sidetracked. I do not want
> to discuss all the alternative ways to avoid callbacks that have already
> been hashes and re-hashed on this list.  I simply want smart and
> unbiased thoughts on my proposal.

Destructuring assignment just to obtain err is a bit tedious, as is
the need to test err instead of using a try block. Can you make wait()
throw?

> While working on luvit I found this
> entire other culture that understand co-routines and they consider the
> people on the node list who defend callbacks (myself included) as
> arrogant and refusing to be open minded about the issue.  I discussed
> the issue with them on their territory where they aren't constantly told
> to shut-up and go away and seen as a bother.  I learned that I
> misunderstood the idea of co-routines and many of my arguments against
> them were false.

Some folks object to the callback complaints and alternatives a bit
too sharply. If an issue generates this much smoke, there's probably a
fire.

> As I said in my nodeconf talk.  It's very important this community stay
> friendly as it grows.  The leadership and core community members need to
> help this out by not being arrogant, but instead understand the problems
> newcomers face.

Hear, hear.
Re: [nodejs] Re: My Humble Co-routine Proposal john.tiger 11/10/11 12:46 PM
On 11/10/11 11:27, Isaac Schlueter wrote:
> I don't want anything killed in SoHo, verbally or otherwise.  That was
> just some jokey wordplay.
@Isaac
I thought it was hilarious - though you might be guilty of listening to
too much Car Talk!

Since Tim has made a lot of good contributions to the Node community
(his original simple framework is what got us using Node), I think his
ideas are worth strong consideration - it would be nice if somehow flow
control (aka sequential operation) gets standardized or included in
core.   In many applications it is a pretty necessary component.


Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 12:55 PM
"I feel like I'm taking crazy pills." -- Mugatu

There are two entirely different problems being talked about and everyone who wants coroutines love to conflate them because it makes for a better example.

The problems are:
1) You have a question you want to ask and the answer to that question is necessary for your application logic. This requires a callback.
2) You have an operation, or set of operations, you want to do to move or mutate data between file descriptors. This requires a stream.

Everyone needs to stop pretending that coroutines are better than streams, it's insane. None of these examples are simpler than.

req.pipe(filed('/path/to/file')).pipe(res)

or

req.pipe(request('http://www.google.com')).pipe(res)

While streams are still changing, and improving, what we can already do with them is quite nice. It's also considerably simpler than any example anyone, ever, has posted with coroutines. Seriously, I feel like I'm taking crazy pills, people will post their contrived example which is a few lines shorter than another contrived example not using streams and claim that it's shorter/simpler, it's not, stop it, seriously.

For the other problem. Asking a question required by application logic, we currently use callbacks. A large amount of indentation occurs when, and only when, the final solution in your application logic requires data from the top of the closure before you asked these questions. That is the problem. It's solvable, I'm not convinced coroutines make it simpler, at least in JavaScript.

The largest problem I see is that the data at the top of the closure is usually a set of Stream objects, so doing that IO is going to cause you to lose data in the streams. This is solvable, we add buffering semantics to .pause().

But I digress. This thread is obsessed with the wrong problem. It's something that is less than obvious when you first use node, so it gets a lot of attention, but it's much less of a problem now than it used to be. The more work we do in modules the simpler this gets. I'm not sold on generic flow control libraries either, the best solutions I've seen are domain specific, but good things are happening.

There are a lot of people saying "this is too complicated, we need to simplify it" and then proposing solutions that have an incredibly high tolerance for new complexity and build solutions squarely outside of where we are actually making progress on solving the problem already.

This is not productive.

-Mikeal

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Re: My Humble Co-routine Proposal Liam 11/10/11 1:19 PM
On Nov 10, 12:55 pm, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> For the other problem. Asking a question required by application logic, we currently use callbacks. A large amount of indentation occurs when, and only when, the final solution in your application logic requires data from the top of the closure before you asked these questions.

Indentation isn't the annoying part. It's these everywhere in lieu of
a single try/catch
  if (err) throw err;

and these
  if (x)
    async(x, nextThing)
  else
    nextThing(null)
  function nextThing(err, result) {
     ...
  }

As for streams, maybe we need more teaching about how to both apply
and implement stream classes. It seems like you're suggesting most
loop constructs can be addressed with streams?
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 1:47 PM
Again, contrived examples disuade us from the real problem.

You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

While it might be annoying to have all these if (err) blocks a much bigger problem occurs when calling a function that throws. Most of the time, we call functions we did not write ourselves, which may have bugs which means they may throw and we can't even wrap them reliably in a try/catch because the error may be in a future callback added to the event system by that code.

We have a proposed solution for this, domains. Domains would allow one single handler around any amount of arbitrary code and inside any of the callbacks it might add to the event system.

Domains are a much simpler solution than coroutines and handle far more cases, they are so simple in fact that advocates of more complex proposals go to great lengths to ignore the domains proposal, much like they ignore streams.

I'm starting to identify a divide here.

node.js prefers solutions to problems that lend themselves well to abstractions. streams are great example, they lend themselves to amazing higher level abstractions that are nearly transparent to the user. Domains are even better, they will mostly be used by frameworks and other libraries and be invisible to application code.

Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

This is not productive.

-Mikeal

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 1:59 PM
> Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

Have you not seen Tims proposal then? I don't see how it pushes it's
API to the highest level. The coroutines stuff stays local to one
routine and it still communicates downward and upward with callbacks.

> This is not productive.

This is not productive.

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 2:05 PM

Yes, I've read Tim's proposal.

It mostly competes with streams. Have you used streams?

>
>> This is not productive.
>
> This is not productive.
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 2:12 PM
> It mostly competes with streams. Have you used streams?

Yes for streaming data. Duh. If it can be used for more, please answer
Liams question, are suggesting streams as solution to loops?

If so, can you please recode following example here done with
streamline with streams instead? (it is a snipped from a real
application)

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
   /**
    * do something with every obj here
    * suppose for this example it needs total count and offset (oi).
    */
});

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 2:31 PM
var c = dbclient.collection('example').find(_)
c.on('obj', function (obj) { 
 c.totalcount
 c.offset
})
c.on('end', function () {})

Or if you only need the full result.

dbclient.collection('example').find(_, function (e, result) {
  result.rows
  result.rows[0].offset
  result.rows[0].value
  result.totalcount
})

Like i stated originally, the scope of this problem is complex logic that requires state from the closure, if you have a question you want to ask that needs little closure state and can itself expose it's newly formed state the abstraction is simple, it is after-all a state machine.

If your complaint is the complexity of building this abstraction then I'm afraid it's falling on deaf ears. The amount one must know about how database cursors makes understanding an asynchronous counter seem trivial by comparison and both must be understood to write any database abstraction effectively.

I will again restate that I don't find most generic solutions adequate and domain specific abstractions have been much more impressive so far.

-Mikeal



On Nov 10, 2011, at November 10, 20112:12 PM, Axel Kittenberger wrote:

var cursor = dbclient.collection('example', _).find(_);
var count = cursor.count(_);
for(var obj = cursor.nextObject(_), oi = 0; obj; obj =
cursor.nextObject(_), oi++) {
  /**
   * do something with every obj here
   * suppose for this example it needs total count and offset (oi).
   */
});

Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 2:34 PM
@Isaac,

Just read your blog post again. You make a lot of assertions and
explain why you would use or not use solution X, Y or Z. Fine, but
this is a bit irrelevant AFAIC.

I did not write streamline for you, and I don't really care whether
you like it or not.

A bit of context: I chose node.js for a project, and got laughed at
initially (server-side Javascript, you must be joking!!). I had a
small team of very skilled engineers. We started to write code the
'callback way'. In a few months we had prototyped a number of things
and written about 10,000 lines of code, all in callback style.

We had been quite successful overall and the team was excited about
the technology but I had to make a hard call: are we on the right
platform? Will we be productive enough? Are we writing code that is
easy to maintain or not? Will we be able to grow the team and maintain
discipline? How much is this going to cost us (in training, in code
reviews, etc.)? Is this viable or are we going against a wall?

My assessment was that, despite the excitement, the early successes
and the fact that node was getting more and more attention, we were
writing code that was fragile and difficult to maintain. We could do
wonders with JS and some things that were really hard/painful before
(dynamic stuff, I/O) had become incredibly elegant and pleasant; but
on the flip side, things that were really easy before (like writing a
simple rule like "if path + '.js' does not exist and path is a
directory and it contains a file called 'index.js' do XYZ else ...")
had become a lot more difficult, painful to write, error prone, less
maintainable, etc.

In the kind of software we are writing, there is about 10% of clever
technical stuff and 90% of mundane not-very-technical logic. Even if
we were in better shape on the clever stuff, we were going against a
wall if we could not maintain our previous productivity and quality
standards on what makes up 90% of our code.

Fortunately, I had used the experimentation time to experiment with
flow control libraries and I had found a way to greatly simplify the
problem with a preprocessor.

So, streamline is not something that I did for myself, or to please
you. It is something that I did because we would have had to drop
node.js if we hadn't found a way to get back on a productive,
disciplined, maintainable track. And, if fibers or generators prove to
solve the problem better than a preprocessor we are going to
transition to them (and streamline is probably going to make this very
painless).

But our assessment was just that there were too many problems with
callback-style async programming, too much risk, unless we could find
a solution that would dramatically change the game. Fortunately we
found one, we are moving along at good speed on node, the team is very
enthusiastic and node is making the news everyday so I'm not getting
laughed at any more (at least not for choosing node).

All this to say that the two code samples that I posted before are not
irrelevant. They are very relevant to what a lot of people are doing.
So I strongly encourage you to write them in callback style and
compare the results with the code I gave: code size, number of
functions, complexity of the execution flow, distance between the
expression of the rules in plain English and their expression in the
code that you wrote, etc.

So, the streamline/fibers magic is probably not for you. But be open
and accept the fact that others may have different requirements than
you and that for them, a bit of magic can turn node.js from a geeky
platform that doesn't fit their bill into a very pleasant platform to
work with at all levels.

My 2 cents, FWIW.

Bruno
Re: My Humble Co-routine Proposal Liam 11/10/11 2:35 PM


On Nov 10, 1:47 pm, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> Again, contrived examples disuade us from the real problem.
>
> You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

I always throw for an unexpected error. You need an app-restarter
anyway, since something could segfault, etc.

And returning 500 is a UI style obscenity. :-)
Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 2:49 PM
@Mikeal,

No, this is not at all a valid translation. The _ is a callback
placeholder so:

* collection is async
* find is async
* count is async
* nextObject is async

And, sorry but this is life, this is working againts a SQL DB (those
still exist) and there is no choice but iterate with a cursor if you
have a large table and you have to scan lots of records (sometimes you
just can't avoid it), simply because that's the way the driver works.

Q: how do you write this with callbacks? It is too easy to just change
the problem and assume that you are going to emit events for all the
selected records, and that the setup calls are just sync.

Note: writing it with streamline takes a few minutes at most and isn't
an headache by any means. The example is actual code that works, not
some kind of pseudo code (same with the other sample I gave).

Bruno
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 2:50 PM
Your example is not removing the streamling callback _ magic? makes
quite some deliberate simplications, I suppose you assume a better
interface from the drivers than there is :-). collection() is a
callback, find() is a callback, count() is a callback (since it only
has to be calculated if you say you need it). etc. Yes its a quite a
bulk with classical async code. Anyway if I see it correctly, with
streams you will be at lastest be very stumped when needed nested
loops.

var c1 = dbclient.collection('example', _).find(_);for(var o1 =
c1.nextObject(_); o1; o1 = c1.nextObject(_) {
  var c2 = dbclient.collection(o1.refer, _).find(_);
  for(var o2 = c2.nextObject(_); o2; o2 = c2.nextObject(_) {
    c2.set('value', 1, _);
  }
}
console.log('finished');
Yes you could pause the c1 stream, start a c2 stream, if it finishes
resume the c1 stream for another object, pause it, starts. I don't buy
this being simpler.

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 2:52 PM
> And, sorry but this is life, this is working againts a SQL DB (those
> still exist)

Dunno how similar the SQL interface is, but that one was originally
against a mongoDB; I try to be cool and modern :-)

Re: My Humble Co-routine Proposal Bruno Jouhier 11/10/11 2:58 PM
Take a look at https://github.com/bjouhier/node-db and
https://github.com/bjouhier/node-db-oracle

We forked it not long ago to add cursors so the cursor stuff is not
documented yet. I'm into it these days so I assumed your example was
SQL. But we're also working with mongoDB :-) Cool stuff.

Bruno
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 3:02 PM
How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/10/11 3:12 PM
On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
> How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.

What the code above does is for every row of a potentially large
collection (or call it table in SQL), it reads a string which contains
the name for other potentially large collections (tables). For these
tables it sets the field 'value' for every row to 1.

Yes its a made up example, didn't had to do a nested loop in practice so far.

Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 6:25 PM
@tim
> - It should be efficient, not use threads and locking in the implementation
> - it can be either a node addon like node-fibers or implemented using
> generators from es.next, as long as the other rules are sastified
> - There should be a clear sigil to know when the coroutine gets
> suspended and other stuff can happen.
>  - The effect needs to be not viral.  If a module I depend on uses
> coroutines, I shouldn't have to be aware or change my usage in any
> way.

node-fibers/future, streamlinejs, and node-green-light all already meet these expectations.


@mark
> When I mentioned the jQuery example, I was thinking of node, not the browser.  I use jQuery and I use couch.js in node.  couch.js uses the signature of db.dosomething(opts, errCallback, successCallback).  couch.js has 30 or 40 funcs in its api and I am not going to wrap them all.

I disagree I feel that these are the exceptions. Just because there exists a few libraries today that break the rules or were around before the rules existed doesn't mean we should throw out the rules altogether. You can write an adapter once to fix all these functions at once if it's important to you.

function fixjQuery(fn) {
  return function() {
    var args = [].slice.call(arguments);
    var cb = args[args.length - 1];
    args[args.length - 1] = function(val) {
      cb(null, val);
    }:
    args[args.length] = cb;
    return fn.apply(this, args);
  }
}


@mikeal
> Everyone needs to stop pretending that coroutines are better than streams, it's insane.

I wasn't pretending that. I've said before that fibers aren't generally helpful with streams. I'm kind of on the fence with db results as to whether those are a stream or callback because they're kind of both for me. Personally my SQL library has a dual-mode steam and single-callback library. I just use whichever is more appropriate for the situation at hand.


> people will post their contrived example which is a few lines shorter than another contrived example not using streams and claim that it's shorter/simpler, it's not, stop it, seriously.

Dude you just posted two contrived examples. And to be honest I've needed `pipe` like twice in my life so far. When I use it, it was amazing but most of my work is not redirecting streams of google.com to response objects.


>You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

Why don't you ever throw? What's wrong with having a try\catch at the top of your incoming request and catching it there? If an uncaught exception is thrown you catch it in the handler and send down a 500 error page. Sometimes you can catch an error before it makes it that far and fail gracefully. Say, an advertisement fails to render you can catch that exception and just blank out that div instead of killing the whole page.

Throwing is a beautiful thing because it means you can't ignore errors. Errors pop up in unexpected places, and having to constantly check for errors and forward them along is silly. And you will absolutely miss errors.


> Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.

Like.. did you even look at the rimraf thing or what? I didn't see you address it in any of your emails. It wasn't a contrived example and resulted in a measurable 40% decrease in code length. It was a reasonably complex program too, I think it was about 80 or so lines of real code.


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 6:30 PM

serving a file an proxying an HTTP request are hardly contrived.



>You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.

Why don't you ever throw? What's wrong with having a try\catch at the top of your incoming request and catching it there? If an uncaught exception is thrown you catch it in the handler and send down a 500 error page. Sometimes you can catch an error before it makes it that far and fail gracefully. Say, an advertisement fails to render you can catch that exception and just blank out that div instead of killing the whole page.

Throwing is a beautiful thing because it means you can't ignore errors. Errors pop up in unexpected places, and having to constantly check for errors and forward them along is silly. And you will absolutely miss errors.

you don't throw in node.js because there isn't way to catch it nicely in callbacks. domains may change this but I'm talking about node.js as it is right now.

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 6:34 PM

On Nov 10, 2011, at November 10, 20113:12 PM, Axel Kittenberger wrote:

> On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
>> How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.
>
> What the code above does is for every row of a potentially large
> collection (or call it table in SQL), it reads a string which contains
> the name for other potentially large collections (tables). For these
> tables it sets the field 'value' for every row to 1.

So, you want to do an operation to every row in a table? Is there any conditional logic for setting or not setting that row and if so does it require information that might be in the closure?

-Mikeal

Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 8:05 PM
you don't throw in node.js because there isn't way to catch it nicely in callbacks. domains may change this but I'm talking about node.js as it is right now.

Yeah but if you're using fibers/future or streamlinejs throwing an exception will just call your callback with an err param. All I was saying is that these solutions are compelling because they fix try/catch. I can throw an exception from deep in a library function behind several async calls and catch it in my HTTP listener. I'm sure you can do this to an extent with pure-JS solutions as well, but all those solutions are essentially rebuilding the try/catch paradigm from scratch via callbacks. With fibers/futures and streamlinejs you get native throw/try/catch/finally support with no extra callbacks or chaining.


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 8:14 PM
Domains are a much simpler solution than coroutines and handle far more cases, 

I have followed this mailing list for a year and have never heard of domains.  Where can I read about them?
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 8:16 PM
I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 8:18 PM
They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list.

That aren't in core yet.

-Mikeal

On Nov 10, 2011, at November 10, 20118:14 PM, Mark Hahn wrote:

Domains are a much simpler solution than coroutines and handle far more cases, 

I have followed this mailing list for a year and have never heard of domains.  Where can I read about them?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 8:32 PM
They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list. 

I hope I'm not a pest, but I can't find anything on this and I'm really curious.  I searched the nodejs-dev list for "domain" and got nothing.  Would it be too much to ask for you to describe domains in 100 words or less?
Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 8:33 PM
I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.

Ok then don't use it. Just don't come into our thread and shit on everything we're doing. I'm saying these features are compelling to me and a non-insignificant group of node users. We don't need or want these features to be in node core. Actually Tim might be saying that, but it's not my position. If you want to solve the problem your own way then I applaud you because diversity is really healthy here. I just don't understand why you feel the need to bust into every thread about fibers and start yelling at everyone. At least Isaac is being constructive and open-minded, all you're doing is saying "fuck this and everyone involved!"
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 8:42 PM
That's concerning. We should get something written up, or maybe just get Ryan to merge them in :)

The basic idea is this (although this is *not* the exact API):

var d = process.createDomain()
// arbitrary code
d.stop()
d.on('error', function (err) { })
d.on('end', function () {})

Any code that runs between creation and the stop call will be in the domain. Any callbacks added to the event system will also be in the domain. So if:

function a (cb) {
  setTimeout(cb, 100)
}

function b (cb) {
  setTimeout(function () {
    throw new Error('asdf')
  }, 100)
}

var d = process.createDomain()
a(function () {
 setTimeout(function () {
    b(function () {})
  }, 100)
})
d.stop()

So In this code, event tho all these functions are being called in the future the event loop they are all attached to this domain and the throw in b will still be trapped by the domain. Once there is nothing left in the event system attached to the domain it fires the end event.

Hope that explains it. There is still some talk about parent/child relationships of domains but that isn't ironed out yet.

-Mikeal

On Nov 10, 2011, at November 10, 20118:32 PM, Mark Hahn wrote:

They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list. 

I hope I'm not a pest, but I can't find anything on this and I'm really curious.  I searched the nodejs-dev list for "domain" and got nothing.  Would it be too much to ask for you to describe domains in 100 words or less?

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 8:55 PM

On Nov 10, 2011, at November 10, 20118:33 PM, Marcel Laverdet wrote:

I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.

Ok then don't use it. Just don't come into our thread and shit on everything we're doing. I'm saying these features are compelling to me and a non-insignificant group of node users. We don't need or want these features to be in node core. Actually Tim might be saying that, but it's not my position. If you want to solve the problem your own way then I applaud you because diversity is really healthy here. I just don't understand why you feel the need to bust into every thread about fibers and start yelling at everyone. At least Isaac is being constructive and open-minded, all you're doing is saying "fuck this and everyone involved!"

You seem to take offense at the idea that we might want to keep the thread productive.

This thread has been examining and identify problems and use cases and evaluating solutions. Solutions that might be usable by most node.js developers.

Tim's proposal is an attempt to fix a problem he perceives and he would like it to be in core.

You interrupted the thread to talk about fibers, as you often do, even though it is out of the scope of this conversation.

Nobody went out of their way to shit on fibers, or shit on this thread, you're the one that brought it up and I stated, factually, that it is not a solution to this problem that can be used by most node.js developers.

I said nothing negative about fibers, I only said that fibers as a solution is not usable by the majority if node.js developers, an indisputable fact.

You are not helping your community (fibers) with this kind of behavior.

-Mikeal
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/10/11 8:55 PM
Hope that explains it. 

Thanks.  That is very interesting.  I will have to think about what it would mean for my code.

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/10/11 8:59 PM

On Nov 10, 2011, at November 10, 20118:55 PM, Mark Hahn wrote:

Hope that explains it. 

Thanks.  That is very interesting.  I will have to think about what it would mean for my code.

It would be a blessing for frameworks like express. express could wrap the body of it's http server handler function in a domain and associate any error created within that with the right request/response and do a nice 500 error.

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/10/11 9:11 PM
On Thu, Nov 10, 2011 at 20:05, Marcel Laverdet <mar...@laverdet.com> wrote:
> With
> fibers/futures and streamlinejs you get native throw/try/catch/finally
> support with no extra callbacks or chaining.

I take issue with the word "native" in that sentence.

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/10/11 9:31 PM
On Thu, Nov 10, 2011 at 14:34, Bruno Jouhier <bjou...@gmail.com> wrote:
> I did not write streamline for you, and I don't really care whether
> you like it or not.

Please read the last 3 paragraphs again, wherein I thoroughly explain
that I don't care either.  (So, why can't we seem to stop talking
about it...?)

> So, the streamline/fibers magic is probably not for you.

The pope is probably not a martian.

Re: My Humble Co-routine Proposal Liam 11/10/11 10:19 PM
These threads always go down this bowl-shaped hole because there are
two camps:
  A believes async logic forces awkward code style on users
  B believes broadly-applicable solutions to these issues are worse
than the problems

I suspect the only way camp A is going to sway the core team is by
congregating on a particular solution and producing a body of work
(including numerous real-world use cases) establishing its value, and
not just for newbies.

The core team is working on solutions to some of the stated problems,
e.g. domains (that's a confusing name, BTW). So you can't win this
battle on the basis of one or two language features which async
breaks.

@Mikeal, when you perceive people barking up the wrong tree, just show
them the right tree, everyone will learn more and faster. And maybe be
open-minded about trees you haven't stared up into yet? @Others, don't
respond to perceived slights with counter-punches. Ask for
clarification, or ignore them.

Async-everywhere is a new concept, and it's going to take a lot of
discussion and trials to discover the best practices for it. We need
to stop getting angry and let that process play out. These fights make
the Node community look sophomoric.
Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 11:46 PM


--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

> You interrupted the thread to talk about fibers, as you often do, even though it is out of the scope of this conversation.

This thread literally has "fibers" in the subject line. Talking about fibers in a thread about fibers is interrupting? I don't think it's that farfetched that the author of the only coroutine library for nodejs would want to chime in. Tim's proposal is not that far of a departure from what exists already in node-fibers. I don't see why we need to throw away all the lessons learned from node-fibers for what amounts to moving a few parameters around.

> I said nothing negative about fibers, I only said that fibers as a solution is not usable by the majority if node.js developers, an indisputable fact.

`npm install fibers`. It works on Linux, OS X, and Solaris. Windows support is feasible once node gets dynamic module loading working. The only thing stopping anyone is personal convictions.

> I take issue with the word "native" in that sentence.

Sorry I mean like.. not reconstructed in userland? try\catch\finally\throw are features of the language. A lot of async libraries essentially simulate that functionality, but you have to register the right callback, or call the right function. With streamline and futures you can just use the regular `throw` statement in JS and catch it somewhere else.

All I'm trying to say in this thread is the situation we have now, strictly in terms of code, is reasonable. I don't think we need node core to adopt any single flow control library. The common (err,val) idiom is enough. If everybody adheres to this standard we can all use whatever flow control library we want. It shouldn't matter if someone is using Step, Q, fibers/futures, or streamlinejs.
Re: [nodejs] Re: My Humble Co-routine Proposal Marcel 11/10/11 11:47 PM
This thread literally has "fibers" in the subject line

Err sorry I meant "co-routines" :-p
Re: My Humble Co-routine Proposal Brandon Benvie 11/11/11 12:15 AM
>>  e.g. domains (that's a confusing name, BTW).

Sidetrack, for the sake of my sanity and many others do not use that name if it ever gets implemented in core. That's nearly the most completely unsearchable keyword I can think of, even filtering it to Node or JS searches.


>> Windows support is feasible once node gets dynamic module loading working

Works fine since 0.5.10 (or .11). Only difference is compilation needs to be done using MSVC against node.lib which is produced alone with node.exe when building. I found a number of Windows optimized implementations for what would be needed to implement the C++ end of Fibers when I looked into it.

Ultimately I decided that it would be better to continue working on various patterns built on top of Harmony WeakMaps and Proxies and some other stuff to allow eventual/soon async promise type execution to be usable by implementation agnostic synchronous consumers of the code. I feel that's probably the best likely long term route for solving my specific goals.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/11/11 12:35 AM

On Nov 11, 2011, at November 11, 201112:15 AM, Brandon Benvie wrote:

>>  e.g. domains (that's a confusing name, BTW).

Sidetrack, for the sake of my sanity and many others do not use that name if it ever gets implemented in core. That's nearly the most completely unsearchable keyword I can think of, even filtering it to Node or JS searches.

unsearchable names are what all the cool kids are doing nowadays :)

unrelated note, PhoneGap project is now named Apache Callback. for serious.



>> Windows support is feasible once node gets dynamic module loading working

Works fine since 0.5.10 (or .11). Only difference is compilation needs to be done using MSVC against node.lib which is produced alone with node.exe when building. I found a number of Windows optimized implementations for what would be needed to implement the C++ end of Fibers when I looked into it.

Ultimately I decided that it would be better to continue working on various patterns built on top of Harmony WeakMaps and Proxies and some other stuff to allow eventual/soon async promise type execution to be usable by implementation agnostic synchronous consumers of the code. I feel that's probably the best likely long term route for solving my specific goals.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/11/11 12:39 AM

On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:

> These threads always go down this bowl-shaped hole because there are
> two camps:
>  A believes async logic forces awkward code style on users
>  B believes broadly-applicable solutions to these issues are worse
> than the problems

I actually believe A and B are both true and am in search of C

C: patterns in async that simplify and/or abstract the awkwardness of async

we've made huge leaps in C over the last few years but we're not done yet. streams is one of these patterns, domains is another.

Re: [nodejs] Re: My Humble Co-routine Proposal Jann Horn 11/11/11 1:28 AM

Would that also allow having stuff like the current http request in a global that's domain-bound?

> >  They were first proposed by R...

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Maili...

Re: My Humble Co-routine Proposal Bruno Jouhier 11/11/11 2:04 AM
Lots of new posts during the night here. A few notes.

1) Domains look cool. When they pop up, I'll change the try/catch/
finally rewrite rules in streamline to use them instead of setting up
all the callback scaffolding that gets generated today. That will
reduce the bloat in the generated code and it will also make it more
efficient. Nice thing is that I should be able to make the switch with
a simple change in the preprocessor: no change in source code!
Generators look cool too, and they will also allow me to greatly
simplify streamline's rewrite rules (again, without having to change
our source code). Preprocessors may look clumsy but they have some
advantages, especially with moving targets.

2) Lots of talking but I still haven't seen any convincing callback
implementation of the two little code samples that I posted yesterday.
If writing code in callback style is not an issue (which is the claim
being made, isn't it?) why doesn't someone produce implementations of
these less-than-10-lines snippets? That's only a few lines so it
should not take more than a few minutes. And don't get it wrong: I did
my share: these snippets are *not* pseudo-code, they are *real code*
(I did not say real JS!) because there is a preprocessor that turns
them into pure callback JS code.

Bruno

On Nov 11, 6:11 am, Isaac Schlueter <i...@izs.me> wrote:
> On Thu, Nov 10, 2011 at 20:05, Marcel Laverdet <mar...@laverdet.com> wrote:
> > With
> > fibers/futures and streamlinejs you get native throw/try/catch/finally
> > support with no extra callbacks or chaining.
>
> I take issue with the word "native" in that sentence.
>
>
>
>
>
>
>
> > On Fri, Nov 11, 2011 at 11:34 AM, Mikeal Rogers <mikeal.rog...@gmail.com>
> > wrote:
>
> >> On Nov 10, 2011, at November 10, 20113:12 PM, Axel Kittenberger wrote:
>
> >> > On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers
Re: My Humble Co-routine Proposal Bruno Jouhier 11/11/11 4:12 AM
Glenn,

Async/await syntax is exactly what streamline.js gives you. The syntax
is just a bit different:
* async function foo(a, b) --> function foo(a, b, _)
* await foo(v1, v2) --> foo(v1, v2, _)

I did not introduce new keywords and I used a "magic parameter"
instead for several reasons: compatibilty with JS editors,
compatibility with CoffeeScript, compatibility with existing callback
APIs (in which the callback is not always in the same position), ease
of implementation (no need to hack existing parsers), etc.
But the concept is fundamentally aligned.

And streamline adds a little very useful feature to the game: futures.
If you call foo as foo(v1, v2) instead of foo(v1, v2, _) you obtain a
future F on which you can wait later with F(_).

Bruno

On Nov 10, 7:30 pm, Glenn Block <glenn.bl...@gmail.com> wrote:
> Reading this thread. I really wish we had async/await type syntax in
> the box. It greatly simplifies writing async code and improves
> readability.
>
> Sent from my Windows Phone
> From: Tim Caswell
> Sent: 11/10/2011 10:12 AM
> To: nod...@googlegroups.com
> Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
> I think the node idiom for callbacks is great for node, but I don't
> want to say all javascript everywhere should use it for async.  When I
> made Step, I made it take advantage of this node idiom and assume it's
> there.  When I released Do before that, I wrote about the importance
> of all node modules adhering to this idiom so we can use whatever
> async library we prefer but still share programs.
>
> That's why I was careful in this co-routine proposal to make sure it
> still used and provided node style callbacks.  In fact it's even more
> agnostic than that, but that's just a side effect of it being so
> minimal.  The _(fn, param, param2) idea works too, but it assume node
> style callbacks and is a little more abstraction.
>
> I guess my proposal is more of a set of rules where coroutines can be
> used safely within node.  I think we all agree on that now.
>
>  - It should be efficient, not use threads and locking in the implementation
>  - it can be either a node addon like node-fibers or implemented using
> generators from es.next, as long as the other rules are sastified
>  - There should be a clear sigil to know when the coroutine gets
> suspended and other stuff can happen.
>  - The effect needs to be not viral.  If a module I depend on uses
> coroutines, I shouldn't have to be aware or change my usage in any
> way.
>
> -Tim Caswell
>
>
>
>
>
>
>
>
>
> On Thu, Nov 10, 2011 at 11:47 AM, Mark Hahn <m...@hahnca.com> wrote:
> >>  I think that this should be the golden rule for async APIs: "Thou
> > shall use the callback(err, result) idiom".
> > That ain't gonna happen.  Libraries like jQuery are too established and
> > won't change.  I've had to reject any sync solution that assumes this
> > because I am too lazy to write 100's of wrappers for the incompatible
> > libraries.
Re: [nodejs] Re: My Humble Co-routine Proposal vincentcr 11/11/11 6:48 AM
>> So, why can't we seem to stop talking about it...?)

Because people take issue with your assertion that 'callbacks are not that hard', which you just throw in the air without any justification, intermixed with calling people idiots.

Yes, in the trivial case, callbacks are indeed simple, and with anonymous functions are eminently readable, even expressive. Certainly, it's a lot simpler than dealing with threads.

However, I find it hard to believe that you haven't found that it takes an awful lot of repetitive verbiage to write an async loop, or even a simple if-then-else with a async call in the middle, or that you don't find dealing with errors, hmm, error-prone. These are the building blocks of programming, which become suddenly non-trivial, sometimes icky, when asyncronicity is involved.

I think that the fact that everybody decides to write his own async lib after 2 weeks of coding in node.js, and that we "can't stop talking about it", should be a hint that perhaps there is something lacking in the platform?

Co-routines - especially when implemented natively - are a proven, elegant solution to this problem. It's not some kind of crazy half-assed idea that just popped into Tim's head. 

Re: [nodejs] Re: My Humble Co-routine Proposal Tim Caswell 11/11/11 7:15 AM
I hereby close this thread from my point of view.  Through all the unfortunate noise I've learned a few important things and am satisfied with what I learned.

 1. Streamline and node-fibers and friends are not evil and are perfectly welcome parts of the community as long as they don't become part of core and newcomers don't use them as a substitute to learning async idioms.

 2. There is a very strong emotional reaction to bringing this subject up (more than I thought).  Perhaps there should be a separate mailing list where such discussions can stay civil and those who don't want to hear about it don't have to.

 3. There are some really cool things coming to node core that promise to help with some of these concerns in a different way.

Existing coroutine implementations are more thought out than my proposal and there seems to be no desire from anyone on either side to put it in core. Consider my proposal retracted.

Lets get back to making cool stuff!

Tim Caswell 



-- Sent from my HP TouchPad

Re: [nodejs] Re: My Humble Co-routine Proposal Bradley Meck 11/11/11 8:12 AM
No need for anyone to get offended. We all write stuff for ourselves. Better to list the pros and cons as we see them, and address the cons as they are brought to our attention (which all sides have cons or we wouldn't be having threads like this); rather than trying to simply talk about opinion.
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/11/11 8:57 AM
>  2. There is a very strong emotional reaction to bringing this subject up
> (more than I thought).  Perhaps there should be a separate mailing list
> where such discussions can stay civil and those who don't want to hear about
> it don't have to.

"The heretic must be cast out, not because of the probability that he
is wrong but because the possibility that he is right."

The strong emotions, the FUD that is spread by people who got only a
rudimentary understanding of coroutines at best. Otherwise I perceived
so far by large node and community always very practically oriented
instead of doctrinal, you do what works for you.

BTW: When I get some time, this will be my next approach to built
something stable&use upon fibers:

coroutine(function(_) {

/* single wait call, last parameter is callback
_ knows when it is waiting or when not, so it can be the same func for
yield and resume. */

var asw = _(func, a, b, c, _);

/* callback not at end, no problem */
var asw = _(setTimeout(_, 15));

/* non standard callbacks, just wrap them */
var asw = _(func, function(a1, a2, a3) { upvalue = a1+a2+a3, _(); };

/* an interface to non standard callbacks like couchedb is easy.  */
asw = _(couchdbFunc, a, b, function success(asw) { _(null, asw)},
function fail(err) { _{err});

/* do two things parallel, the answer will be in an array[2].
_ sees that the first parameter is instanceof Array. */
asw = _([func1, a, b, c, _], [func2, a2, b2,c2, _]);

});

The possible downer I see to this is 'this' not being in the funcs
what you might expect? I honestly never truely undestood how
javascript handles this and just fix it when its wrong.

RE: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/11/11 10:51 AM
Cool, will check it out, thanks for the clarification.

Sent from my Windows Phone
From: Bruno Jouhier
Sent: 11/11/2011 4:13 AM
To: nodejs
Subject: [nodejs] Re: My Humble Co-routine Proposal
Glenn,

Bruno

RE: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/11/11 11:05 AM
Tim, I think this is a great discussion to have. As someone very new to node figuring out how to manage async patterns is one of the first big things you slam into. The next thing you slam into is there are ten different ways to do it.


Sent from my Windows Phone

From: t...@creationix.com
Sent: 11/11/2011 7:15 AM

To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal

I hereby close this thread from my point of view.  Through all the unfortunate noise I've learned a few important things and am satisfied with what I learned.

 1. Streamline and node-fibers and friends are not evil and are perfectly welcome parts of the community as long as they don't become part of core and newcomers don't use them as a substitute to learning async idioms.

 2. There is a very strong emotional reaction to bringing this subject up (more than I thought).  Perhaps there should be a separate mailing list where such discussions can stay civil and those who don't want to hear about it don't have to.

 3. There are some really cool things coming to node core that promise to help with some of these concerns in a different way.

Existing coroutine implementations are more thought out than my proposal and there seems to be no desire from anyone on either side to put it in core. Consider my proposal retracted.

Lets get back to making cool stuff!

Tim Caswell 



-- Sent from my HP TouchPad

On Nov 11, 2011 2:39 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:

On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:

> These threads always go down this bowl-shaped hole because there are
> two camps:
> A believes async logic forces awkward code style on users
> B believes broadly-applicable solutions to these issues are worse
> than the problems

I actually believe A and B are both true and am in search of C

C: patterns in async that simplify and/or abstract the awkwardness of async

we've made huge leaps in C over the last few years but we're not done yet. streams is one of these patterns, domains is another.

Re: My Humble Co-routine Proposal tjholowaychuk 11/11/11 8:01 PM
I definitely dont really want to get involved in the arguments, but my
perspective
as both an application developer and library developer is that
callbacks are 100%
fine for libraries. This is perhaps because libraries are generally
quite small & focused,
but you're writing applications with well over 100k of js callbacks
become a huge pain for several
reasons. One reason being that it's simply tough to look at, which is
pretty important when you're
writing "controllers" that are more declarative in nature, secondly
it's reasonably difficult (more so at least)
to write a robust service because node and related APIs have so many
points of failure. You
have to try/catch this and that for immediate exceptions, add an
"error" listener for
things here and there, let alone handle all the (err) arguments. It
can be done, sure, but
it's MUCH nicer at this level to have multiple stacks and simply:

try {

} catch (err) {

}

Other than memory constraints there's almost no argument,
callbacks are nice sometimes, coroutines are nice sometimes I
dont think they should eliminate each-other they're complimentary.

On Nov 11, 11:05 am, Glenn Block <glenn.bl...@gmail.com> wrote:
> Tim, I think this is a great discussion to have. As someone very new to
> node figuring out how to manage async patterns is one of the first big
> things you slam into. The next thing you slam into is there are ten
> different ways to do it.
>
> Sent from my Windows Phone
> ------------------------------
> From: t...@creationix.com
> Sent: 11/11/2011 7:15 AM
> To: nod...@googlegroups.com
> Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
>
> I hereby close this thread from my point of view.  Through all the
> unfortunate noise I've learned a few important things and am satisfied with
> what I learned.
>
>  1. Streamline and node-fibers and friends are not evil and are perfectly
> welcome parts of the community as long as they don't become part of core
> and newcomers don't use them as a substitute to learning async idioms.
>
>  2. There is a very strong emotional reaction to bringing this subject up
> (more than I thought).  Perhaps there should be a separate mailing list
> where such discussions can stay civil and those who don't want to hear
> about it don't have to.
>
>  3. There are some really cool things coming to node core that promise to
> help with some of these concerns in a different way.
>
> Existing coroutine implementations are more thought out than my proposal
> and there seems to be no desire from anyone on either side to put it in
> core. Consider my proposal retracted.
>
> Lets get back to making cool stuff!
>
> Tim Caswell
>
> -- Sent from my HP TouchPad
> ------------------------------
> On Nov 11, 2011 2:39 AM, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
>
> On Nov 10, 2011, at November 10, 201110:19 PM, Liam wrote:
>
> > These threads always go down this bowl-shaped hole because there are
> > two camps:
> > A believes async logic forces awkward code style on users
> > B believes broadly-applicable solutions to these issues are worse
> > than the problems
>
> I actually believe A and B are both true and am in search of C
>
> C: patterns in async that simplify and/or abstract the awkwardness of async
>
> we've made huge leaps in C over the last few years but we're not done yet.
> streams is one of these patterns, domains is another.
>
> --
> Job Board:http://jobs.nodejs.org/
> Posting guidelines:https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group athttp://groups.google.com/group/nodejs?hl=en?hl=en
Re: My Humble Co-routine Proposal tjholowaychuk 11/11/11 8:02 PM
damn that formatted really ugly!
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 2:40 AM
Hi all. I've been busy this week, so I just caught up with this thread. I'd like to start by being constructive and responding to Bruno's challenge.

https://gist.github.com/1360330

My solutions speak for themselves. I'm sure lots of folks will have both large and small problems with what I did. But I think it's a worthy response for a couple of reasons.

- It's pure js
- It uses available solutions to reduce callbacks in async programming
- It shows how piping can solve real problems that aren't just about data streams

The pipe example is a complete bastardization of fd-backed data streams as they exist right now. I'm taking the Stream api as it is and applying it to other types of steaming data. I've been playing with this type of thing in some other contexts as well. I'm not sure if it's generally a good idea yet, but I really like it. More on that in another thread. Perhaps this weekend.

You'll also notice that my solutions include some re-imagining of the requirements. And some assumptions. I hesitated to do this at first. But then I realized that I don't need to read things the way that they are dictated to me. My job as a programmer is to develop solutions to the problem given. Sometimes that means looking holistically and applying solutions at different levels rather than having some local piece of code be more complex than it needs to be. Sometimes the constraints we put on ourselves are of our own making (this is a truism I try to remember in all walks of life). My code does what's needed. It's not much longer than the original. It's not callback heavy. It's even a little more flexible. It does use libraries and abstractions. But they are no more (or less) readable or understandable than what Bruno has done with magic placeholders. Looking forward to some feedback. I already have some responses to expected retorts :)

:Marco
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 3:09 AM
I'd also like to respond about measuring the complexity of code against productivity and developer skill. It took a little time to develop my solutions to your challenge examples. But not an unreasonable amount of time. Less than it took to write this post. I spend a lot of time thinking about and designing code before writing it anyway. If we're only measuring time pounding at the keyboard, then sure, you win. It's always going to be true that writing simple imperative code using the blocking flow control constructs that we all know and love is "easier". And async control flow with callbacks "hard". Not because the language makes it hard, but because humans don't really think that way when trying to solve tasks like this. I understand that you would like to try to soften this blow, and I'm not attacking your work. Also please try to take my next comments at face value because I mean no offense. I'm responding to your story because you seem to want feedback on our experience.

I'm not sure your arguments about how you ended up with streamline make sense to me. You claim that your team is highly skilled. But that your code ended up being fragile and difficult to maintain. So by that logic, it takes some kind of expert to write complex callback logic that isn't fragile? I don't buy it. If your team was having trouble with node, perhaps node isn't the solution you should've been using in the first place. Especially since you keep saying that 90% of the code you write is the mundane kind. Why wouldn't Python, PHP, or Ruby be sufficient for you? You could even use something like event machine or tornado for the parts that you needed to be more performant and then have the mundane parts stay *actually* sync.

Instead you took on node.js which is still young and still working on elegant solutions to these new paradigms. You even pushed for it against opposition in your organization if I remember correctly. You also took on an unnecessary build-time dependency for your project and for your team. You took on a personal maintenance project that you'll have to support for your team and anyone else who decides to use it (which may be a lot based on your hard advocacy here). You also added this onus to every developer who has to maintain your code after you. What they're going to do is say "I don't know what this streamline thing is but it's not just javascript. Just add more underscores until it works".

You wrote 10,000 lines of code *before* feeling like you were going down the wrong path. And your solution was to add an entire layer of leaky abstraction on top of that instead of just switching to a more traditional technology. In short, you trust your NIH solution more than you trusted your team to come through for you with the platform that *you* pushed them to use.

And you did all of this because of what you perceive to be a large cognitive price to using callbacks? I can't help but feel like you could've reduced that price almost as much just by building some nice abstractions in js. Streamline itself is surely a testament to what you can do in pure js. And you would've saved yourself many other potential headaches from these other decisions.

When I read back on this, it sounds harsh and I actually feel like I should apologize for the brutal honesty. But that's how I view your situation as you've outlined it. I've done a lot of personal work to stop panning these alternative solutions to async control flow. I've actually come to respect them a lot even while I don't personally see a reason to make the tradeoffs they require. I enjoy the debate about these things because I also feel that non of the current solutions are sufficiently elegant. But your personal experiences that you're using to push your point just don't hold water for me.

:Marco

On Thursday, November 10, 2011 2:34:19 PM UTC-8, Bruno Jouhier wrote:
@Isaac,

Just read your blog post again. You make a lot of assertions and
explain why you would use or not use solution X, Y or Z. Fine, but
this is a bit irrelevant AFAIC.

I did not write streamline for you, and I don't really care whether
you like it or not. 

A bit of context: I chose node.js for a project, and got laughed at
initially (server-side Javascript, you must be joking!!). I had a
small team of very skilled engineers. We started to write code the
'callback way'. In a few months we had prototyped a number of things
and written about 10,000 lines of code, all in callback style.

We had been quite successful overall and the team was excited about
the technology but I had to make a hard call: are we on the right
platform? Will we be productive enough? Are we writing code that is
easy to maintain or not? Will we be able to grow the team and maintain
discipline? How much is this going to cost us (in training, in code
reviews, etc.)? Is this viable or are we going against a wall?

My assessment was that, despite the excitement, the early successes
and the fact that node was getting more and more attention, we were
writing code that was fragile and difficult to maintain. We could do
wonders with JS and some things that were really hard/painful before
(dynamic stuff, I/O) had become incredibly elegant and pleasant; but
on the flip side, things that were really easy before (like writing a
simple rule like "if path + '.js' does not exist and path is a
directory and it contains a file called 'index.js' do XYZ else ...")
had become a lot more difficult, painful to write, error prone, less
maintainable, etc.

In the kind of software we are writing, there is about 10% of clever
technical stuff and 90% of mundane not-very-technical logic. Even if
we were in better shape on the clever stuff, we were going against a
wall if we could not maintain our previous productivity and quality
standards on what makes up 90% of our code.

Fortunately, I had used the experimentation time to experiment with
flow control libraries and I had found a way to greatly simplify the
problem with a preprocessor.

So, streamline is not something that I did for myself, or to please
you. It is something that I did because we would have had to drop
node.js if we hadn't found a way to get back on a productive,
disciplined, maintainable track. And, if fibers or generators prove to
solve the problem better than a preprocessor we are going to
transition to them (and streamline is probably going to make this very
painless).

But our assessment was just that there were too many problems with
callback-style async programming, too much risk, unless we could find
a solution that would dramatically change the game. Fortunately we
found one, we are moving along at good speed on node, the team is very
enthusiastic and node is making the news everyday so I'm not getting
laughed at any more (at least not for choosing node).

All this to say that the two code samples that I posted before are not
irrelevant. They are very relevant to what a lot of people are doing.
So I strongly encourage you to write them in callback style and
compare the results with the code I gave: code size, number of
functions, complexity of the execution flow, distance between the
expression of the rules in plain English and their expression in the
code that you wrote, etc.

So, the streamline/fibers magic is probably not for you. But be open
and accept the fact that others may have different requirements than
you and that for them, a bit of magic can turn node.js from a geeky
platform that doesn't fit their bill into a very pleasant platform to
work with at all levels.

My 2 cents, FWIW.

Bruno


On Nov 10, 7:27 pm, Isaac Schlueter <i....@izs.me> wrote:
> On Thu, Nov 10, 2011 at 06:54, Axel Kittenberger <axk...@gmail.com> wrote:
> > I posted this once and do it again. I wonder when something that is a
> > practical consideration like callbacks becomes suddenly a doctrinal
> > thing to some people where even thinking otherwise makes them send you
> > verbally to be killed in Downtown New York,
>
> I don't want anything killed in SoHo, verbally or otherwise.  That was
> just some jokey wordplay.
>
> > call you stubborn, and
> > what else gives. Anyway Tim is right, coroutines are not replacement
> > or alternative for callbacks, they are simply another flow-control
> > options for the user. So what gives the hatred?
>
> I started writing a reply, and it turned into a blog post.
>
> http://blog.izs.me/post/12604303054/experts-idiots-and-taste
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/12/11 3:30 AM
Thats quite some aggressive questioning. I just say, I love how
streamline made one of my projects (heavily DB reliant) much easier
and am thankfull for that. Maybe the answer would be, yes shouldn't
have used node from start for this one - but its a pity, if something
like streamline extends its field of problems its suitable for.

The solution with streaming everything is that it doesn't go well into
logical boxes. A loop with an if condition. Nested loops, etc.

I never considered streams and coroutines to be opposites. Other
languages got both.. But if you want to take that perspective as
counterparts, I daresay coroutines is the more powerfull concept. You
can code quite easily a stream with a coroutine, but not vice versa.

A realization of a pump that does draining, buffer overflows, and all
that (Supposing out calls the callback when its kernel buffers are not
full)

function mypump(callback) {
   coroutine(function(wait) {
     try {
        while(!in.endOfStream()) {
         out.write(input.get(wait), wait);
        }
     } catch (err) { callback(err) };
     callback(null, 'fin');
  }
};

Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 4:23 AM
Marco,

It's true that my sample was not very well written. It can
advantageously be rewritten with less _exist calls and without
checking that the directory exists (which is overkill if we are only
testing two files inside it but could make sense if there were more
things to do with the directory).

Here is something that seems to match your implementation:

function _extendPath(_, path) {
   var exts = [".js", "/main.js", "/index.js"];
   for (var i in exts) {
         if (_exists(_, path + exts[i])) return path + ext[i];
    }
    return path;
}

BTW, took me 2 minutes to write it.

The challenge that I gave could be (approximately) expressed in plain
English as A:

  if there's no file called path + ".js" and if path is a directory
  then if there is a "main.js" or "index.js" file inside it return the
extended path
  otherwise return path

The modified version corresponds to B:

  try extending path with ".js", "/main.js" or "/index.js"
  if one of these extended path exists return the first that does.
  otherwise (all extended paths failed) return path.

My point is not about whether B is better than A. It is about how easy
it is to translate A or B (or lots of other rules like A and B) into
code? about the time it will take someone to understand the code
(signal-to-noise ratio is important here), about how difficult it will
be to change the code if the rules change, etc.

It only takes me a few minutes to translate a rule like A or B into
code, and my claim is that a Javascript programmer will "immediately"
understand the code that I have written.

Your callback implementation is not awful at all but the signal-to-
noise ratio is lower, the programmer needs to know caolan's async
library, etc.

And I still would like to see what A's control flow translates into
(even if it is less efficient than B). There are cases where the rule
is naturally expressed in something that resembles A and where you
cannot reformulate so easily.

Bruno
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 5:51 AM


On Nov 12, 12:09 pm, Marco Rogers <marco.rog...@gmail.com> wrote:
Not because the language makes it hard, but because
> humans don't really think that way when trying to solve tasks like this. I
> understand that you would like to try to soften this blow, and I'm not
> attacking your work.

This is probably the most important point to me. Should we be forced
into "thinking differently" because we have a different runtime, or
should we get the runtime closer to the way we think "naturally".

And this is where the cursor is different for people who write
"libraries" and people who write "applications".

If you write a library, you should probably go the extra mile and re-
think your algorithms to cope with the new constraints. This is how
you will win on performance and you might discover new patterns and
new ideas during your journey. Cool!

If you write an application, you have to encode a lot of mundane rules
and you don't have the luxury to "rethink" them. You need something
that lets you translate your rules as efficiently and naturally as
possible.

>
> I'm not sure your arguments about how you ended up with streamline make
> sense to me. You claim that your team is highly skilled. But that your code
> ended up being fragile and difficult to maintain. So by that logic, it
> takes some kind of expert to write complex callback logic that isn't
> fragile? I don't buy it. If your team was having trouble with node, perhaps
> node isn't the solution you should've been using in the first place.
> Especially since you keep saying that 90% of the code you write is the
> mundane kind. Why wouldn't Python, PHP, or Ruby be sufficient for you? You
> could even use something like event machine or tornado for the parts that
> you needed to be more performant and then have the mundane parts stay
> *actually* sync.

I also considered these solutions. I cannot disclose much about what
we are doing but there were lots of good reasons *not* to go with
Python, PHP or Ruby. And frankly I'm not regretting the node.js choice
any single day :-). Streamline is precisely what allows us to have the
mundane parts "expressed" in sync form, although they get processed
"asynchronously". Why the hell would I want to go back to blocking
sync APIs when I can have the best of both worlds?

>
> Instead you took on node.js which is still young and still working on
> elegant solutions to these new paradigms. You even pushed for it against
> opposition in your organization if I remember correctly. You also took on
> an unnecessary build-time dependency for your project and for your team.
> You took on a personal maintenance project that you'll have to support for
> your team and anyone else who decides to use it (which may be a lot based
> on your hard advocacy here). You also added this onus to every developer
> who has to maintain your code after you. What they're going to do is say "I
> don't know what this streamline thing is but it's not just javascript. Just
> add more underscores until it works".

There is no build-time dependency. The files are preprocessed on-
demand at require time and cached.

And the maintenance is low. I rewrote the algorithm once (to align it
on what I had published on my blog) and I had to fix a few edge cases
in the transformation. But I do this as a hobby. This is not taking
significant time off my work days (and it is taking me less time than
posting on the mailing list :-).

Our developers don't care that this is not strictly Javascript (we try
to hire people who are not too religious about languages :-). I did
not have to train them, they caught it right away (a simple
"placeholder" rule is much easier to learn than new patterns, new
APIs, etc). And they like it!

The rule is not "add more underscores until it works", it is: "put an
underscore everywhere you would have a callback otherwise". You make
it look like this is trial and error and guesswork. It is not!

>
> You wrote 10,000 lines of code *before* feeling like you were going down
> the wrong path. And your solution was to add an entire layer of leaky
> abstraction on top of that instead of just switching to a more traditional
> technology. In short, you trust your NIH solution more than you trusted
> your team to come through for you with the platform that *you* pushed them
> to use.
>
> And you did all of this because of what you perceive to be a
> large cognitive price to using callbacks? I can't help but feel like you
> could've reduced that price almost as much just by building some nice
> abstractions in js. Streamline itself is surely a testament to what you can
> do in pure js. And you would've saved yourself many other potential
> headaches from these other decisions.

You are imagining things here. The streamline decision has not created
headaches. It has created peace of mind instead! The project is moving
at good speed, the main risks (code too difficult to write and
maintain, high training costs, node.js not getting proper traction --
was not obvious when we started) are behind us and the team and
management are very enthusiastic about what we are doing. So don't
make it sound like this was a blunder. It is exactly the opposite.

Streamline is actually a bit of a life insurance for us: "domains" are
coming, generators too, and coroutines/fibers are also getting
traction. I don't know what is going to win (or even whether there is
going to be a winner) but I know that the code that streamline
generates today is not the code we will be generating in the long term
(and this is comforting because some of the callback patterns are a
bit scary). I can change the preprocessor and take advantage of new
runtime features as they come. So, our code will perform better and
better with time, and we won't have to rewrite our modules, just
upgrade the preprocessor! And when things settle down and when the
translation becomes so trivial that it does not add much value in the
chain, we'll just preprocess one more time and use the generated code
as our master source (and we'll say goodbye to streamline).
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 5:59 AM
And here is how I write the pump in streamline:

while (data = in.read(_))
  out.write(_, data);

And if I need to transform data between in and out (because everything
is not a proxy), I will write it as:

while (data = in.read(_)) {
  data = tranform(data, _);
  out.write(_, data);
}

Of course, it looks dull. So what?

Bruno
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 6:06 AM
Forgot one thing: I can also pipeline things. Here is how it goes with
the read:

var readF = in.read();
while (data = readF(_)) {
   readF = in.read();
   data = tranform(data, _);
   out.write(_, data);
}


This message has been hidden because it was flagged for abuse.
Re: My Humble Co-routine Proposal Liam 11/12/11 10:02 AM
On Nov 12, 3:09 am, Marco Rogers <marco.rog...@gmail.com> wrote:
> So by that logic, it
> takes some kind of expert to write complex callback logic that isn't
> fragile? I don't buy it. If your team was having trouble with node, perhaps
> node isn't the solution you should've been using in the first place.
> Especially since you keep saying that 90% of the code you write is the
> mundane kind. Why wouldn't Python, PHP, or Ruby be sufficient for you? You
> could even use something like event machine or tornado for the parts that
> you needed to be more performant and then have the mundane parts stay
> *actually* sync.

Hey Marco! Dude, don't forget that the main appeal of Node to (most?)
users is that it's Javascript on the server. JS + Posix is a huge
productivity win. Its low per-connection overhead is icing on the
cake.

And the workarounds necessary for async apis are candle-wax on the
icing. Loss of try/catch isn't aesthetic, it's a significant reduction
of the language.
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 11:37 AM
I'm not suggesting that callback programming is perfect as it is. Quite the opposite in fact. As I said, that's why I always participate vigorously in these discussions. At any point, I'm prepared to change my outlook if a truly elegant solution presents itself, that doesn't have what I perceive to be downsides. My feeling is that these alternate solutions have their place, but they come with other tradeoffs. I prefer to stick with pure javascript and tradeoffs I'm comfortable with. As I said, I'm not panning these solutions at all (at least not anymore, I'm reformed :)

I think one thing that bothers me is that try/catch is an issue, but it's not a huge issue to me. Since I've started programming javascript (years before node), I've used try/catch less and less and instead taken a different approach. Whitelisting inputs, parameter checking, sensible defaults, good tests (still working on this one). The result of embracing these things is that I find I can confidently deploy code without a bunch of try/catches everywhere. Sure in some places they are still needed. Functions that you can can't change and you can't stop from throwing, e.g. JSON.parse. But those are usually synchronous and a small local try/catch is not a problem. This line of thinking is where Mikeal is coming from when he says things like "you never throw". The point is that with this scheme, throwing errors is *no longer the primary way you signal exceptions*. If you buy that (I'm sure lots of people don't, and that's fine), but if you do buy that, then you don't miss try/catch so much. And it's not such a huge deal that a control flow solution doesn't support it well.

This is probably the best post I've managed to write to try and explain my thinking around this subject. I'm still learning too. But I hope it helps to explain that not everyone on this side is just stubborn and hating on you guys. It's funny to me because we all agree that async programming is different. So we have to think differently about how we construct our programs. Yet other constructs, like try/catch, are resistant to this same type of rethinking. The current batch of popular programming languages aren't the last word on how to develop solutions. My feeling is that in 50 years they'll look significantly different. And those programmers will think our constructs are either quaint but flawed, or a huge pain and not elegant.

:Marco

On Saturday, November 12, 2011 10:02:32 AM UTC-8, Liam wrote:
Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 11:49 AM


On Saturday, November 12, 2011 9:04:00 AM UTC-8, Vladimir wrote:
I think that the fact that everybody decides to write his own async lib after 2 weeks of coding in node.js, and that we "can't stop talking about it", should be a hint that perhaps there is something lacking in the platform?

Let me do some score here.

var upper_score_group = ['Bruno Jouhier', 'Axel Kittenberger', 'Tim Caswell', 'Glenn Block', 'Angel Java Lopez', 'Diogo Resende', 'Liam', 'Marcel Laverdet', 'Bradley Meck', 'vincentcr', 'Mark Hahn', 'Vladimir Perlov'] ;

var under_score_group = ['Mikeal Rogers', 'Isaac Schlueter', 'Marco Rogers'];

var undefined_score_group = [];

if (upper_score_group.length > under_score_group.length) { // 12 > 3
   res.end('upper_score_group is a winner!')
}

Winner of what exactly? When I enter these discussions, I try to get people's input and respond with my own thoughts in hopes of progressing the discussion. I didn't know there was a prize at the end. It may be important to agree that there's no goal in this debate. Bruno, Marcel, and co are not trying to get their solutions into core. Ryan, Mikeal and co are not trying to keep these guys from participating the in the community (it may seem that way sometimes, but trust me, I know these guys). At this point, the debate only has 2 reasons for existing that I can see.

1) We *might* stumble onto some nice 3rd option that everyone likes.
2) Nerds like to argue.
Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/12/11 12:17 PM
Yeah, Marco's right, there's really no "winning" here.

It's not as if we have to agree.  Everyone just says what they think,
then responds to everyone who thinks differently by saying "Nuh uh!
It's better my way!"

Not  value judgement.  Just an observation.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/12/11 12:17 PM
Don't count me in the upper_score (whatever that means).  I tried all the sync libs and was unsatisfied.  So I thought maybe they just didn't match my way of thinking and I wrote my own.  I still wasn't satisfied so I'm back to callbacks.  I do still hold out hope that some ideal solution will come along but it probably ain't gonna happen.

I have been working on ideas that flatten the left margin syntactically without any semantics change. I use coffeescript so this is particularly challenging.  In case anyone cares, here is my current thinking.  It doesn't even require a lib, much less compiling.  It is sort of perl-ish with every callback being $n.  It is also sort of basic-ish in terms of line numbering.  It is obviously a work in progress.

$0 = -> setTimeout $1, 100
$1 = -> db.open docId, options, $2, $3
$2 = (err...) -> console.log 'error', err
$3 = (doc) -> ... do something with doc ...
$0()

Pros: No nesting margin problem.  Callbacks are obvious.  No thinking up function names.
Con: Ugliness.
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 12:31 PM
Hi Marco,

I just want to react on what you are saying about try/catch. I've
programming with try/catch for the last 25 years (and I even wrote my
own TRY/CATCH macros in C with setjmp/longjmp at a time where the
language did not have it -- got the idea from Stallman's emacs
internals). This is probably the language feature that is the most
misunderstood and I'd like to say a few words about it.

IMO, try/catch goes with the "contracting metaphor" (Bertrand Meyer -
Object Oriented Software Construction): every function as a "contract"
behind it. If the contract can be fulfilled you "return". If it cannot
be fulfilled, you "throw".

So, there is not a single API for something like readFile(path), there
are at least two APIs that correspond to two contracts:
* readFile: path must be the path of a valid, readable, file, call
will throw if this is not the case.
* tryReadFile: path may be anything, call will return null if file
does not exits, cannot be read, etc.

And, if you are lucky and have such dual API here is how you should
use it:
* if you are in a situation where you know that path will always exist
and be readable (unless a malicious user or process removed it, or the
file system got corrupted, or whatever -- but you consider all this
"exceptional" and there is not much you can do about it anyway) then
you should use readFile.
* otherwise, you should use tryReadFile and test for null.

But, and this is the important part, DO NOT PUT A TRY/CATCH AROUND
EVERY readFile call. if readFile cannot fullfil your contract, your
function won't be able to fulfill its contract either. So just let the
exception percolate.

If you follow this, you may have lots of "throw" statements (as soon
as some contract cannot be fulfilled you throw -- this is called "fail
fast") and very few "try/catch" constructs: you will only catch in a
few "strategic" places where you have a way to retry the operation,
compensate, etc. and, in the end, fulfill your own contract (every
function has a contract).

And then you will have *great* exception handling and robust and clean
code. You will have a clear separation between all the conditions that
you consider "normal" and that you handle through "normal" programming
constructs (if/else tests) and "exceptional" conditions that you deal
with in a few places (in a web app, this is typically the top level
dispatch of a request where you format a "an error has occured"
message and return it to the user -- may sound lousy but that's just
life: if you cannot fullfil a contract don't try to pretend that you
fulfilled it, be honest and inform the user that something went
wrong).

So when I see a developper write code with try/catch everywhere, I
know that he has completely misundersood the language feature and that
I need to spend a bit of time with him.

The worst is when I see someone write:

  try { foo() } catch (ex) { throw ex; }

Then I know that the guy has completely misunderstood the feature!!!

So, unfortunately, try/catch is a bit of a subtle issue. If misused
(and it often is), it leads to lots of extra noise in the code and it
obfuscates the exception handling. But if used well, it does marvels.

For a great example of try/catch in action, see Marcel's response to
the rimraf challenge: https://gist.github.com/1131136

A bit off topic but when I see someone say "I'm not using try/catch
and I'm happy because I got rid of all these try/catch constructs", I
just feel that that person did not understand the language feature in
the first place.

Bruno
Re: My Humble Co-routine Proposal Liam 11/12/11 12:39 PM
On Nov 12, 11:37 am, Marco Rogers <marco.rog...@gmail.com> wrote:
> My feeling is that in 50
> years they'll look significantly different. And those programmers will
> think our constructs are either quaint but flawed, or a huge pain and not
> elegant.

I'll say. See http://flowlang.net/ and hold on to your hat.
Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/12/11 12:58 PM
I can't take flow seriously as a programming language because there's
no hello-world sample program on the first page.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/12/11 1:05 PM
Flowlang is interesting in the context of this thread.  It is basically programming with pipes which is what Mikeal likes.

 
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 1:08 PM


On Saturday, November 12, 2011 4:23:48 AM UTC-8, Bruno Jouhier wrote:
Marco,

It's true that my sample was not very well written. It can
advantageously be rewritten with less _exist calls and without
checking that the directory exists (which is overkill if we are only
testing two files inside it but could make sense if there were more
things to do with the directory).

See this is telling though. I took your challenge and ended up writing code without streamline that's still short, just as good IMO, and even you admit has certain advantages over what you ended up with at first. Even though you have the massive "advantage" of sync looking code. I don't presume to say I know what that means exactly. But I think it hurts your argument somewhat.
 

Here is something that seems to match your implementation:

function _extendPath(_, path) {
   var exts = [".js", "/main.js", "/index.js"];
   for (var i in exts) {
         if (_exists(_, path + exts[i])) return path + ext[i];
    }
    return path;
}

BTW, took me 2 minutes to write it.

That's great. You beat me by 8 minutes. What are you going to do with all that extra time :P 

Yes, how long it takes to develop solutions matters. But as developers we get better and faster at applying the same patterns the more we do it. The way I code has converged down to at least as fast as anyone else's on average. And I still don't have a dependency. I think this nitpicking about "how fast can you think of this" is kind of spurious and makes you sound snarky. Hence the snark in return :) Also what does it say that you think writing your "core business logic" should take as little time as possible?



The challenge that I gave could be (approximately) expressed in plain
English as A:

  if there's no file called path + ".js" and if path is a directory
  then if there is a "main.js" or "index.js" file inside it return the
extended path
  otherwise return path

The modified version corresponds to B:

  try extending path with ".js", "/main.js" or "/index.js"
  if one of these extended path exists return the first that does.
  otherwise (all extended paths failed) return path.


This isn't quite the right comparison. You went from describing requirements to describing implementation. "Try extending path with main.js" in B is the way you implement "if there is a "main.js" in A. But regardless of that, my translation was just fine. Any programmer I consider to be skilled in js programming would handle that translation just fine as well. Sure you can shave minutes off your time. But until you're shaving off an order of magnitude, it's really not convincing.

My point is not about whether B is better than A. It is about how easy
it is to translate A or B (or lots of other rules like A and B) into
code? about the time it will take someone to understand the code
(signal-to-noise ratio is important here), about how difficult it will
be to change the code if the rules change, etc.

I also don't think it's genuine that you claim callbacks are hard to understand but won't admit that streamline can also be hard for some. All the underscores interspersed in my js are weird. I have to think twice about where they go. Your signal is my noise. And then you say "just place them wherever you would put a callback". Well then that presumes I know where all the callbacks go and I should be able to write nice robust code without streamline doesn't it? It may be more "noisy" to some, but to me it's just fine. It's all relative I guess. And if I still can't do callbacks well, but I can do streamline, it feels to me like that means I can only program business logic in node by trusting some abstraction and hoping it doesn't leak. That doesn't sound good either.

I'm sorry if I'm being obtuse here, and I feel I'm dangerously close to disparaging you or your team which I certainly DO NOT want to do. I believe you've had great success with streamline and I applaud that. We should all do what works. But your arguments come across as though you're surprised anyone else can write complex logic in node confidently without some abstraction. I just don't see that as a general truth. If that's not what you're implying, my apologies in advance. But I'd like for you to clarify what your argument is because it'll help me assure that I'm responding to you int he right context.

"Everyone should use Streamline"
"Use Streamline if you want, I don't care"
"Don't use Streamline unless you think like I do"
"If you don't like Streamline, I suspect there's something wrong with you"
"Here's Streamline, please use it. But if you don't like it, keep it to yourself"

Or are we just nerds having a good-natured argument at this point? I'm cool with that. For the record, my general argument is this:

These alternative flow control solutions certainly have advantages, but they also introduce some negatives that I'm not comfortable with. And considering that I'm pretty comfortable with vanilla javascript, I really don't see a compelling reason to use them. It's more like "the devil you know..." type of thing I guess.

:Marco



It only takes me a few minutes to translate a rule like A or B into
code, and my claim is that a Javascript programmer will "immediately"
understand the code that I have written.

Your callback implementation is not awful at all but the signal-to-
noise ratio is lower, the programmer needs to know caolan's async
library, etc.

And I still would like to see what A's control flow translates into
(even if it is less efficient than B). There are cases where the rule
is naturally expressed in something that resembles A and where you
cannot reformulate so easily.

Bruno

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/12/11 1:10 PM
Well, sure, except for the fact that Flowlang doesn't exist.

http://www.flowlang.net/p/status.html
http://www.flowlang.net/p/source-code.html
https://github.com/flowlang/flow

It's just one guy, a manifesto, blog with 4 posts over the last 2
years, and a mailing list with virtually no activity.

Until you implement it, it's not a thing, it's the idea of a thing.
I've got an idea about time travel.  A manifesto, too.

On Sat, Nov 12, 2011 at 13:05, Mark Hahn <ma...@hahnca.com> wrote:
> Flowlang is interesting in the context of this thread.  It is basically
> programming with pipes which is what Mikeal likes.
>
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 1:45 PM

A bit off topic but when I see someone say "I'm not using try/catch
and I'm happy because I got rid of all these try/catch constructs", I
just feel that that person did not understand the language feature in
the first place.


Ha! We have a winner!

"If you don't like Streamline, I suspect there's something wrong with you"

I'm gonna respond to this more seriously later, because I think it's interesting. Just couldn't resist.

:Marco
 
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 2:17 PM
No Marco, you got it wrong: I did not think one second about
streamline while writing this. I wrote similar posts on forums years
ago at a time were streamline was still associated with Reynolds
numbers in my mind. Was just a distraction!

If you don't get the point, fine. If you have interesting follow up
comments please post them. We might get completely off tracks but this
thread has gone a bit wild anyway. As long as they remain polite and
professional, debates are ok.

Bruno
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 2:30 PM
> I also don't think it's genuine that you claim callbacks are hard to
> understand but won't admit that streamline can also be hard for some. All
> the underscores interspersed in my js are weird. I have to think twice
> about where they go. Your signal is my noise.

There is just a limit to the amount of "bad faith" I can take.

I'm trying to be constructive and interesting in what I write (maybe a
few people will google "programmming by contract" or "Bertand Meyer"
and they will learn a few things -- and if I can give them a bit of
advice, don't take everything BM says as pure gold, there are great
things but also things that don't work at all in his books -- just try
things out and make you mind by yourself -- and if you're not
interested, fine, just ignore my post).

But when I get this kind of response, I don't feel like argumenting
any further.

Bruno.
Re: My Humble Co-routine Proposal Bruno Jouhier 11/12/11 2:52 PM
> do what works. But your arguments come across as though you're surprised
> anyone else can write complex logic in node confidently without some
> abstraction. I just don't see that as a general truth. If that's not what
> you're implying, my apologies in advance. But I'd like for you to clarify
> what your argument is because it'll help me assure that I'm responding to
> you int he right context.
>
> "Everyone should use Streamline"
> "Use Streamline if you want, I don't care"
> "Don't use Streamline unless you think like I do"
> "If you don't like Streamline, I suspect there's something wrong with you"
> "Here's Streamline, please use it. But if you don't like it, keep it to
> yourself"
>

Here is what I say:

"Here is Streamline. You can take a look at it. You can use it if it
helps you solve your problems. It is open source and licensed under
MIT terms. So I may support you, but only to a point. You are adult
and responsible: make your mind, take your responsibilities."

I may add: "some people will tell you that streamline is bad, that it
will prevent you from understanding the true nature of async
programming, etc. Fine, take this into account too, but make up your
mind by yourself, try things out, don't be dogmatic. Be curious. If
something new comes along, look at it, give it a try. What matters in
the end is that you be successful in your projects. Good luck!";

Bruno.
Re: My Humble Co-routine Proposal Liam 11/12/11 2:54 PM
Bruno, I found your insight on try/catch really insightful, thank you.
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 3:31 PM


On Saturday, November 12, 2011 2:17:35 PM UTC-8, Bruno Jouhier wrote:
No Marco, you got it wrong: I did not think one second about
streamline while writing this. I wrote similar posts on forums years
ago at a time were streamline was still associated with Reynolds
numbers in my mind. Was just a distraction!

I'm not sure I'm getting it wrong at all. You're a fan of your particular brand of try/catch programming. That's great. You think it's a great technique, proven by years of usage. And if people don't use it the way you've learned, it's probably because they don't understand it enough. Fast forward to today. You like node but found it difficult if not impossible to utilize your tried and true try/catch structure. So you created streamline at least in part because it allowed you to do so. So it stands to reason that you think everyone should use streamline or some abstraction that enables traditional try/catch (used properly of course). If they argue against it, they're probably a little misguided. I don't think I read to far into that.

If that is your stance, be bold man. You won't hurt my feelings :)

The only other explanation I have for your last post is that you're perfectly fine if I don't want to use try/catch. But before you accept that, you want me to prove that I'm competent enough to understand it as well as you do. Thus the subtle jab at my level of understanding. That's totally fine too. I respect your experience and have actually learned several things from you since you appeared on this list. I enjoy being schooled, believe it or not. I will respond in a bit, and you can draw whatever conclusions you like.


If you don't get the point, fine. If you have interesting follow up
comments please post them. We might get completely off tracks but this
thread has gone a bit wild anyway. As long as they remain polite and
professional, debates are ok.

Perhaps, I'm presuming things that aren't there. But if so, I would ask that you examine the way you make your points instead of assuming there's no way people can misunderstand you. Also, I'm all for threads going off track if it's a useful debate. Doesn't bother me in the slightest.

:Marco
 

Bruno

On Nov 12, 10:45 pm, Marco Rogers <marco....@gmail.com> wrote:
> > A bit off topic but when I see someone say "I'm not using try/catch
> > and I'm happy because I got rid of all these try/catch constructs", I
> > just feel that that person did not understand the language feature in
> > the first place.
>
> Ha! We have a winner!
>
> "If you don't like Streamline, I suspect there's something wrong with you"
>
> I'm gonna respond to this more seriously later, because I think it's
> interesting. Just couldn't resist.
>
> :Marco
Re: My Humble Co-routine Proposal Marco Rogers 11/12/11 3:43 PM
I think you just proved my point. You can't always control how your point comes across. I don't think my above paragraph has any negativity in it. Where do you think I went off the rails? If anyone else thinks I'm being disrespectful, please enlighten me. If you object to the term "weird". I'm sorry but that it is a purely subjective view of what happens when I look at streamline code. It's not meant to disparage or to suggest that anyone else but me feels that way. But it IS meant to illustrate that your arguments about signal to noise are also subjective and easily countered.

Let me take a step back for a second. I had a realization a while back that most of what I do here is take people's arguments and deconstruct them to try and decide if they are valid or not. The definition of "valid" is of course my own. But the definition of "invalid" usually just boils down to the fact that the things you assert as truths or even widely held truths are instead easily open to interpretation. Your arguments about "callbacks are hard to read, streamline is easier", are one such argument that I won't concede, because it doesn't ring true to me. That doesn't require you to change your mind, but it is one less thing you can use to convince me to change my mind. If that offends you, or you think it makes me naive or inexperienced, I'm cool with that man. Don't get me wrong. I'm just some fucking guy. Don't lose any sleep over what I think.

:Marco

Re: My Humble Co-routine Proposal Mark Stone 11/12/11 5:24 PM
Hey I just wanted to interject that as a relative newcomer I am loving
this discussion (even if I can only follow the details of about 40% of
it).

I suspect there are no absolute right or wrong answers, but I think
its healthy to have so vigorous a discussion, and I think it is a good
sign for the Node community that passions are so strong. And I have a
lot of confidence in Tim; anything he works on will likely be a net
gain for Node. People can make their own decisions about whether or
not to make use of his contributions.

I came to Node because JavaScript is something I've dabbled in for a
long time, and the notion of server side JavaScript was appealing.
Having done some reading before I plunged in, I also felt that
architecturally Node was a much better approach to web serving and web
services than, say, PHP, at least for the sort of code I was
interested in writing. I have stayed with Node, and continue to
struggle to wrap my head around it, because I have become a real
believer in functional programming.

Getting from "believer" to "adept practitioner" has been a real
struggle for me. I learned programming in an imperative world; my
first languages were Algol and Fortran, and my first big project in
college was to write an Algol compiler for the PDP-1170. Object-
oriented programming came along after I had stopped taking programming
classes, and something about it always struck me as awkward. Not just
hard to learn, but philosophically awkward. Functional programming is
even harder to learn, and I don't know if I will ever completely get
there, but it is elegant. Flat out elegant, no question.

As a beginner aspiring to code Node the right way and use it in the
ways that it is well suited, I don't want a bunch of synchronous
crutches and workarounds. I want Node, Node modules, and Node examples
to nudge me towards doing things the correct way. We beginners need
tools to do what we want, but we also need to be broken of bad habits.
Don't encourage us to continue those bad habits, and don't be shy
about telling us off when we're being idiots. Just because JavaScript
is easy and available, don't let us derail your community or take Node
in a direction it shouldn't be going in.

You guys all rock. Keep up the good discussion and the good work.

-Mark
--
Mark  Stone || mark....@gmail.com || 253-223-2159 || Technical
Project Manager, Adxstudio
Co-author  and Editor, "Open Sources", "Open Sources 2.0"
Alumnus: VA Linux systems, Wizards of the Coast, Microsoft (Server &
Tools Business)
This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/13/11 2:22 PM
I know of no open-source project that is a democracy.  Without central control it would grow to be a big mess.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Raynos 11/13/11 3:27 PM
I don't know where we are going here, but personally I would be heavily disappointed if you guys managed to push some kind of application level architecture decisions like mechanisms for flow control or synchronous code emulation into node.

Letting in those kind of decisions is a slippery slope downhill.

Now if we can get a low level construct into node that's actually useful, we may listen.

Other then that the whole flow control vs fibers&friends is a holy war that node core needs to stay completely out of.
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/13/11 3:44 PM
After all if we have here dictatorship then somebody should declare it explicitly. 

I don't think you have to read tea leaves to see that Ryan could win any argument that he cared to take on.  He developed it.

Java was developed by a company, not a person.  Each of Perl, Ruby, Python have dictators for life.  I assume Ryan is the same.

Re: [nodejs] Re: My Humble Co-routine Proposal Adam Crabtree 11/13/11 3:49 PM
I've already released an async try/catch with long stack traces a while back, called not unexpectedly "trycatch", http://github.com/crabdude/trycatch:

trycatch(function try() {
  fs.readFile('does-not-exist.js', function(err){
    if(err) throw err; // caught w/ long stack trace
  });
}, function catch(e) {
  console.log(e.stack);
});

I also went into the details a little bit more here on another thread (not realizing this one), "[nodejs] try/catch/throw (was: My Humble Co-routine Proposal)", https://groups.google.com/group/nodejs/browse_thread/thread/d444acb2c231ad45#msg_2acd175ed211b4c4.

I also rolled it into step and added error-coalescing and called it stepup, http://github.com/crabdude/stepup, to show it can be incorporated into existing flow-control libs for some nice results.

Cheers,
Adam Crabtree

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/13/11 7:18 PM
As a person who has been around node from the beginning and someone reasonably friendly with the core guys, I'll offer some input and advice. These are my own views only. I'll try not to speak for the core guys and I may be wrong about any of it.

First off, for newer people who may not be aware, Ryan is BDFL of Node.

https://groups.google.com/forum/#!searchin/nodejs/BDFL/nodejs/j7pb2JMInY8/oz5YPPhGHs4J

He is the primary author. He spent the long hours of development, shepherded it through people's initial negative reactions and shipped it. He has the last word about what goes in. I think it's great that the node community feels such passion about what happens with node, but as Mark suggests, it's not scalable for everyone to get changes in. You can't listen to everybody, you can't accept everything, and pure democracy without centralized vision doesn't work.

This doesn't mean that the core node guys don't welcome community. They love it and appreciate it as much as the rest of us do. Lots of ideas from the community have made it into node. But they also take their responsibility very seriously. Their responsibility is not only to continue to try to make node as awesome as possible for the people who use it. But it is also to accept the consequences for whatever node becomes. They are doing the hard work here. The rest of us are building our applications and we have our own pain points. We develop some solution and we think everybody should use it. But our first obstacle is to convince the core guys that it fits as a general solution that is good for as many people as possible.

Some people seem to think that doing this convincing is too hard right now. I personally feel that it is just as hard as it needs to be. It shouldn't be easy to convince people that something you built for you and your team is right for the thousands, or even tens of thousands of people who will eventually use node. You should have to fight tooth and nail as Vladimir suggests. And if you succeed, that's awesome! You convinced smart people who live and breath node, that they should take on the job of maintaining and supporting your addition for everybody for the foreseeable future. While you go off and do your next thing. I hope phrasing it like that adds a little more weight to what we're talking about so people see it clearly.

All of that being said, I think the node community has been working fairly well for a long while. But it has been a long time where node has been focused on api stability and windows support, to the exclusion of almost everything else. I'm hoping now that 0.6.0 is out, things will open back up a bit and we will see more improvements going in. So if you want to see changes, let me offer some suggestions.

- Commit to supporting your project. As I said, if it goes into node core, it's not you who will ultimately be responsible for it. Letting people know you will be around may go a long way.
- Become a reliable node contributor. People like to hawk their own wares, but if they've spent no time fixing other bugs, helping newcomers, writing docs, they are likely to get less consideration from the core guys. That's just the way it is. This is a community and it goes both ways. You need to be more than just "a guy on the mailing list with a good idea". (this is just my opinion, core guys may feel different). I try to practice what I preach. I argue a lot, but I never push hard for my own ideas because I don't have as much time to support them.
- Don't just show that your solution solves your problem. Show that it solves lots of other people's problems as well.
- Be prepared to argue. More importantly, be prepared to actually address the concerns people have with your solution. If the core guys raise a problem with your proposal, it's not enough to say just say "I don't agree that it's a problem". Because you'll still be on the outside. They are the gatekeepers, so you need to either convince them, or become a gatekeeper yourself.
- You probably won't get node to run on a fork of v8. There are several good reasons not to do this. But whether you agree or not, the decision is not yours. You are not the one that has to deal with the consequences. You can still argue about this too if you like, but if your argument doesn't include an offer to maintain the patches yourself in perpetuity, don't expect to get very far. This may feel like a shutdown, and it is. It is one of the few places where it makes sense to stand firm IMO. The one saving grace that you should always remember is that you are free to fork node whenever you like.

Like I said, I know the core guys. I'm privy to some of the discussions that lead to node changes. Ryan talked to me about domains weeks ago (I think it's a good idea, but I'm not completely sold. I want to see it). My point is that believe it or not, the core guys almost always go through all of the steps above (with each other and with others in the community) before they add something major to node. People here on the list don't always see it, because it happens informally. But there is due diligence with the things that get added.  I think it's only fair that people here do the same. Instead of pushing them, accusing them of not listening or being dictators (with the exception of Ryan. He is a dictator).

I don't have a dog in this fight. I think the community is working fairly well, and I trust the core guys. I hope I've at least given a constructive frame for the discussion.

:Marco

Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/13/11 7:19 PM
Is there supposed to be a link here Vladimir?
Re: [nodejs] Re: My Humble Co-routine Proposal verge 11/13/11 7:25 PM
+1

Michael McFall
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
Re: [nodejs] Re: My Humble Co-routine Proposal Jann Horn 11/13/11 11:31 PM
2011/11/12 Vladimir Perlov <vlad...@gmail.com>:

> Members of upper score group should:
> 2) Try to design solution that will resolve "nested callbacks" issue.

They did that in various ways. Look at streamlinejs, fibers and
caolans async. Oh, and at coco, it also has some kind of "streamlinejs
light".

> 3) Start discussion "dosomething(errCallback, successCallback) vs. func(err,
> reply)"

Huh? Did anyone ever propose to leave the "func(err, result)" pattern?
And also, that can't be changed anymore anyway.

> 4) Start to design community wide voting system

Why would we need a voting system? People can post +1/-1 comments on
github, that's enough. Also, we have a dictator, as people already
told you.

> 5)  Think about creating infrastructure as way to provide more power to
> community in general

We have IRC. We have the mailing list. We have NPM. Anyone can publish
or install whatever modules he wishes to publish or install on/from
npmjs.org, and if you think stuff in core should be changed, open a
Github issue or talk about it on this Mailinglist. And if you really
think it should be changed, make a pull request.

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/14/11 12:45 AM
> After all if we have here dictatorship then somebody should declare> it explicitly.

If you think anything hasn't been declared explicitly, then you
haven't been looking.


> Question is to whom belong the NodeJS?
> Who is going to decide the way NodeJS is going?

Ryan Dahl is the BDFL of Node.js.
http://en.wikipedia.org/wiki/BDFL#Examples_of_people_sometimes_referred_to_as_Benevolent_Dictator_For_Life

The license (also present on every single file in the source tree):
https://github.com/joyent/node/blob/master/LICENSE

The CLA that contributors sign: http://nodejs.org/cla.html

The trademark policy: http://nodejs.org/trademark-policy.pdf

In short, if you want to create a little commune that decides lines of
code by picking tea leaves out of a hat, that's fine.  Fork the code
and go do that.  Pick a new logo and name, and follow the rules set
forth by the MIT license.

The governance of the Node.js project is not likely to change soon.


> You are investing your time to work with NodeJS, and usually every investor
> would like to have some influence over the system.

If someone gives you software for free, that doesn't imply that you
have a right to tell them what to do.

Bug reports, tests, docs, patches, modules, questions, answers, and
discussion are all welcome, of course :)  The best way to gain
influence in the node project is by helping out.

> For some things it's not enough to be located in user land :)

I wrote this for you a while back:
https://github.com/joyent/node/wiki/node-core-vs-userland

There are very few things that are not better in userland.

Re: My Humble Co-routine Proposal Bruno Jouhier 11/14/11 6:02 AM
Hi Marco,

These sync/async harsh debates may give the impression that there is a
severe problem. There is obviously a different viewpoint coming from
some people like me, who are pushing node in directions that the core
team may not have anticipated (building boring traditional apps rather
than innovating with event driven architectures).

But I don't follow Vladimir at all here. My goal is not to rank
people, or make a putsch, or whatever. I think that it is becoming
clear that application guys like a number of us need a more
"classical" model to work with: sync style, with try/catch etc. And we
already have solutions for this: things like fibers and streamline.
Moreover these solutions are completely decoupled from node core. We
don't need changes to node core, we can consume standard node modules
(core one and non-core ones) and the modules that we create with our
tools can be consumed like regular modules by people who don't use our
tools. We may be using heretic tools or libraries internally but we
are not imposing them onto others. So rather than argue in sterile
ways around those tools, we should accept that we are doing different
things, that each side probably knows better what its problems are and
how to handle them best. We should not be wasting our time bashing on
others for using a different approach.

I also think that it is an opportunity for node to have people like us
who push the technology in other directions, as long as we don't
sidetrack the project and we let the core team explore what they need
to explore to satisfy the needs of new architectures. I think that
there is also a big opportunity for node in more traditional apps,
simply because Javascript is a great language, the async model is
great, performance is amazing, etc.

One of the great things about node (and Javascript BTW) is it very low
level of coupling. It is what allowed us (Marcel, myself) to create
pseudo-sync solutions that interoperate completely seamlessly with
node code, without requiring any change from core.

I think that this is a great success, even if we keep arguing. THE
DECOUPLED MODEL WORKS AND WE NEED TO KEEP IT THIS WAY. So the last
thing I want to see is all sorts of things being pushed into core:
Node core does not need streamline (BTW this would be as silly as
saying it needs CoffeeScript), it does not need fibers, it does not
need any built-in routing framework, it does not need built-in drivers
for all sorts of databases. It needs to continue doing what it does
today, with APIs that are as simple as possible, with implementations
that are as efficient as possible, etc.

The great thing about the decoupled model is that it enables a true
Darwinian process. There will be great modules that will survive and
prosper, and weak ones that will go away. So, instead of getting a
bloated aggregate of mediocre modules that have been blessed by the
core group after all sorts of lobbying efforts, we will get a healthy
competition that will give us best of breeds.

One very important thing is to have a very small and very simple set
of standards that everyone follows and that allow all the modules to
live together. Today, we have modules, core APIs, the C++ extension
API, the callback(err, result) pattern and the Emitter pattern.
Personally, I am perfectly happy with this set, I can move forwards
with it and with a small number of quality modules developed outside
of core.
So, instead of rushing new things into core, it is more important to
solidify this. If someone develops a modules with fibers, he won't be
creating fragmentation, as long as his external API follows the
guidelines. If on the other hand, someone starts to monkey patch the
core APIs, or expose async APIs that have very different signatures,
then we will have fragmentation.

Also, I would like the debate between the "pseudo-sync guys" and the
core team to get more relaxed. We are on the same boat and we have a
vested interest to succeed together. But for this (and sorry if I
sound a bit patronizing here), I think that we need consideration both
ways. We (the guys outside of core) are your "customers", we are not
"stupid" and you need to listen to your customers, not try to impose
your own dogma upon them. We have the right to design our solutions
with tools that are different from yours, as long as we adhere to and
promote the standards that make it possible for modules of different
origin to work together.

Bruno

On Nov 14, 4:18 am, Marco Rogers <marco.rog...@gmail.com> wrote:
> As a person who has been around node from the beginning and someone
> reasonably friendly with the core guys, I'll offer some input and advice.
> These are my own views only. I'll try not to speak for the core guys and I
> may be wrong about any of it.
>
> First off, for newer people who may not be aware, Ryan is BDFL of Node.
>
> https://groups.google.com/forum/#!searchin/nodejs/BDFL/nodejs/j7pb2JM...http://en.wikipedia.org/wiki/Benevolent_Dictator_For_Life
Re: [nodejs] Re: My Humble Co-routine Proposal Raynos 11/14/11 6:03 AM
Isaacs +1
This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/14/11 9:27 AM
Vladimir, please stop.

Nobody has the right to tell someone else how he or she should run
hier/her free project differently or else! You got the right to
suggest and reason, if its refused you stilll got the right hit "fork
this repository" in github and do your own variant. This is quite a
power the license gives you! You only get this with free software! Its
not like a properiatery software where when you don't get the
manufacturer to do you biddings your out in the dark. The Linux kernel
for example has a rich history of forks, some of them got even merged
back into "vanillla" as they call Linus` version later on. Some stay a
different flavor forever.Thats how it runs.

Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/14/11 9:45 AM
Agreed.

This isn't a political system where the citizens chose government and can expect redress. This is a project that welcomes community, but is led in a particular direction by a core team. If a bunch of erlang people showed up and started yelling "hey make your project more like ours because I'm a customer", is Ryan obligated to do so just because there's a lot of them? No, I don't think so. It's a little like someone having a party and inviting you over. And you come in, put your muddy boots up on the table and say "thanks, but you should clean up around here and move this furniture around so it suits me better".

Also, Vladimir, several people have given you lots of suggestions on how to gain more influence on what happens with node. There is a path, but it sounds like you don't like that path. Instead you want to create your own by trying to gather a crowd to back you. Not cool bro.

:Marco
Marco Rogers
marco....@gmail.com | https://twitter.com/polotek

Life is ten percent what happens to you and ninety percent how you respond to it.
- Lou Holtz
Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/14/11 9:48 AM
+1 to most of the stuff here. Thanks Bruno.

:Marco



--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en



--
Marco Rogers
marco....@gmail.com | https://twitter.com/polotek

Life is ten percent what happens to you and ninety percent how you respond to it.
- Lou Holtz
This message has been hidden because it was flagged for abuse.
Re: My Humble Co-routine Proposal Bruno Jouhier 11/14/11 10:00 AM
Finally breaking the ice wall! Thanks.


On Nov 14, 6:48 pm, Marco Rogers <marco.rog...@gmail.com> wrote:
> +1 to most of the stuff here. Thanks Bruno.
>
> :Marco
>
>
>
>
>
>
>
> ...
>
> read more »
This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal sotonin 11/14/11 10:28 AM
Vladimir it's time to drop it. Your voting system is not going to happen, time to move on. You can argue all day until you are blue in the face, that's not how it works though. Ryan is the sole decision maker on things of this nature with the nodejs software.
Re: [nodejs] Re: My Humble Co-routine Proposal Tim Caswell 11/14/11 10:30 AM
Marco, Bruno, Axel, Marcel,

Awesome stuff! Thanks for keeping the conversation friendly.  I don't
care what other say, not *all* geeks like to argue.  Personally I
can't stand it unless it's necessary which is why I usually stay out
of these threads.  Remember than node itself is still a very small toy
to the majority of 9-5 developers in the world and that's our real
challenge.  We should avoid infighting as much as possible or it will
destroy the value of node as a platform.

-Tim Caswell

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Raynos 11/14/11 10:32 AM


On Mon, Nov 14, 2011 at 5:50 PM, Vladimir Perlov <vlad...@gmail.com> wrote:
>But I don't follow Vladimir at all here. My goal is not to rank
>people, or make a putsch, or whatever. I think that it is becoming

I'm not going to rank people.
NodeJS by nature is shareable/virtual kind of property.
All of us participating in creating this property.

nodejs is a project. It's a piece of code. It has authors. Those authors create the property. All we do is give feedback and suggest input
 
It's natural to share power and eventually profit from this property.
Of course to do it properly you need appropriate business infrastructure.
Definitely having email list is not enough :)

A mailing list is sufficient for what we need. Which is feedback and input
 
By the way it's really basic stuff I'm talking about.


>Moreover these solutions are completely decoupled from node core. We
>don't need changes to node core, we can consume standard node modules
I disagree on it. For example I would like to make from Javascript language some kind of data structure.
Data structure is good for transformation, language is not.
Streamline library is a good start point in this direction.
I could be completely wrong on this, but I have the right to have a real chance to promote this idea to the core using voting system.
Every knows that competition of ideas is good thing to have!
So I'm trying to introduce some competition to the core team decisions.

You can introduce competition to the core team by forking node. And making your own project, and then show us how this project is better.
 
Is it too much to ask?

Yes
 

>are not imposing them onto others. So rather than argue in sterile
>ways around those tools, we should accept that we are doing different
>things, that each side probably knows better what its problems are and
>how to handle them best. We should not be wasting our time bashing on
>others for using a different approach.
You should understand better then me that some things must go up.
And I'm not talking about some driver for database.

It's painful to see that even people like you believe that unlimited power is fine thing to have.

>I also think that it is an opportunity for node to have people like us
>who push the technology in other directions, as long as we don't
>sidetrack the project and we let the core team explore what they need
>to explore to satisfy the needs of new architectures. I think that

Sorry Bruno, your technology/approach is going to undefined land :)
Seems only I see that some projects/ideas are dying.

Of course it is. Bruno himself says he doesn't want it in core. It doesn't belong in core
 

>sound a bit patronizing here), I think that we need consideration both
>ways. We (the guys outside of core) are your "customers", we are not
>"stupid" and you need to listen to your customers, not try to impose
>your own dogma upon them. We have the right to design our solutions
>with tools that are different from yours, as long as we adhere to and
>promote the standards that make it possible for modules of different
>origin to work together.

We are not customers here!
Any customer has explicit right to request some changes!

And you have that right. Don't confuse the right to request with the right to demand.

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/14/11 10:32 AM
Vladimir,

You're so far off on so many points, I don't even know where to begin.
 Have you even tried to investigate any of your claims?


> You obviously don't understand very important thing.> The nodeJS is not a project.> It's ecosystem.> Do you understand difference between someone project and ecosystem?

Do you?

If you want to be a part of the NodeJS "ecosystem", then contribute to
it.  Write useful programs, and share them.  If you solve peoples'
problems, then they will use your solutions.

If you're talking about node.js the software project, ie "node-core",
then yes, you are talking about a specific software project, with a
specific owner, and not any kind of "ecosystem".


> I'm investing my time to promote and build this property.


How exactly are you investing your time to promote and build node?


> Keep at least in mind that this property including not just core modules.
> I'm telling you this is virtual kind of property.


Node programs published on npm and github are the property of their
respective authors.


> We don't MS SQL driver in this land?
> Why I'd like to ask?
> Because the core team didn't ask mister Microsoft to help us on this.


Microsoft is helping tremendously with the node project.


> I'm investing my time in this ecosystem.


I don't even know what that means.  In what way is your time invested
in this ecosystem?  It sounds like you really expect some measure of
control over node-core, but you don't seem very interested in doing
any work.

(I suspect I'm being trolled.)

Re: My Humble Co-routine Proposal Bruno Jouhier 11/14/11 10:33 AM
> >Moreover these solutions are completely decoupled from node core. We
> >don't need changes to node core, we can consume standard node modules
>
> I disagree on it. For example I would like to make from Javascript language
> some kind of data structure.
> Data structure is good for transformation, language is not.
> Streamline library is a good start point in this direction.
> I could be completely wrong on this, but I have the right to have a real
> chance to promote this idea to the core using voting system.
> Every knows that competition of ideas is good thing to have!
> So I'm trying to introduce some competition to the core team decisions.
> Is it too much to ask?

Vladimir,

You may need this and this may be a great feature but this may also be
a feature that 80% of the people don't care about because they are
doing something completely different than what you do. And similarly,
there may be something that some other guy would love to have in core
(and that makes sense too - support for some protocol for example) but
that you don't care about (you don't use this protocol). If we let
everyone push things into core we are going to end up with another
bloated, slow, mediocre framework. There seems to be a consensus that
no one wants this.

Also, Ryan had the initial idea of node.js. He and his teams have
produced great things so far. Even if we don't always agree on
everything, we have to acknowledge this. So I'm fine with having Ryan
as a dictator. If I don't agree with something, or have something to
suggest, I will speak up but in the end, Ryan has to decide and it is
good for all of us to have someone with a vision in command. If, for
whatever reason, Ryan and his team suddenly start to go into weird
territories, we will always have the option to fork.

> You should understand better then me that some things must go up.
> And I'm not talking about some driver for database.
>
> It's painful to see that even people like you believe that unlimited power
> is fine thing to have.
>

I'm interested in debates and I won't give up on some of my technical
convictions so easily (as you may have guessed :-) but I'm also a
pragmatic. There is nothing in the current core that's preventing me
to move forwards on my project. So why would I try to break what works
so well for me? Things would be very different if the core team was
preventing me from using streamline and forcing me to write everything
with callbacks (dunno how they could do this). But that's not what's
happening. Come on!

Bruno
Re: [nodejs] Re: My Humble Co-routine Proposal Raynos 11/14/11 10:34 AM


There is no MSSQL driver because no-one cares. If someone really wanted to talk to MSSQL with node they would have written a driver.

The fact it doesn't exist means we simply don't need to talk to MSSQL.
 
Why I'd like to ask?
Because the core team didn't ask mister Microsoft to help us on this.
It's simple thing to do.
Why I don't have the way to request to make this move from the core team?

It's called stop wasting the core's teams time on this and write the driver yourself. Then publish it as an open source module.

This is called contributing to the community, maybe you should try it.

Re: [nodejs] Re: My Humble Co-routine Proposal Raynos 11/14/11 10:35 AM
It's not cool to have dictator and follow his way like children.
Ryan should understand that we need a new kind of political system here or our real potential will be pretty limited.


"we need". Who is this we? Is this we, you? I don't think we need a political system. 
Re: [nodejs] Re: My Humble Co-routine Proposal Marco Rogers 11/14/11 10:36 AM
Sotonin. That's not correct either. Ryan is not the *sole* decision maker. He is the *final* decision maker. There is  a middle ground here. Vladimir doesn't think it's valid I guess. But just because he is at one extreme, we shouldn't swing to the other.

:Marco
Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/14/11 10:41 AM
Vladimir,
It seems no one wants to introduce voting into the node project, except you.
You've been out-voted.  We like our current governance.
Re: [nodejs] Re: My Humble Co-routine Proposal sotonin 11/14/11 10:56 AM
Ah. yes that's what i meant. What it boils down to at the end of the day is Ryan can overturn anything. Thanks for the correction
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/14/11 11:07 AM
They will die because they don't have enough time to lead endless debates. 

I find it humorous to think someone would choose a software package because of these debates.  I'm personally contributing just to avoid work.

In regards to Ryan being a dictator, you can vote with your feet.  Switch from node to something that is more to your liking.
 
Re: [nodejs] Re: My Humble Co-routine Proposal Ryan Dahl 11/14/11 12:14 PM
Dear Nerds,

Node does not modify the JavaScript runtime. This is for the ECMA
committee and V8 to decide. If they add coroutines (which they won't)
then we will include it. If they add generators we will include them.
Node does not overstep its bounds. We provide bindings to system
functions in order to create network programs - we do not determine
the language. No need to discuss this further.

Ryan

This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/14/11 1:15 PM
I don't think the length of these arguments are an indication that there is a large problem because the same small group of people have been making the same argument for months. Nothing has changed but it's no different or worse and I don't see a huge amount of growth in the people making the arguments only in their tolerance for writing longer and long posts and threads.

PS. I'm sick today so I'm likely pissy.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/14/11 1:31 PM
You need to find some patience. We *just* wrapped up porting node to windows it'll be a little while before something like an MSSQL driver has time and priority.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/14/11 2:07 PM
> You obviously do not understand very basic economic things.

I have a deed lying around somewhere that says "master of social
economics". By itself it doesn't mean all that much, I still don't
understand the financial crisis at all, but at some point I must at
least have grabbed up some basics.

> We are not customers here!
> Any customer has explicit right to request some changes!
> You have the right to say "thank you" to the core team :)

I'll tell you your rights: Permission is hereby granted, free of
charge, to any person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use, copy,
modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished
to do so ...

This if very hefty and powerful! But it does not say "to tell the
author(s) what do next". Recently I had a chance to listen to Richard
Stallmans talk (I guess he does the same over and over again :-), but
it was really insightful to focus on "free software" not as free as in
beer (which it happens to be also), but as software free in the sense
that does not take rights from you. You do not get that one with
properitary closed source. The author(s) excerts no control
fundamental over you, having all the source you still can change it
anytime, or you can have someone change it for you.

Central actors that are called here "BDFL" are central because
everybody works with them. While they sure can be eccentricity to a
certain level, there is a limit to it, when too much people want to do
it differently they just fork the projects due to the license and work
toward someone else. Thats how the ecosystem works. I have seen closed
online games, where the owners really enjoyed excerting power, etc. I
would be on your side there, but really this isn't the case here.

If you really need a MS-SQL, and no one has done it yet, you either a)
do it yourself, b) wait until someone might do it c) pay someone to do
it, or just use another database. Thats how the ecosystem works.

Can you now please stop hijacking Tims insightful discussions with
your insurgent energies, and direct those instead to all the real
injustice that is on this world?

This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/14/11 2:12 PM
(taking this off list, as you're getting a bit personal)
On Mon, Nov 14, 2011 at 13:29, Vladimir Perlov <vlad...@gmail.com>
wrote:> Sometimes you have pretty weak arguments.
Such as?

> "Microsoft is helping tremendously with the node project."> You think that using the word "tremendously" will help me understand why we> don't have support from Microsoft in implementing MY SQL driver> implementation.
You implied that Microsoft isn't interested in helping with node.
They've helped with money, time, and resources.  This was the line I
was responding to:

> Because the core team didn't ask mister Microsoft to help us on this.
I didn't realize that the bug up your shorts was the lack of a MS SQL
driver.  I thought you were talking about NodeJS.
If you want a MSSQL driver, write one.  If you want Microsoft's help,
ask them.  Why would you think that anyone else ought to do this for
you?  Are you a child?
> One day ago you were honest person in my eyes.> As honest person you should say "I don't care about having MSSQL driver in> NodeJS land!".> And you are not saying it :)
On the contrary, I would be delighted to see a MSSQL driver in NodeJS
land.  That's why I repeatedly encouraged you to write one.  I love
seeing new things added to the node ecosystem.
I care a lot more about a bunch of other things.  Focus is saying "No"
to a lot of things you care about.  Having priorities means making
sacrifices and acknowledging the realities of finite time and
attention.

> I hope at least somebody will understand why you refused to do it.
You're so confusing.  Why would anyone even have to "refuse" such a
thing?  What ever gave you the impression that anyone ought to do it?

> Isaac, you should read your very recent blog, at least some parts of it :)
Which parts?

> I have request that somebody from the top of NodeJS ask Microsoft to help with MS > SQL driver implementation.> How long should I wait for response?
Here's your response: Do it yourself, or hire someone willing to do it for you.

>>There is no MSSQL driver because no-one cares. If someone really wanted to
>> talk to MSSQL with node they would have written a driver.
> Just recently I have participated in thread where people were looking for MS
> SQL driver.
> So I'm not alone who ask for it.

Recently I was one of a lot of people wanting a million dollars.

Therefor, give me a million dollars.

Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/14/11 2:12 PM
On Mon, Nov 14, 2011 at 14:12, Isaac Schlueter <i...@izs.me> wrote:
> (taking this off list, as you're getting a bit personal)

Ha, guess not.  Whoops. *^_^*

This message has been hidden because it was flagged for abuse.
Re: My Humble Co-routine Proposal Liam 11/14/11 2:34 PM
It's kind of amusing to see folks get all aroused by a few none-too-
carefully considered suggestions.

If you're tired of coding and need some R&R, may I suggest http://nytimes.com
or http://cuteoverload.com
This message has been hidden because it was flagged for abuse.
RE: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/14/11 8:23 PM
Please don't follow the jcp

Sent from my Windows Phone

From: Mark Hahn
Sent: 11/13/2011 2:22 PM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal

I know of no open-source project that is a democracy.  Without central control it would grow to be a big mess.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Oleg Podsechin 11/14/11 10:33 PM
Hi Mikeal, 

I'm sure there are also those that wholeheartedly agree with this "small group of people" but don't post here. 

I for one am glad there is somebody with as much free time and as eloquent as Bruno who can get the point across in a civil manner while I get on with building common-node and higher level synchronous packages for production use.

Regards,
Oleg

PS. I'm in SF later this week, would be nice to meet & chat about what you're working on
Re: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/14/11 11:23 PM
HI Vlad

I work at Microsoft on our node efforts. If we hear folks from the community asking for SQL support we will definitely look into it. I'd echo what everyone else said though, this is OSS, nothing stops anyone from putting together a sql driver and releasing it via npm.

Cheers
Glenn
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/14/11 11:56 PM
> I could provide you many other examples that will prove my point regarding weakness> of open software model.
No, Its not about "open source", its about freedom:
http://www.youtube.com/watch?v=4m3jk3jKzH0

> I'd like to have possibility to pay for MS SQL driver implementation with virtual currency :)

Well if you want to live in your radical neoclassic mindset, do it
with "real" money instead, its called a labor contract. A few
(wo)man-months should do it.

C++ bindings are easy (was: My Humble Co-routine Proposal) Liam 11/15/11 3:02 AM
On Nov 14, 11:56 pm, Axel Kittenberger <axk...@gmail.com> wrote:
> A few (wo)man-months should do it.

Writing a binding is easy if you're familiar with the API you want to
bind, a matter of days or weeks. The V8 and libeio APIs are pretty
easy to learn.

V8 API docs: http://izs.me/v8-docs/hierarchy.html

I've just posted a first draft of a binding for the Xapian indexer,
which demonstrates a lot of binding-code techniques:
https://github.com/networkimprov/node-xapian

This SQLite binding demonstrates use of Buffer, and of ev_async to hit
a callback repeatedly from an async routine (however, it's not a good
guide to use of Local<type>) https://github.com/developmentseed/node-sqlite3
Re: [nodejs] C++ bindings are easy (was: My Humble Co-routine Proposal) Axel Kittenberger 11/15/11 3:08 AM
Indeed, I was just on purpose giving an unprecise and long estimation
since I didn't want to mess with anyone work who actually does it.
"But that guy on the internet said, it can be done in 2 weeks".

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Mike Pilsbury 11/15/11 2:29 PM
I've been working on such a project (https://github.com/pekim/tedious). It's not quite ready yet, hence no big announcement to date. In fact there's loads more to do.

However it is definitely inching towards being usable for simple queries (and other statements) with SQL Server.

This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Mark Hahn 11/15/11 3:09 PM
NodeJS ecosystem doesn't have one easy and common way to vote or ask for something. 

Yes it does.  Use this mailing list.
 
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/15/11 7:20 PM

On Nov 15, 2011, at November 15, 20116:50 PM, Vladimir Perlov wrote:

> >No, Its not about "open source", its about freedom:
> To have more freedom it's time to kill state's monopoly to produce and control money.

While I might agree with this sentiment in the abstract it doesn't accurately reflect the reality of nodejs.

node.js copyright is owned by Joyent and licensed publicly under the MIT license. That means that anyone is free to do, literally, whatever they want with it and derive whatever monetary value they like without restriction.

If it were GPL or AGPL the situation would be different, then restrictions would apply to people using the public license that require a new license to avert.

Given the situation I can't comprehend the intention of your comment.

Re: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/15/11 7:45 PM
Well soon (very soon) we'll have a site where you can vote in the issue tracker.

Glenn
Re: [nodejs] Re: My Humble Co-routine Proposal Glenn Block 11/15/11 7:46 PM
Yeah, that's definitely one way!

On Tue, Nov 15, 2011 at 3:09 PM, Mark Hahn <ma...@hahnca.com> wrote:
NodeJS ecosystem doesn't have one easy and common way to vote or ask for something. 

Yes it does.  Use this mailing list.
 

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
This message has been hidden because it was flagged for abuse.
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/15/11 11:51 PM
n Wed, Nov 16, 2011 at 3:50 AM, Vladimir Perlov <vlad...@gmail.com> wrote:
>>No, Its not about "open source", its about freedom:
> To have more freedom it's time to kill state's monopoly to produce and
> control money.

Did you even watch a few minutes of the RMS video I linked, or did you
just post something that came into your mind in a minute. There is NO
monopoly. And thats what free software is about.

Why can't you just _ask_ devs if someone wants to do a MSSQL driver,
or _ask_ people how many would be interested. Why you force down your
demanding "voting" nonsenes. Its called manners.

Re: [nodejs] Re: My Humble Co-routine Proposal Brandon Benvie 11/18/11 11:40 PM
I just gotta perk up and make a point here. JavaScript hasn't been owned by anyone since the minute Microsoft copied it bug for bug and made JScript. Since then anything that survived did so because of natural selection. There's been numerous server-side/non-browser implementations over the last decade, many of which still exist. Node has been around for less than 3 years and has captured mind-shared that none of them ever did.

Node has gotten popular as an implementation of a JavaScript host for a couple fundamental reasons: it implements an extremely useful set of base JS Host level functionality and it's dead easy to get it running on almost anything. Pretty much the same reasons that JS became popular in browsers and the same fundamental reason reason that every popular browser has gained popularity.

If you're looking for a vote, or a justification for a given party's right to govern, you're looking in the wrong place. In this world, the web world, that vote is done in real-time and the votes are cast via the willing dedication of nerds who decide to use, contribute, or (directly/indirectly) help to popularize a given project. Nothing else matters in the end except a whole asston of money.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/19/11 12:09 AM
The JavaScript specification is maintained at ECMA by TC-39, participants include Google, Mozilla, Microsoft and Yahoo.

While very little happened for many years the most recent specification, ES5, has been widely implemented in all browsers and is the version currently in node.js. The next version, Harmony, also contains features that are now implemented in v8 and available in node.js.

node.js does not define a language, that is one of the stated goals of the project, we're not writing a language.

-Mikeal

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Brandon Benvie 11/19/11 1:50 AM
For the most part I agree with what you said Mikeal. V8 implements the specification and also provides tentative non-final implementations of future spec items. It is important to note that a stated goal and large factor in TC-39's current operating procedure is in converting de facto standards into de jure standards, and like it or not Node has become an important implementation in influencing what those de facto standards may be in two or five years, and likely will become an ever increasing factor commensurate with its growing user base. By no stretch does that directly require anything of Node's development team and it would be unfair and unenforceable to suddenly say "YOU ARE NOW THE GATEKEEPER OF X". But it's also not accurate to disclaim the influence on the future.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/19/11 9:11 AM
Correct, but there is a line.

If the language wants to add binary or streams then we have a lot of domain expertise and can show what has worked for us. But, we implemented those without modifying the language and do not plan on modifying the language ever in the future.

-Mikeal

On Nov 19, 2011, at November 19, 20111:50 AM, Brandon Benvie wrote:

For the most part I agree with what you said Mikeal. V8 implements the specification and also provides tentative non-final implementations of future spec items. It is important to note that a stated goal and large factor in TC-39's current operating procedure is in converting de facto standards into de jure standards, and like it or not Node has become an important implementation in influencing what those de facto standards may be in two or five years, and likely will become an ever increasing factor commensurate with its growing user base. By no stretch does that directly require anything of Node's development team and it would be unfair and unenforceable to suddenly say "YOU ARE NOW THE GATEKEEPER OF X". But it's also not accurate to disclaim the influence on the future.

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/19/11 9:23 AM
Fibers also do not modify the language. Just saying. At least not any
less than isolates, domains and the such.
Re: [nodejs] Re: My Humble Co-routine Proposal Mikeal Rogers 11/19/11 9:32 AM
wasn't implying that they did, just trying to limit the scope of debate.
Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/19/11 11:19 AM


On Sat, Nov 19, 2011 at 12:23 PM, Axel Kittenberger <axk...@gmail.com> wrote:
Fibers also do not modify the language. Just saying. At least not any
less than isolates, domains and the such.


Yes. They do. I will spare everyone the rehash but it's important that you don't gloss over this fact. I'm not saying this makes them evil, but you cannot handwave away the very real change in runtime semantics -- and very real hazard it introduces.

On the other hand, domains and isolates have no impact on the runtime semantics of the language (as far as I know). Of course, c extensions and preprocessors can pretty much do whatever the hell they want, changing language semantics in various ways. Node allows this -- as it should -- but as a community we have to understand the implications -- and weigh them honestly.
Re: [nodejs] Re: My Humble Co-routine Proposal Isaac Schlueter 11/19/11 6:50 PM
Fibers absolutely do modify the language, by adding "yield" semantics
that don't exist otherwise, and are impossible to do using JavaScript
only.  (Iterators are similar, but only a spidermonkeyism, afaik.)

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>

Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/19/11 7:05 PM


On Sat, Nov 19, 2011 at 9:50 PM, Isaac Schlueter <i...@izs.me> wrote:
Fibers absolutely do modify the language, by adding "yield" semantics
that don't exist otherwise, and are impossible to do using JavaScript
only.  (Iterators are similar, but only a spidermonkeyism, afaik.)


Just to clarify: iterators (as they exist in SM and es-next) don't introduce yield semantics -- they're just a convention, a protocol dance specifically followed by for..of, but one that can otherwise be coded by hand in js (albeit more painfully, especially w/ respect to the handling StopIteration exception).

My guess is you're referring to generators -- these happen to let you write nice lazy iterators (among other things), but like fibers and coros, also have yield semantics that don't otherwise exist in the language. The difference between generators and fibers/coros is that generators require any yield to be explicitly labeled (so called "single frame continuations"), so no function could ever yield out from under you. This is the hazard of fibers.

Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/20/11 2:14 AM
> The difference between generators and fibers/coros is that generators require> any yield to be explicitly labeled (so called "single frame continuations"), so
> no function could ever yield out from under you. This is the hazard of> fibers.

Tim has shown it in his proposal, I have shown it with
node-green-light, this is not true, if done with a proper wrapper to
fibers. You get a yield function handle when making a new stack, a
function that has not that handle been passed down cannot yield under
you. We can ague about performance of switching the stack or general
matters of taste, etc, but that argument is just not true if done
right.

Anyway fibers do not change the V8 itself, not the parser, not the JIT etc.

Re: My Humble Co-routine Proposal Bruno Jouhier 11/20/11 3:46 AM
Hi Dean,

This is an interesting question: does streamline break JS semantics?

Let us consider the following:

transform(function() {
        console.log("Hello world");
})();

function transform(fn) {
        var s = fn.toString().replace("Hello", "Goodbye");
        return new Function(s.substring(s.indexOf('{')));
}


Question: did we break Javascript here?

This snippet only uses legit JS features AFAIK (unless you impose
strict mode). So, from a "formal" standpoint, it does *not* break JS.

Now, if you replace transform by streamline.transform, does this
change the answer? NO. So, at least from a formal standpoint,
streamline does *not* break JS.

Now, if you "judge" this from a "semantic" standpoint, you will
probably judge that the example above "breaks" JS. And then you'll
probably conclude that "JS is broken", simply because JS allows eval.

But, even if you ban eval, by imposing strict mode for example, you
can still emulate it, for example with Narcissus. So, strict mode does
not really fix the problem, it only makes it more difficult (and very
inefficient) to "break" JS.

So, what's really breaking JS above is not eval, it is the more
innocuous fn.toString() !!!

My conclusion: any Turing complete language that supports
fn.toString() is broken!

Does this make any sense, or am I completely off?

Bruno

Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/20/11 4:02 AM


No, fibers STILL change the runtime semantics, just as generators do. It's just that you can get cute and hack your way into making yielding explicit. It still break's RTC and thus it's not javascript, but a new beast. That's all fine, just don't mislead people -- it doesn't do any of us any favors.
Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/20/11 4:38 AM


On Sun, Nov 20, 2011 at 6:46 AM, Bruno Jouhier <bjou...@gmail.com> wrote:
Hi Dean,

This is an interesting question: does streamline break JS semantics?
 

It's not js, and I don't remember you claiming it to be.

 
Let us consider the following:

transform(function() {
       console.log("Hello world");
})();

function transform(fn) {
       var s = fn.toString().replace("Hello", "Goodbye");
       return new Function(s.substring(s.indexOf('{')));
}


Question: did we break Javascript here?
 

Just because you didn't spell out the word "eval" doesn't mean you're not using it. Does that mean that anything you can run with emscripten is javascript? ;). 

You can literally run ANYTHING if you can preprocess and eval. That's not what we're talking about.

 
This snippet only uses legit JS features AFAIK (unless you impose
strict mode). So, from a "formal" standpoint, it does *not* break JS.
 

It's not about "breaking" javascript -- it's about violating invariants assured by javascript about its runtime environment. When you use eval all bets are off, and we all know it.

 
Now, if you replace transform by streamline.transform, does this
change the answer? NO. So, at least from a formal standpoint,
streamline does *not* break JS.
 

You use the word "formal" very informally -- the above statement is a fallacy.

 
Now, if you "judge" this from a "semantic" standpoint, you will
probably judge that the example above "breaks" JS. And then you'll
probably conclude that "JS is broken", simply because JS allows eval.
 

Quite the contrary -- this is explicitly allowed by the language (as you know).

 
But, even if you ban eval, by imposing strict mode for example, you
can still emulate it, for example with Narcissus. So, strict mode does
not really fix the problem, it only makes it more difficult (and very
inefficient) to "break" JS.
 

Regardless of your turing tarpit handwaving, run-to-completion semantics are an invariant promised by the language. When you bypass it but try and tell people otherwise you're lying. Plain and simple.

 
So, what's really breaking JS above is not eval, it is the more
innocuous fn.toString() !!!

My conclusion: any Turing complete language that supports
fn.toString() is broken!
 

This is a non-sequitur.

 
Does this make any sense, or am I completely off?


If it's a reply to the point I was trying to make, I'd say it's pretty far off the mark. The language sets forth a set of rules by which the language is interpreted. Of course you could write your own interpreter. Or you could preprocess any language into javascript and eval. Whatever language one codes in, it's important that they comprehend the rules of evaluation. A javascript programmer has certain expectations -- misleading them in any way does them a disservice.
Re: [nodejs] Re: My Humble Co-routine Proposal Axel Kittenberger 11/20/11 6:29 AM
> This is an interesting question: does streamline break JS semantics?

Streamline is IMHO a domain specific language (extension) (DSL) that
like any DSL makes a writing code for a particular domain a lot
easier, because it caters for the logics in that domain. In case of
Streamline its writing async code with getting less twists your head.
DSL's are a great idea for complicated stuff. They allow you to write
the logics by caring less about the implementation details. Look at
the ruby community, its full of DSLs. What many of them do there is
catch a parser error, and translate it do smoething meaningful
instead. I'd say, if you have the time to write a DSL for a more
complicated problem you got, instead of hacking something away just
the get it off the list, the DSL might really be worth a
consideration.

Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/20/11 7:17 AM


I agree. DSLs are lovely in languages with hygienic macros and good homoiconic properties. As it stands, js has neither, thus this makes a lot of js developers uncomfortable, and for good reason. 

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

Re: My Humble Co-routine Proposal Bruno Jouhier 11/20/11 10:17 AM
> It's not about "breaking" javascript -- it's about violating invariants
> assured by javascript about its runtime environment. When you use eval all
> bets are off, and we all know it.

But banning eval (and new Function(str), of course) isn't enough. You
can use Narcissus.evaluate(str) instead. If you do, there won't be any
calls to eval or new Function(str) under the hood and the bets will
still be off!

My point was that the problem is not just with eval, it is also with
Function.prototype.toString(). This function is what makes it possible
to write a transform(fn)() that executes a modified version of fn (in
which fn's invariants may not hold). I know that this is not strictly
"executing fn" but still, I find it rather disturbing (given the fact
that you did not have to hack the runtime to do it). This is something
you can do in Javascript and that you cannot do in lots of other
languages (because they don't have anything like
Function.prototype.toString).

> If it's a reply to the point I was trying to make, I'd say it's pretty far
> off the mark. The language sets forth a set of rules by which the language
> is interpreted. Of course you could write your own interpreter. Or you
> could preprocess any language into javascript and eval. Whatever language
> one codes in, it's important that they comprehend the rules of evaluation.
> A javascript programmer has certain expectations -- misleading them in any
> way does them a disservice.

I agree that it is pretty far off. It does not invalidate what you say
about Fibers and  the single frame continuations rule.

Bruno

Re: [nodejs] Re: My Humble Co-routine Proposal Dean Landolt 11/20/11 3:45 PM


On Sun, Nov 20, 2011 at 1:17 PM, Bruno Jouhier <bjou...@gmail.com> wrote:
> It's not about "breaking" javascript -- it's about violating invariants
> assured by javascript about its runtime environment. When you use eval all
> bets are off, and we all know it.

But banning eval (and new Function(str), of course) isn't enough. You
can use Narcissus.evaluate(str) instead. If you do, there won't be any
calls to eval or new Function(str) under the hood and the bets will
still be off!

My point was that the problem is not just with eval, it is also with
Function.prototype.toString(). This function is what makes it possible
to write a transform(fn)() that executes a modified version of fn (in
which fn's invariants may not hold). I know that this is not strictly
"executing fn" but still, I find it rather disturbing (given the fact
that you did not have to hack the runtime to do it). This is something
you can do in Javascript and that you cannot do in lots of other
languages (because they don't have anything like
Function.prototype.toString).


Technically javascript doesn't have this either -- it remains unspecified but IIRC it exists everywhere but some older Blackberry devices. But good luck parsing all those "[native code]" functions :)

AFAIK there's nothing inherently unsafe about what Function.prototype.toString does. You say this doesn't exist in lots of other languages but I'd argue Function.prototype.toString is a feature not a bug, a feature lisps are quite literally built of. Hell, any language with macros has to have it. Javascript's implementation is un(der)specified and incomplete, but this is slated to be corrected and IIRC macros are part of the long term road map. In fact, Brenden Eich is on the record as saying something like the language "won't be complete without hygienic macros".

 
> If it's a reply to the point I was trying to make, I'd say it's pretty far
> off the mark. The language sets forth a set of rules by which the language
> is interpreted. Of course you could write your own interpreter. Or you
> could preprocess any language into javascript and eval. Whatever language
> one codes in, it's important that they comprehend the rules of evaluation.
> A javascript programmer has certain expectations -- misleading them in any
> way does them a disservice.

I agree that it is pretty far off. It does not invalidate what you say
about Fibers and  the single frame continuations rule.

Bruno

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

More topics »