Yes coroutines. true non-threaded pure cooperative in scripting engine
processing is something very nice that is missing in Javascript
language spec. But wouldn't this be more of a V8 thing to support or a
heavy patch to it than a node thing?
I considered also moving Lsyncd to node instead and recoding it from
Lua over to Javascript, but JS is missing few features it relies upon.
For me most important, true associative arrays (that is table keys can
be anything not just integers and strings) and weak link tables
(garbage collector can remove it, if an object is only referenced by
such tables) So I rather wait for Luvit for this :-) On the other
hand, I like how JS offers me well established conveniences like
"switch", "continue", ++ operators, ?: operator, where Lua-ism can be
a bummer.
I currently work on another web project that will surely be fixed onto
node, because javascript server and client side gives here some
interesting synergy, because it shares quite some bunch of code
between server and client (and also for some people having to
work/learn with 1 language only is also a bon).
Javascript and Lua. I always think as analogy C and Pascal a decade or
two ago. The former is the more industrial language, the later the
more academic. While the later is actually the tad "cleaner" language
(whatever that means) on the large scale the formers industry power
just outruns. That doesn't remove interesting niches for the later
tough.
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
BTW: I recently coded up something very similar if I see this correctly:
https://github.com/axkibe/node-green-light
Didn't echo on node mailinglist a lot. Got just told to use
streamline. Fiber-anything got a massive negative touch to it. (And
yes while I also dislike pthreads a great deal, there is nothing wrong
with true (green) coroutines). IMHO if there would be true V8 support
for coroutines, instead of hacking pthreads with global locks, things
might look different, would they?
Nah, it'd still be confusing (if I understand you correctly). Example:
function User(name) {
...
}
User.prototype.setPassword = function(newPassword) {
var salt = makeRandomChars(16)
this.store('salt', salt);
var passwordHash = hash(newPassword, salt);
this.store('hash', passwordHash);
}
As soon as "store" becomes a blocking function, executing this
function twice might well cause you to have corrupt data in your DB.
Great. In streamlinejs, I'd be able to see the underscore and could
figure out that some caution is needed here.
What's wrong with streamlinejs? It seems to work, it has useful stack
traces, it doesn't really introduce large run-time performance
penalties... and you can even use it in the browser. If you have a
tool that works, why make a magic wand? Sure, it looks impressing and
you can do some weird magic with it that nobody can master without
studying gray magic for some time, but that's not what I expect a good
developer to want. I expect him to want something unambiguous that
just works.
PS: Yes, I do realize that that example would create inconsistent
state if the program fails at some point between those store calls.
Let's just assume it's an in-memory DB that gets cleared after each
restart so that the example becomes even more contrived.
IMHO you didn't understand coroutines. This doesn't just doesn't mean
that anything can suddendly block/reenter.
As Tim wrote
"In this proposal you create a co-routine explicitly outside of the
main stack. (The main stack is *never* suspended). Within that new
stack, you can `wait()` and `resume()`. This makes it possible to use
any normal non-co-routine-aware async function and using function
composition, it's trivial to make your co-routine an async function
with a callback as well."
No function will wait() resume() which did not create the coroutine in
first place. In this proposal (and with greenlight) it even cannot,
since it misses the relevant handlers. If the "magic" handler is
called '_' or something else is a matter of taste. Its imho this basic
misunderstanding that causes the promp refusal. Think of a coroutines
basically just as one routine.
You might actually follow one of the links to see the proposals.
Also I personally usually just vanilla callbacks for everything
because I'm used to it. I have some background in functional
programming and it's natural to me. But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.
Anything that looks like blocking code without the user having to use
slightly special syntax is evil. Callbacks are great and viral in
this regard. They are just difficult for certain things and debugging
is tricky with the short stack traces. My proposed coroutines are
good because you have to explicitly call wait() to suspend your
co-routine and you can only do this within the body of a
Fiber(function () { ...}) block. The call to Fiber is non-blocking
itself so it's caller isn't affected at all. This explicit and
obvious place where state can change out from under you is the same as
callbacks and thus acceptable. The underscore in streamline is like
that too, but code transformation seems a little too much for this.
But that's just my personal taste.
Also I personally usually just vanilla callbacks for everything
because I'm used to it. I have some background in functional
programming and it's natural to me. But obviously this is a real
problem for many people and so I think a simple and elegant thing like
my proposal being in core would help end this silly debate.
Ted
a(resume);
b(resume);
c(resume);
d(resume);
wait();
wait();
wait();
wait();
Looked it up, generators are effectively indeed the same thing as coroutines - as far I see it. Suppose then we just have to wait until ECMA 1.7 comes to V8
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need. Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has? Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows. From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard. This includes promises and CPS style callbacks
as well as my proposal. The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.
>>>> nodejs+un...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>>
>>>
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to nod...@googlegroups.com
>>> To unsubscribe from this group, send email to
>>> nodejs+un...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>> --
>> Job Board: http://jobs.nodejs.org/
>> Posting guidelines:
>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>> You received this message because you are subscribed to the Google
>> Groups "nodejs" group.
>> To post to this group, send email to nod...@googlegroups.com
>> To unsubscribe from this group, send email to
>> nodejs+un...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
After endless debates on the best way to "fix" callbacks with solutions ranging from code transformers to fibers to real threads, I've had some time to think about this problem myself.
As an experiment, I "ported" (more like re-implemented) node to the lua language as Luvit < https://github.com/creationix/luvit>. I learned three very important things in this experiment.
1. Lua is not JavaScript. It's pretty darn close and I like both for unique reasons, but JavaScript is *far* more popular due to this thing we call the internet.
2. V8's JS to C++ conversion is really slow compared to luajit's Lua to C layer. Also V8 uses a lot more ram and starts up much slower.
3. Co-routines in Lua are really nice and with very little extra sugar make a simple, obvious, and effective way to write sync style code while living in an async world.
Because of discovery #1, I don't think Luvit will ever become more popular than nodeJS, especially for anything web related. I see Luvit being used for things like mobile devices where performance *really* matters, JIT can be disabled (think iOS), and the built-in FFI makes SDL and openGL possible without bindings.
I do think that we can take what I learned from Lua's co-routines and bring it to node with minimal changes to the language. This proposal needs two things:
1. A new co-routine primitive as either a node add-on or an addition to the language.
2. De-structuring assignment to keep the syntax sane since async functions tend to "return" more than one value.
In this proposal you create a co-routine explicitly outside of the main stack. (The main stack is *never* suspended). Within that new stack, you can `wait()` and `resume()`. This makes it possible to use any normal non-co-routine-aware async function and using function composition, it's trivial to make your co-routine an async function with a callback as well.
This integrates well with existing node code and doesn't introduce any new hazards that aren't already there. No new hazards is a hard requirement for anything like this. Programming in node is hard enough already, let's not make it any harder.
For an example, see this: <https://gist.github.com/1349451>
@Ted, I've never thought that faux blocking was good for parallel
tasks. I would just use vanilla callbacks (or if you prefer promises)
for that, but as Axel pointed out, it's almost possible with my
proposal, the tricky part is you don't know which resume happens first
so getting any interesting data out of the resume callback is
impossible. The first wait will suspend the co-routine, but the first
resume called (which is unknown since the I/O waits are parallel and
a,b,c, and d are async) will resume it and pass the results to the
first wait, then it will suspend on the second wait and so forth. The
last wait won't resume till all four resumes have been called in any
order.
@Dean, Indeed generators are very close to what I want, but I'm not
sure they allow the type of argument passing I need. Since it's not
part of ES yet, maybe there is time to use a fuller implementation
like lua has? Also I don't understand how single-frame continuations
fix interleaving hazards that this still allows. From what I
understand anything that doesn't block the event loop (which is a
requirement for this architecture), will have some sort of
interleaving hazard. This includes promises and CPS style callbacks
as well as my proposal. The best you can that I know of is to make it
obvious to the caller the points at which other stuff can happen.
On Nov 9, 2011, at 11:18 AM, Tim Caswell wrote:
> @Ted, I've never thought that faux blocking was good for parallel
> tasks. I would just use vanilla callbacks (or if you prefer promises)
> for that, but as Axel pointed out, it's almost possible with my
> proposal, the tricky part is you don't know which resume happens first
> so getting any interesting data out of the resume callback is
> impossible. The first wait will suspend the co-routine, but the first
> resume called (which is unknown since the I/O waits are parallel and
> a,b,c, and d are async) will resume it and pass the results to the
> first wait, then it will suspend on the second wait and so forth. The
> last wait won't resume till all four resumes have been called in any
> order.
Have you seen tamejs? What do you think of their solution to this
problem:
var res1, res2;
await {
doOneThing(defer(res1));
andAnother(defer(res2));
}
thenDoSomethingWith(res1, res2);
Certainly not impossible. Just have to wrap the callback to wrap the
answer where it came from.
a( function(err, asw) { resume(['a', arguments]); }
b( function(err, asw) { resume(['b', arguments]); }
c( function(err, asw) { resume(['c', arguments]); }
d( function(err, asw) { resume(['d', arguments]); }
results = {};
for (var i = 0; i < 4; i++) {
var asw = wait();
results[asw[0]] = results[asw[1]];
}
Well looks a tad more bloated already. Certainly not simpler than the
async version would be. Albeit as Tim said, this is not the strong end
of the concept, but it still works. Might also be offloaded to a
convenience library, like async stuff does anyway.
> Can we please identify the problem with callbacks. I'm not saying there isn't one, but most "examples" i see are so contrived and easily
> handled with named functions that I start to grit my tether whenever someone makes a comment like this.
It depends on the usecase, when logic gets more complex with classic
callbacks I often see myself writing statemachines. While
statemachines are great and efficient once they work right, humans (or
at least I) are not great in getting them proper or debugging them.
For small tasks and handlers I do classic callbacks, for more complex
logic I now use streamline, like that database data analyzer I
recently worked. Streamline does quite well in moving some of the
implementation details out of the way, so I can work on a more logical
level. I haven't yet figured out how to interprete its linenumbers in
backtraces. I honestly don't care about microseconds in my use case.
However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.
Correctly. For the implementation with fibers at least as long you do
not out trick by requiring the fibers modules directly. With that sort
of API Tim and I suppose have done more or less the same with
greenlight. if you did not get the wait handler down the stack for the
current context (or red sign as I called it) you cannot yield.
However, I really wonder if at the end of the day, proper native
supported coroutines might even be faster than the closures and temp
objects like streamline generates for you.
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
>
> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).
Isolated CO-routines ? Care to explain how would that work or what do you mean ?
> (snip)
>
> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.
What overhead?
The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.
Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.
> (snip)
>
> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.
See below.
> (snip)
>
> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.
Yeah. You first, please.
--
Jorge.
NoNewNoCoRoProCo, SoHo.
> On 09/11/2011, at 23:32, Mikeal Rogers wrote:
>> (snip)
>
>>
>
>> In order to avoid coroutines from altering shared state you'll need to use an isolate, or some kind of sandbox, which will be more expensive than a closure (at least in v8).
>
>
> Isolated CO-routines ? Care to explain how would that work or what do you mean ?
In javascript you avoid many side effects by having a "lock" on the whole vm while your code runs. You "block" the entire VM and state can only mutate in between callbacks. So if i do something like:
var blah = 'asdf'
doSomething(function (text) {
blah = text
require('module').doSomethingelse()
write(blah)
})
I **know** that i'm writing the value of text that was passed to MY FUNCTION, even tho this variable can be mutated from outside of my scope because I only call a function that doesn't have access to that scope. I avoid side effects from another doSomething callback that might be running in "parallel". This is a common problem in threaded programming.
Let's say this callback is run many times in parallel, and that doSomethingelse() is a fiber that can go off and do IO. Other doSomethign callbacks are going to fire in parallel and mutate blah. The way to avoid this is to create an isolate that copies the state from the parent isolate and any mutations are copy-on-write so i can't mutate the state of the other isolates.
If you don't do that you get all the same horrible side effects that plague threaded programming.
>
>> (snip)
>
>>
>> You have to remember than we get to throw out the stack and only hold a reference to the function which holds a reference to *only* the objects in the closure that are referenced in that function (magic!). With coroutines you keep the stacks around, which has some overhead.
>
> What overhead?
>
> The stacks are created (on function call) and destroyed (on exit) anyway, with or without fibers: it does not matter whether you destroy them now or a second later, the cost of destruction is exactly the same.
No, with fibers the stack is "yeilded" and stays around, the new stack goes off and does IO and pauses the parent stack until it returns. With callbacks you exit out of the event loop and the stack goes away, the function that is referenced in the event system stays around because there is a reference to it.
>
> Buuut, with callbacks+cps there's 2 calls, thus there's 2 stacks created and 2 stacks destroyed, while with fibers there's only one.
>
>> (snip)
>>
>> I've seen some benchmarks of haskell and lua using coro that are very impressive but I don't think they fit in to javascript nearly as well and I doubt they can be optimized to the same degree.
>
>
> See below.
>
>> (snip)
>
>>
>
>> Let's stick to talking about the tradeoffs and refrain about things we are not qualified to make assumptions about.
>
> Yeah. You first, please.
> --
> Jorge.
>
Please don't forget about the W16 project too - that idea shows some real promise IMHO: https://github.com/sheremetyev/w16
There is no need for isolates here. There is still only only one active
stack at a time and the main "thread" (as lua calls it) or stack can
never be suspended. Since the main "thead" can never be suspended,
there is no interleaving hazard except when the main "thread" explicitly
yields by letting the call stack unwind.
As far as implementation, I'm sure that's possible. node-fibers does
much of what I need and also the proposed new features Dean mentions
also seem to do what I need. The thing I want to propose is a sane
solution that's not any more dangerous than callbacks already are and is
as minimal as possible.
Going back to the hazard example:
var blah = 'asdf'
doSomething(function (text) {
blah = text
require('module').doSomethingelse()
write(blah)
})
This is actually not safe even in a syncronous world without callbacks
or co-routines if the module has a reference to the doSomething function
(IE it re-enters). But assuming you don't make that simple mistake, yes
it's safe. Adding async calls into doSomethingelse() or my proposed
co-coroutine API doesn't change this in any way. Unless one of your
direct descendents in the call stack re-calls doSomething before it
returns, you can assume that blah still holds the value you put in it.
This is not true of all co-routine implementations which is why I'm
making this distinction.
Marak, I completely agree with you. That's why my proposal is so
minimal and integrates cleanly with async callbacks. From the outside
it's no different than using Step. The caller doesn't have to worry
that there might be some bad magic inside there it has to worry about.
The functions you call from within the co-routine are also normal async
functions that take a callbacks.
While it does add more complexity to the system, it's 100% optional
complexity and you don't have to worry about it if you don't want to.
You don't even have to worry about it if the libraries you use make use
of it. There is no concept of "fiber-aware" functions. Think of it as
a slightly more elegant version of step.
Despite an admirable effort, this is still magical and highly
complicated for anyone who doesn't grok it deeply.
It's too hard. I'm gonna stick with my hand-rolled flow control tricks.
On Wed, Nov 9, 2011 at 21:46, Bradley Meck <bradle...@gmail.com> wrote:
> Errata : when saying we don't have the interpreter lock in v8 I am talking
> about isolates and not the use of lockers.
>
On 11/09/2011 04:57 PM, _doq wrote:
> Rant warning!
>
> I came to this group based off a comment from polotek about:
> 1. There is a group, and the devs are super involved in it!
> 2. There were a few uneducated programmers attempting to change the
> whole of nodeJs via combating what they think is a problem with callbacks.
>
> At first I was like, hmmm don't seem to be that much static out
> here... boy was I wrong. I am glad you like lua, but the issue with
> callbacks in every case I have seen is user error, as in they really
> don't get it. Maybe there is a really big learning curve I am not
> seeing, but all I can see is a bunch of stubborn coders trying to
> convert others to there way of thinking rather getting on board with
> the coop para-dime, this reminds me of the transition from procedural
> programming to oop.
>
I do feel there is a large misunderstanding of my goals here.
First and foremost, let's set the record straight. I'm a strong
advocate of callback based programming and have been since the early
days of node. I even fought against blocking require because async
require was more elegant to me and I didn't see the point of using
blocking require just to make it easier for people coming from other
systems. I celebrated when promise.wait was removed and again when
"promises" were removed all-together. I've spent *years* trying to help
newcomers to node understand callbacks and tried to convince them to not
use hacks and instead just learn it "right". I understand the community
has grown a lot and I've been rather inactive for the last year while
working at HP, but I an *not* trying to convince anyone that co-routines
are better than callbacks. In fact I personally rarely use co-routines
even in Lua. I am definitely in the callback camp on this issue.
Here are some of my articles and presentations on the topic. I won't
bother you by searching through the mailing list and posting those as well.
- http://howtonode.org/control-flow
- http://howtonode.org/control-flow-part-ii
- http://howtonode.org/control-flow-part-iii
- http://howtonode.org/do-it-fast
- http://howtonode.org/step-of-conductor
- http://jsconf.eu/2010/speaker/techniques_and_tools_for_tamin.html
However, despite my efforts and the efforts of others, many newcomers to
node constantly have a hard time with the callbacks. I don't believe
that I'm just smart and everyone else is dumb. Maybe it's just my
background in functional programming, but callbacks are very easy for me
and thus it's hard to have empathy for people who struggle with it. But
the fact is that it's been a constant struggle for new people ever since
node was announced. One summer it seemed to be all this list was about
and all conference presentations on node talked about.
This "humble proposal" as I put it is a compromise, a proposal, to help
those coming to node. I have very strict rules about it's impact on the
rest of the system before I even consider promoting it. It must not
cause any more hazard than is already there. It must integrate cleanly
with normal callback based async functions. It must be exposed
externally as a normal async function. It must use internally normal
async functions.
I am sad that my post to the mailing list got sidetracked. I do not want
to discuss all the alternative ways to avoid callbacks that have already
been hashes and re-hashed on this list. I simply want smart and
unbiased thoughts on my proposal. While working on luvit I found this
entire other culture that understand co-routines and they consider the
people on the node list who defend callbacks (myself included) as
arrogant and refusing to be open minded about the issue. I discussed
the issue with them on their territory where they aren't constantly told
to shut-up and go away and seen as a bother. I learned that I
misunderstood the idea of co-routines and many of my arguments against
them were false.
> Stuff like this makes me not want to even look at your code, "Lua is
> not JavaScript. It's pretty darn close," yeah... it is just as close
> as C is to VB. I am really surprised though all your attempts you
> didn't just learn to use callback more elegantly rather then trying to
> re-core the language.
>
I'm sorry my post offended you. I know JavaScript far more than I know
Lua, but in my limited experience with Lua (read a couple books and
wrote a few thousand lines of code), it's very much like JavaScript.
I'm not talking syntax but semantics. Functions are almost identical
(first class values), closures work the same. Tables are similar to
objects, metatables can emulate prototypes and are a lot like proxies.
There is type coercion with truthy and falsy values. They both can run
on JIT capable vms. Primitives are passed by value, but anything
mutable is passed by reference. I could go on, but this is not a lua
forum. My point here is that using libuv with luajit is about as close
to a nodejs experience as you're going to get in *any* non-javascript
environment. It was important for my experiment to use a language that
already has co-routines in the community instead of hacking them onto
v8. I'm expert in node and the node architecture, not in co-coutines.
If I was going to evaluate co-routines, I needed the help of a community
that understood co-routines and has used them for years in real projects.
> Rant End!
> --
As I said in my nodeconf talk. It's very important this community stay
friendly as it grows. The leadership and core community members need to
help this out by not being arrogant, but instead understand the problems
newcomers face. It's hard to understand someone else via email if we
don't see eachother face to face first, but remember there are real
humans there with real feelings and concerns.
-Tim Caswell
Personally I'm torn between Fibers, Streamline and classical
approaches and do something else with every subproject. As said, there
is some limit to me in coding logics where classical callbacks start
to be come very tiresome - that is loops for me. And nested loops is
where it really gets me down and a solution like streamline really
eases the day a lot. Maybe node is really more suited for the simpler
I/O stuff, and if you code too much logics you should have better
chosen something with classical CGI/processes from start? Anyway if
one looks through the node module list, theres a hugh load of "control
flow" modules - are they all "just stubborn", or is there in fact an
itch they try to scratch?
And yes, Lua and Javascript are surprisingly quite close cousins but
without any common history (except scheme long down the road), call it
convergent evolution, as it happens occasionally in nature.
To go back on your original proposal, aside coding it with fibers as
interlocked pthreads that are built in the V8 and accepting their
costs I see noway to make node/javascript do efficient cooperative
routines without altering the V8, right? I was wrong before when I
said, I suppose through generators they will come anyway, since
generators are Javascript 1.7 (Mozilla) not ECMAScript and Google
targets with V8 only ECMAScript and whatever Webkit implements. Thus
if, we should best convince webkit to make something coroutine like,
then Google will duplicate that, then it will automatically come to
node :-). Personally I consider the Javascript 1.7 take on coroutines
not too great, calling it a generator and the way it is presented is
as one possible use of coroutines than the other way around.
Didn't know that, thank you for clearing that up.
// Lets choose _ as blocking highlighter since thats well established.
coroutine(function(_) {
// anything that does not have '_' cannot yield.
// this one runs one asyncFunc. It expects the last paramter not
given there to be the callback,
// which returns (err, asw). asw is returned, err is thrown.
var asw = _(asyncFunc, param1, param2);
// this place is reached after asyncFunc calls its callback.
// and since it has been requested this is how you could do two
things in parallel.
var asw1, asw2;
_(asyncFunc1, param1, param2, function(asw) { asw1 = asw },
_, asyncFunc2, param2, param2, function(asw) { asw2 = asw });
// the magic difference is _ appearing as its own paramter so here
it uses itself as
// seperator of two calls to be run simultanously. Dunno maybe it
would be simpler instead
// of adding callbacks to return an array?
// this place is reached after asyncFunc1 and asyncFunc2 call its callback
});
// this one immediatly returns into the main stack (or when coroutine
first runs into _ or returns, I don't know now)
That's why I was careful in this co-routine proposal to make sure it
still used and provided node style callbacks. In fact it's even more
agnostic than that, but that's just a side effect of it being so
minimal. The _(fn, param, param2) idea works too, but it assume node
style callbacks and is a little more abstraction.
I guess my proposal is more of a set of rules where coroutines can be
used safely within node. I think we all agree on that now.
- It should be efficient, not use threads and locking in the implementation
- it can be either a node addon like node-fibers or implemented using
generators from es.next, as long as the other rules are sastified
- There should be a clear sigil to know when the coroutine gets
suspended and other stuff can happen.
- The effect needs to be not viral. If a module I depend on uses
coroutines, I shouldn't have to be aware or change my usage in any
way.
-Tim Caswell
I don't want anything killed in SoHo, verbally or otherwise. That was
just some jokey wordplay.
> call you stubborn, and
> what else gives. Anyway Tim is right, coroutines are not replacement
> or alternative for callbacks, they are simply another flow-control
> options for the user. So what gives the hatred?
I started writing a reply, and it turned into a blog post.
http://blog.izs.me/post/12604303054/experts-idiots-and-taste
Sent from my Windows Phone
From: Tim Caswell
Sent: 11/10/2011 10:12 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
I just had an idea :P
async function bar(a, b) {
doSomething()
return(err, results) // <-- "special return"
}
async function baz(a, b) {
doSomethingElse()
return(err, results)
}
{
err1, results1 = bar(a, b)
err2, results2 = baz(a, b)
doSomethingAfterBarAndBaz()
}
doSomethingNotBlocked() // <-- because it's outside that scope
---
Diogo R.
+1
---
Diogo R.
Sent from my Windows Phone
From: Diogo Resende
Sent: 11/10/2011 11:08 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: My Humble Co-routine Proposal
---
Diogo R.
--
Since Tim has made a lot of good contributions to the Node community
(his original simple framework is what got us using Node), I think his
ideas are worth strong consideration - it would be nice if somehow flow
control (aka sequential operation) gets standardized or included in
core. In many applications it is a pretty necessary component.
There are two entirely different problems being talked about and everyone who wants coroutines love to conflate them because it makes for a better example.
The problems are:
1) You have a question you want to ask and the answer to that question is necessary for your application logic. This requires a callback.
2) You have an operation, or set of operations, you want to do to move or mutate data between file descriptors. This requires a stream.
Everyone needs to stop pretending that coroutines are better than streams, it's insane. None of these examples are simpler than.
req.pipe(filed('/path/to/file')).pipe(res)
or
req.pipe(request('http://www.google.com')).pipe(res)
While streams are still changing, and improving, what we can already do with them is quite nice. It's also considerably simpler than any example anyone, ever, has posted with coroutines. Seriously, I feel like I'm taking crazy pills, people will post their contrived example which is a few lines shorter than another contrived example not using streams and claim that it's shorter/simpler, it's not, stop it, seriously.
For the other problem. Asking a question required by application logic, we currently use callbacks. A large amount of indentation occurs when, and only when, the final solution in your application logic requires data from the top of the closure before you asked these questions. That is the problem. It's solvable, I'm not convinced coroutines make it simpler, at least in JavaScript.
The largest problem I see is that the data at the top of the closure is usually a set of Stream objects, so doing that IO is going to cause you to lose data in the streams. This is solvable, we add buffering semantics to .pause().
But I digress. This thread is obsessed with the wrong problem. It's something that is less than obvious when you first use node, so it gets a lot of attention, but it's much less of a problem now than it used to be. The more work we do in modules the simpler this gets. I'm not sold on generic flow control libraries either, the best solutions I've seen are domain specific, but good things are happening.
There are a lot of people saying "this is too complicated, we need to simplify it" and then proposing solutions that have an incredibly high tolerance for new complexity and build solutions squarely outside of where we are actually making progress on solving the problem already.
This is not productive.
-Mikeal
You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.
While it might be annoying to have all these if (err) blocks a much bigger problem occurs when calling a function that throws. Most of the time, we call functions we did not write ourselves, which may have bugs which means they may throw and we can't even wrap them reliably in a try/catch because the error may be in a future callback added to the event system by that code.
We have a proposed solution for this, domains. Domains would allow one single handler around any amount of arbitrary code and inside any of the callbacks it might add to the event system.
Domains are a much simpler solution than coroutines and handle far more cases, they are so simple in fact that advocates of more complex proposals go to great lengths to ignore the domains proposal, much like they ignore streams.
I'm starting to identify a divide here.
node.js prefers solutions to problems that lend themselves well to abstractions. streams are great example, they lend themselves to amazing higher level abstractions that are nearly transparent to the user. Domains are even better, they will mostly be used by frameworks and other libraries and be invisible to application code.
Every coroutine proposal I've seen pushes it's API to the highest level, crushing hope for a transparent abstraction. This is why contrived examples are so important to their justification. The only way to present a solution that pushes it's API through many layers of abstraction is to present the problem as something that is not already abstracted.
This is not productive.
-Mikeal
Have you not seen Tims proposal then? I don't see how it pushes it's
API to the highest level. The coroutines stuff stays local to one
routine and it still communicates downward and upward with callbacks.
> This is not productive.
This is not productive.
Yes, I've read Tim's proposal.
It mostly competes with streams. Have you used streams?
>
>> This is not productive.
>
> This is not productive.
>
Yes for streaming data. Duh. If it can be used for more, please answer
Liams question, are suggesting streams as solution to loops?
If so, can you please recode following example here done with
streamline with streams instead? (it is a snipped from a real
application)
var c1 = dbclient.collection('example', _).find(_);for(var o1 =
c1.nextObject(_); o1; o1 = c1.nextObject(_) {
var c2 = dbclient.collection(o1.refer, _).find(_);
for(var o2 = c2.nextObject(_); o2; o2 = c2.nextObject(_) {
c2.set('value', 1, _);
}
}
console.log('finished');
Yes you could pause the c1 stream, start a c2 stream, if it finishes
resume the c1 stream for another object, pause it, starts. I don't buy
this being simpler.
Dunno how similar the SQL interface is, but that one was originally
against a mongoDB; I try to be cool and modern :-)
What the code above does is for every row of a potentially large
collection (or call it table in SQL), it reads a string which contains
the name for other potentially large collections (tables). For these
tables it sets the field 'value' for every row to 1.
Yes its a made up example, didn't had to do a nested loop in practice so far.
>You never throw, you probably pass that error to a response handler for errors which generates a 500 page, or a logger than logs the error, you never throw.Why don't you ever throw? What's wrong with having a try\catch at the top of your incoming request and catching it there? If an uncaught exception is thrown you catch it in the handler and send down a 500 error page. Sometimes you can catch an error before it makes it that far and fail gracefully. Say, an advertisement fails to render you can catch that exception and just blank out that div instead of killing the whole page.Throwing is a beautiful thing because it means you can't ignore errors. Errors pop up in unexpected places, and having to constantly check for errors and forward them along is silly. And you will absolutely miss errors.
> On Fri, Nov 11, 2011 at 12:02 AM, Mikeal Rogers <mikeal...@gmail.com> wrote:
>> How about you explain what you are trying to do, in english, and I'll show you some code that does it simply.
>
> What the code above does is for every row of a potentially large
> collection (or call it table in SQL), it reads a string which contains
> the name for other potentially large collections (tables). For these
> tables it sets the field 'value' for every row to 1.
So, you want to do an operation to every row in a table? Is there any conditional logic for setting or not setting that row and if so does it require information that might be in the closure?
-Mikeal
> Domains are a much simpler solution than coroutines and handle far more cases,I have followed this mailing list for a year and have never heard of domains. Where can I read about them?
> They were first proposed by Ryan at NodeConf SummerCamp and have been mentioned a few times on the nodejs-dev list.I hope I'm not a pest, but I can't find anything on this and I'm really curious. I searched the nodejs-dev list for "domain" and got nothing. Would it be too much to ask for you to describe domains in 100 words or less?
> I'm using, and talking about, the node.js that 99.9% of the people use. I'm interested in solving the problem within node, you're talking about something else.
Ok then don't use it. Just don't come into our thread and shit on everything we're doing. I'm saying these features are compelling to me and a non-insignificant group of node users. We don't need or want these features to be in node core. Actually Tim might be saying that, but it's not my position. If you want to solve the problem your own way then I applaud you because diversity is really healthy here. I just don't understand why you feel the need to bust into every thread about fibers and start yelling at everyone. At least Isaac is being constructive and open-minded, all you're doing is saying "fuck this and everyone involved!"
> Hope that explains it.Thanks. That is very interesting. I will have to think about what it would mean for my code.
I take issue with the word "native" in that sentence.
Please read the last 3 paragraphs again, wherein I thoroughly explain
that I don't care either. (So, why can't we seem to stop talking
about it...?)
> So, the streamline/fibers magic is probably not for you.
The pope is probably not a martian.
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
>> e.g. domains (that's a confusing name, BTW).
Sidetrack, for the sake of my sanity and many others do not use that name if it ever gets implemented in core. That's nearly the most completely unsearchable keyword I can think of, even filtering it to Node or JS searches.
>> Windows support is feasible once node gets dynamic module loading workingWorks fine since 0.5.10 (or .11). Only difference is compilation needs to be done using MSVC against node.lib which is produced alone with node.exe when building. I found a number of Windows optimized implementations for what would be needed to implement the C++ end of Fibers when I looked into it.
Ultimately I decided that it would be better to continue working on various patterns built on top of Harmony WeakMaps and Proxies and some other stuff to allow eventual/soon async promise type execution to be usable by implementation agnostic synchronous consumers of the code. I feel that's probably the best likely long term route for solving my specific goals.
> These threads always go down this bowl-shaped hole because there are
> two camps:
> A believes async logic forces awkward code style on users
> B believes broadly-applicable solutions to these issues are worse
> than the problems
I actually believe A and B are both true and am in search of C
C: patterns in async that simplify and/or abstract the awkwardness of async
we've made huge leaps in C over the last few years but we're not done yet. streams is one of these patterns, domains is another.
Would that also allow having stuff like the current http request in a global that's domain-bound?
> > They were first proposed by R...
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Maili...
"The heretic must be cast out, not because of the probability that he
is wrong but because the possibility that he is right."
The strong emotions, the FUD that is spread by people who got only a
rudimentary understanding of coroutines at best. Otherwise I perceived
so far by large node and community always very practically oriented
instead of doctrinal, you do what works for you.
BTW: When I get some time, this will be my next approach to built
something stable&use upon fibers:
coroutine(function(_) {
/* single wait call, last parameter is callback
_ knows when it is waiting or when not, so it can be the same func for
yield and resume. */
var asw = _(func, a, b, c, _);
/* callback not at end, no problem */
var asw = _(setTimeout(_, 15));
/* non standard callbacks, just wrap them */
var asw = _(func, function(a1, a2, a3) { upvalue = a1+a2+a3, _(); };
/* an interface to non standard callbacks like couchedb is easy. */
asw = _(couchdbFunc, a, b, function success(asw) { _(null, asw)},
function fail(err) { _{err});
/* do two things parallel, the answer will be in an array[2].
_ sees that the first parameter is instanceof Array. */
asw = _([func1, a, b, c, _], [func2, a2, b2,c2, _]);
});
The possible downer I see to this is 'this' not being in the funcs
what you might expect? I honestly never truely undestood how
javascript handles this and just fix it when its wrong.
Sent from my Windows Phone
From: Bruno Jouhier
Sent: 11/11/2011 4:13 AM
To: nodejs
Subject: [nodejs] Re: My Humble Co-routine Proposal
Glenn,
Bruno
@Isaac,
Just read your blog post again. You make a lot of assertions and
explain why you would use or not use solution X, Y or Z. Fine, but
this is a bit irrelevant AFAIC.
I did not write streamline for you, and I don't really care whether
you like it or not.
A bit of context: I chose node.js for a project, and got laughed at
initially (server-side Javascript, you must be joking!!). I had a
small team of very skilled engineers. We started to write code the
'callback way'. In a few months we had prototyped a number of things
and written about 10,000 lines of code, all in callback style.
We had been quite successful overall and the team was excited about
the technology but I had to make a hard call: are we on the right
platform? Will we be productive enough? Are we writing code that is
easy to maintain or not? Will we be able to grow the team and maintain
discipline? How much is this going to cost us (in training, in code
reviews, etc.)? Is this viable or are we going against a wall?
My assessment was that, despite the excitement, the early successes
and the fact that node was getting more and more attention, we were
writing code that was fragile and difficult to maintain. We could do
wonders with JS and some things that were really hard/painful before
(dynamic stuff, I/O) had become incredibly elegant and pleasant; but
on the flip side, things that were really easy before (like writing a
simple rule like "if path + '.js' does not exist and path is a
directory and it contains a file called 'index.js' do XYZ else ...")
had become a lot more difficult, painful to write, error prone, less
maintainable, etc.
In the kind of software we are writing, there is about 10% of clever
technical stuff and 90% of mundane not-very-technical logic. Even if
we were in better shape on the clever stuff, we were going against a
wall if we could not maintain our previous productivity and quality
standards on what makes up 90% of our code.
Fortunately, I had used the experimentation time to experiment with
flow control libraries and I had found a way to greatly simplify the
problem with a preprocessor.
So, streamline is not something that I did for myself, or to please
you. It is something that I did because we would have had to drop
node.js if we hadn't found a way to get back on a productive,
disciplined, maintainable track. And, if fibers or generators prove to
solve the problem better than a preprocessor we are going to
transition to them (and streamline is probably going to make this very
painless).
But our assessment was just that there were too many problems with
callback-style async programming, too much risk, unless we could find
a solution that would dramatically change the game. Fortunately we
found one, we are moving along at good speed on node, the team is very
enthusiastic and node is making the news everyday so I'm not getting
laughed at any more (at least not for choosing node).
All this to say that the two code samples that I posted before are not
irrelevant. They are very relevant to what a lot of people are doing.
So I strongly encourage you to write them in callback style and
compare the results with the code I gave: code size, number of
functions, complexity of the execution flow, distance between the
expression of the rules in plain English and their expression in the
code that you wrote, etc.
So, the streamline/fibers magic is probably not for you. But be open
and accept the fact that others may have different requirements than
you and that for them, a bit of magic can turn node.js from a geeky
platform that doesn't fit their bill into a very pleasant platform to
work with at all levels.
My 2 cents, FWIW.
Bruno
On Nov 10, 7:27 pm, Isaac Schlueter <i....@izs.me> wrote:
> On Thu, Nov 10, 2011 at 06:54, Axel Kittenberger <axk...@gmail.com> wrote:
> > I posted this once and do it again. I wonder when something that is a
> > practical consideration like callbacks becomes suddenly a doctrinal
> > thing to some people where even thinking otherwise makes them send you
> > verbally to be killed in Downtown New York,
>
> I don't want anything killed in SoHo, verbally or otherwise. That was
> just some jokey wordplay.
>
> > call you stubborn, and
> > what else gives. Anyway Tim is right, coroutines are not replacement
> > or alternative for callbacks, they are simply another flow-control
> > options for the user. So what gives the hatred?
>
> I started writing a reply, and it turned into a blog post.
>
> http://blog.izs.me/post/12604303054/experts-idiots-and-taste
The solution with streaming everything is that it doesn't go well into
logical boxes. A loop with an if condition. Nested loops, etc.
I never considered streams and coroutines to be opposites. Other
languages got both.. But if you want to take that perspective as
counterparts, I daresay coroutines is the more powerfull concept. You
can code quite easily a stream with a coroutine, but not vice versa.
A realization of a pump that does draining, buffer overflows, and all
that (Supposing out calls the callback when its kernel buffers are not
full)
function mypump(callback) {
coroutine(function(wait) {
try {
while(!in.endOfStream()) {
out.write(input.get(wait), wait);
}
} catch (err) { callback(err) };
callback(null, 'fin');
}
};