node-fibers or generators

[john.tiger]

for a project we do not want lots of nested callbacks so
1) where is node in regards to using harmony generators ?
2) is node-fibers a fork or add on module ?
3) if we use node-fibers and then later generators becomes "standard" -
would there be much rewriting required ?

[Marco Rogers]

1) node will support harmony generators the second they are stable in v8. The language evolves with v8, node doesn't touch it.

2) To my knowledge, node-fibers is a fork of node with patches to both node and v8 to enable fibers. So you can't just run node, you have to run node-fibers.

3) I think it would be very easy to paint yourself into a corner here. But there are lots of abstractions on top of fibers that could possibly be implemented later with generators instead. Keep in mind that when you dig into the details, fibers seem to allow deeper semantic changes in node than the version of generators in harmony. I'm not the best person to explain the difference, but I've understood enough of the talk to know they are not equivalent paradigms. Generators enable a subset of functionality that fibers does.

Hope this helps.

:Marco

[Mikeal Rogers]

fibers and generators (as described in harmony) have a large delta between them.

if you read the harmony spec for generators you'll notice that the yield statement pushes it's way all the way up the call stack so you can't abstract it away the same way common-node does with fibers.

what you should probably do is just use normal node, the way 99% of the community does :)

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Marcel Laverdet]

2) As of node 0.6.0, node-fibers is just a module. It never was a fork, but in node 0.4.0 it was much more of a hack. In node 0.6.0 it works just like any other module, you install it, require it, and use it.

3) Depending on how you use fibers, it may be possible to rewrite your code using generators, it may not be. Fibers will give you a superset of functionality and it's up to you to look at what generators will be and build your code to support them in the future if that's what you want.

[Mikeal Rogers]

i think this probably still needs further clarification.

fibers (when used as a module as Marcel describes) takes a callback and within that callback you're in a fiber. this could be adapted to "when you're in a yield" instead of "when you're in a callback".

but, common-node (and maybe some other libraries) have abstracted away that callback when you're living in their abstraction by using fibers under the hood (i'm sure Oleg will pop in any minute to explain how). the way yield is described for generators does not lend itself to ever being abstracted to that degree.

this isn't a judgement call about which one is better/worse, personally, i don't like fibers or generators :), but it's an important distinction for people who think that "all these callbacks" are a issue that needs to be solved by abstracting them all away (or in the case of common-node want to create a Narwhal compatible environment). one of these tools potentially does that (although it doesn't by default when used as a module) and the other never can.

[john.tiger]

On 02/10/2012 11:56 AM, Marcel Laverdet wrote:

2) As of node 0.6.0, node-fibers is just a module. It never was a fork, but in node 0.4.0 it was much more of a hack. In node 0.6.0 it works just like any other module, you install it, require it, and use it.

3) Depending on how you use fibers, it may be possible to rewrite your code using generators, it may not be. Fibers will give you a superset of functionality and it's up to you to look at what generators will be and build your code to support them in the future if that's what you want.

@Marco, notes that I've seen say generators may appear in V8 by end of 2012 - that's too far away for us

@Marcel - thks, our use is mainly for real clean sequencing of algorithms - so mostly looking at using yield()

[Oleg Podsechin]

but, common-node (and maybe some other libraries) have abstracted away that callback when you're living in their abstraction by using fibers under the hood (i'm sure Oleg will pop in any minute to explain how). the way yield is described for generators does not lend itself to ever being abstracted to that degree.

Thanks for mentioning Common Node Mikeal. For anyone who wants to check it out, here's a link: http://olegp.github.com/common-node/

There's nothing too fancy going on. I've just created synchronous wrappers for all modules that do IO. As a result, no code outside my library should yield, addressing the interleaving hazard concern that some have expressed. In fact, I've been meaning to add a mechanism which detects yields in the application code and displays a warning.

I do mean to make a fork of the library which uses generators once they're available. The upgrade path should be fairly straightforward with the yield keyword added before any method that does IO. However, based on this comment http://erik.eae.net/archives/2011/12/29/00.00.29/#comment-104393 it looks like generators won't be in Node for another year or so.

Oleg

[john.tiger]

the issue for us is that quick maintainability is as important as performance - the example of mongo query nested callbacks (like in the tests of node-mongo-native) is just way too messy (and if everyone else thinks it's fine then I'm in the 1% that don't).  

I'll check out common-node - I actually had considered Narwhal but when I saw all those java libs - yuk!  Maybe better for those already in java environments.  I like to keep stuff simple. 

[Oleg Podsechin]

the issue for us is that quick maintainability is as important as performance - the example of mongo query nested callbacks (like in the tests of node-mongo-native) is just way too messy (and if everyone else thinks it's fine then I'm in the 1% that don't).  

If you're using MongoDB, check out https://github.com/olegp/mongo-sync 

[Alexey Petrushin]

> the example of mongo query nested callbacks (like in the tests of node-mongo-native) is just way too messy

I was feeling exactly the same when I saw this samples. After Ruby it looked like I should write a lot more code to do the same things. :)

I tried to solve it and created mongo-model http://bit.ly/yaumij 

It uses Fibers and completely eliminates (well, almost) callbacks, I'm using it, but it's new and there still may be errors.

By the way, there's also Aced CoffeeScript http://maxtaco.github.com/coffee-script/

It doesn't depend on Fibers or Generators and seems to be interesting, but it's still very new and experimental.

[Mikeal Rogers]

comments like this make me scratch my head.

there are thousands of developers here on this mailing list maintaining node code using callbacks. there are products that do daily deploys across thousands of node processes that use callbacks.

there are hundreds of thousands of liberal arts graduates using and maintaining code that uses jquery (with callbacks!).

if you think that callbacks are a blocker for maintainability i can't help but wonder why all these other people seem to be fine but the it's such a blocker for you that you would need to opt out of what the vast majority of the community are using, testing, and running in production every day.

[Axel Kittenberger]

> comments like this make me scratch my head.

Lets not go into that discussion again. I think last time we
concluded, that projects are vastly different in their setup, and thus
in their needs. I can tell myself of different projects where on one I
was happy with callbacks and others it just went down. Last time we
had even talked down to examples where it was not succefully explained
by the "you doing it all wrong crowd" who said everything that
experiments with alternative designs to code than callbacks is doing
heresy.

Anyway. I won't jump into that discussion once again :-)

@john.tiger. You can also have a look at node-streamline. I rewrote a
web based monbodb live data analytics engine from callbacks to
streamline and was quite happy with the result. And don't let you drag
into these religous debattes, you just decide what works for you :-)

[Mikeal Rogers]

i found the pattern. i bet we can demonstrate it with graphs.

flow control mentioned -> fibers mentioned -> someone calls fibers (incompatible || a fork || "not node") -> Marcel defends fibers as "just a module" -> Oleg mentions common-node as "not just a module" -> Mikeal makes a disparaging remark -> Axel disagrees, promotes streamline.

if you start at any single point the other will always descend from it. sometimes Oleg and Mikeal are swapped in position.

[Bruno Jouhier]

John,

You may want to take a look at streamline.js.

It is a preprocessor that generates either pure callbacks or code for
node-fibers (thanks to Marcel who did all the hard work here).

I had started to investigate a third preprocessing option to target
generators but I stopped because I could only play with generators in
Firefox and they did not implement the full harmony stuff (especially
the generator delegation feature which make life a lot simpler with
nested callbacks). My gut feeling is that streamline source should be
easy to translate to generators but the proof is always in the
pudding...

Preprocessing can be viewed as a really ugly solution but it has one
big advantage: it allows you to change the targeted platform and take
advantage of future features without having to change your source
code.

Bruno

[john.tiger]

On 02/10/2012 02:35 PM, Mikeal Rogers wrote:

comments like this make me scratch my head.

there are thousands of developers here on this mailing list maintaining node code using callbacks. there are products that do daily deploys across thousands of node processes that use callbacks.

there are hundreds of thousands of liberal arts graduates using and maintaining code that uses jquery (with callbacks!).

if you think that callbacks are a blocker for maintainability i can't help but wonder why all these other people seem to be fine but the it's such a blocker for you that you would need to opt out of what the vast majority of the community are using, testing, and running in production every day.

I could be wrong but I think you live in a bubble - and it's why programmers are getting $150k per year - some of whom aren't worth 50k.   Obviously, I'm not the only one who sees the problem and why others have come up with modules and work arounds. 

@Alexey - yes, we have a lot of old code in Ruby and it's much simpler for others to quickly understand or for us to pick up after not working on it for 2 yrs - we could go back to ruby & sinatra but have gotten to like javascript and like the performance, scalability, and the use of one language (I don't want to get in a js/ruby argument here) - it's more an issue of cleaner synch wrapped code vs nested callback code.  

[Mark Hahn]

>  I could be wrong but I think you live in a bubble - and it's why programmers are getting $150k per year - some of whom aren't worth 50k.   

So, we are incompetent.  That is good to know

[Marco Rogers]

Sorry for misrepresenting. I hadn't heard of the change. Nice work.

:Marco

On Friday, February 10, 2012 10:56:43 AM UTC-8, Marcel wrote:

2) As of node 0.6.0, node-fibers is just a module. It never was a fork, but in node 0.4.0 it was much more of a hack. In node 0.6.0 it works just like any other module, you install it, require it, and use it.

3) Depending on how you use fibers, it may be possible to rewrite your code using generators, it may not be. Fibers will give you a superset of functionality and it's up to you to look at what generators will be and build your code to support them in the future if that's what you want.

On Fri, Feb 10, 2012 at 12:55 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

fibers and generators (as described in harmony) have a large delta between them.

if you read the harmony spec for generators you'll notice that the yield statement pushes it's way all the way up the call stack so you can't abstract it away the same way common-node does with fibers.

what you should probably do is just use normal node, the way 99% of the community does :)

On Feb 10, 2012, at February 10, 20121:35 PM, Marco Rogers wrote:

1) node will support harmony generators the second they are stable in v8. The language evolves with v8, node doesn't touch it.

2) To my knowledge, node-fibers is a fork of node with patches to both node and v8 to enable fibers. So you can't just run node, you have to run node-fibers.

3) I think it would be very easy to paint yourself into a corner here. But there are lots of abstractions on top of fibers that could possibly be implemented later with generators instead. Keep in mind that when you dig into the details, fibers seem to allow deeper semantic changes in node than the version of generators in harmony. I'm not the best person to explain the difference, but I've understood enough of the talk to know they are not equivalent paradigms. Generators enable a subset of functionality that fibers does.

Hope this helps.

:Marco

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

[john.tiger]

On 02/10/2012 03:35 PM, Mikeal Rogers wrote:
> i found the pattern. i bet we can demonstrate it with graphs.
>
> flow control mentioned -> fibers mentioned -> someone calls fibers (incompatible || a fork || "not node") -> Marcel defends fibers as "just a module" -> Oleg mentions common-node as "not just a module" -> Mikeal makes a disparaging remark -> Axel disagrees, promotes streamline.
>
> if you start at any single point the other will always descend from it. sometimes Oleg and Mikeal are swapped in position.

LOL - okay, since I started this, I'll try to end it - thks to all for
input. Takeaway: harmony generators probably not until end of year.
@Alexey's mongo stuff very interesting (certainly simple). We will look
at common-node as well. Node-Fibers similar but a bit different than
upcoming generators but apparently does work and does offer simpler
sequential operations. Personally, would rather deal with nested
callbacks than the weird js streamline stuff.

[Bruno Jouhier]

@john,

streamline is no weirder than CoffeeScript, which is also a
preprocessor.

Also, if you take the time to experiment with it, you will see that
the code generated in fibers mode is orders of magnitude simpler than
the code generated in callback mode. The fibers output is almost
identical to the streamline source. The only noticeable differences
are the async function declarations and async function calls that are
a little more verbose in the output than in the source. All the non-
async stuff is passed verbatim, line numbers are preserved, even
comments are preserved!

I expect the generators output to be very similar to the fibers
output. So, as your question was about a solution that prepares for
the future, I think that streamline qualifies. The day generators hit
the road, you'll just have to preprocess one last time and say goodbye
to streamline. The output of the preprocessor will be your new source.
And, unless I missed something big, this output will be, like the
current fibers output, clean and very close to your streamline source.

Bruno

[Mark Hahn]

>  streamline is no weirder than CoffeeScript, 

Except that streamline changes semantics and CS doesn't.  Sorry, I have to defend CS.

[Marco Rogers]

I'm totally staying out of the original discussion, but I do want to take a tangent here.

When people talk about "semantics" what do they mean exactly? In my mind, just because cs semantics are *compatible* with js semantics doesn't mean they are "the same". For example, the removal of the var keyword changes semantics when you're programming cs.

If I see this

function foo() {

  bar = "Hello World";

}

As long as there is no enclosing scope, I know that I've just created a global.  Those are the rules of javascript, love em or hate em. When I do the same in cs

foo = ->

  bar = "Hello World";

I do not have a global and there's no way to get one.

This is just a simple example, and you can argue that this is bad and you should never do this. I actually agree and I hope that proves I'm not trolling here. The question I'm asking is, isn't this different semantics? Is there a different word I'm thinking of to describe this? Somebody school me on the proper definition.

:Marco

[Chris Scribner]

Hi, I'm the author of a fibers abstraction called asyncblock.
(https://github.com/scriby/asyncblock).

I read over the details of harmony generators, and I can tell that
existing fibers based solutions will not be able to transparently
switch over to generators. There are two main problems:

1. The function needs to have an asterisk after it (not a huge deal)
2. The existing syntax can't work directly as you must use yield
directly from the generator function. It's not like fibers where we
can yield further up the call stack of a function running in a fiber

So, I think the only reasonable solutions for long term planning and
avoiding CPS style code involve syntax transformation. Asyncblock has
beta support for syntax rewriting, and I plan on basing it off
generators when they become available in v8.

The syntax rewriting lets you write code like

asyncblock(function(flow){
//defer makes the first access to file1 yield until it's ready
    var file1 = fs.readFile(path, 'utf8').defer();

//sync makes the fiber yield until the result is obtained
    var file2 = fs.readFile(path, 'utf8').sync();

    console.log(file1 + file2);
});

Right now that code compiles to use fibers. There's plenty of good CPS
transforms out there of you want to avoid fibers for now, or of you
need windows or browser support (be careful about losing line numbers
though).

Good luck making your decision,

Chris

[Jeffrey Zhao]

Jscex (https://github.com/JeffreyZhao/jscex) is easy to switch to harmony:

var concatAsync = Jscex.compile("async", function(path1, path2) {
// read first file
var file1 = $await(fs.readFileAsync(path1, 'utf8'));

// read second file
var file2 = $await(fs.readFileAsync(path2, 'utf8'));

console.log(file1 + file2);
}));

You can even make it parallel easily.

var concatAsync = Jscex.compile("async", function(path1, path2) {

var data = $await(Task.whenAll(
"file1": fs.readFileAsync(path1, 'utf8'),
"file2": fs.readFileAsync(path2, 'utf8')));

console.log(data.file1 + data.file2);
}));

No preprocessor, no external compilation, just run directly with any ES3
engines.

The doc in English is outdated but the samples
(https://github.com/JeffreyZhao/jscex/tree/master/samples/async) are always
fine:

Jeffrey Zhao
Blog: http://blog.zhaojie.me/
Twitter: @jeffz_cn (Chinese) | @jeffz_en (English)

console.log(file1 + file2);
});

Chris

--

[Liam]

@john.tiger, It seems that no one drilled down on your application.
What async routines do you expect to call in such a deeply nested way
that callback code becomes unwieldy?

There's often a way to write it such that it's not so deeply nested,
for example some async ops can be done concurrently and hit the same
callback. Or a sequence of database ops could be represented as an
array of queries that's handed to a routine that sequences them.

[Mark Hahn]

>  In my mind, just because cs semantics are *compatible* with js semantics doesn't mean they are "the same". 

It's a matter of degree.  Most of CS maps directly into JS keeping the exact same semantics even though the syntax is wildly different.  The best example is -> of course.  Scope assignment does have different semantics, but even that is quite mild.  

The big difference in my mind is that I can think of what the resulting JS will be almost perfectly when I read or write CS.  My mind isn't good enough to do that with streamline.

[Bruno Jouhier]

Jscex still does preprocessing (the Jscex.compile call). It just does
it on the fly.

Streamline works the same way, but the compile directive is hidden at
the top of the module.

// magic autocompile directive
if (!require('streamline/module')(module)) return;

// serial concat
function concatAsync(path1, path2, _) {
console.log(fs.readFile(path1, 'utf8', _) + fs.readFile(path2,
'utf8', _));
}

// trivial wrapper to get 'futures' around fs.readFile
function readFile(path1, encoding, _) { return fs.readFile(path,
encoding, _); }

// parallel concat
function concatAsyncParallel(path1, path2, _) {
var rd1 = readFile(path1, 'utf8');
var rd2 = readFile(path2, 'utf8');
console.log(rd1(_) + rd2(_));

[jeffz]

You're right. The Jscex.compile is the mark of a "processed function" rather than a normal function. It's designed to run with any ES3 engines without any external support, so it supports any JS environment.


老赵 ※ Jeffrey Zhao

Sent from my Samsung Galaxy Tab

[Jann Horn]

2012/2/11 Mark Hahn <ma...@hahnca.com>:

>>  In my mind, just because cs semantics are *compatible* with js semantics
>> doesn't mean they are "the same".
>
> It's a matter of degree.  Most of CS maps directly into JS keeping the exact
> same semantics even though the syntax is wildly different.  The best example
> is -> of course.  Scope assignment does have different semantics, but even
> that is quite mild.

Hmm... I wouldn't really say that. For example, there are loop
expressions and autoreturn. Also, the body of a "class" has some
magic.

> The big difference in my mind is that I can think of what the resulting JS
> will be almost perfectly when I read or write CS.  My mind isn't good enough
> to do that with streamline.

Hmm... I don't think that it's necessary to do that in either of those
languages. IMO, having to think about what the generated JS is or
having to look at it (e.g. for debugging) is a major bug, just like
having to look at the generated machine code that gcc creates from C++
code is a bug, too. (That's also why line number mapping for
coffeescript is being worked on, it's basically complete, just without
tests and with a bug that makes it not work correctly inside of string
interpolation. https://github.com/jashkenas/coffee-script/pull/2050 )
You only have to know what the code will do, not what it will look
like. And that usually is a lot easier. I can't tell you how
streamline works, but I grasped the basic concept "underscore is like
a blocking call that resumes on callback", and I think that this
should be sufficient. Are you going to say gcc is bad because it
optimizes code in ways you don't know or understand?

[Bruno Jouhier]

> > The big difference in my mind is that I can think of what the resulting JS
> > will be almost perfectly when I read or write CS.  My mind isn't good enough
> > to do that with streamline.

If you are not comfortable with the callback rewriting done by
streamline, you have the option of running it in "fibers" mode. In
this mode, the generated code is very similar to the source (and,
unlike CS today, line numbers and comments are preserved). For
example, take a look at:

https://github.com/Sage/streamlinejs/blob/master/lib/compiler/compile--fibers.js
https://github.com/Sage/streamlinejs/blob/master/lib/compiler/compile_.js

As Jann mentions, this is a general issue with compilers. At some
point, you just have to *trust* the compiler and the people behind it.
V8, for example, is applying all sorts of amazing tricks to our code.
I never felt the need to look at the instructions that it generates. I
just trust it, and, if one day I find something odd, I'll file a bug
report and I have confidence that the V8 gurus will fix it. Same thing
with CS, and with streamline.

Bruno

[Alexey Petrushin]

> there are thousands of developers here on this mailing list maintaining node code using callbacks.

But how is this different from "there are thousands Java and C projects, You can do whatever You want with it, why do You ever need another languages?"

I personally don't like to use control flow helpers, like asyncjs because it just partially solves the problem. Instead of callbacks You got state-machines. Instead of callbacks now You has to write state-machine for every single piece of logic in Your app.

That's the reason why I believe "heresy" technics like fibers and so on are interesting.

Because in some cases they allow You to get the best of the two world at the same time - simple async concurrency model and the simple, ordinary code.

Node core is fast, low level and simple - and it's ok. But isn't it good that we also have alternative high-level abstractions that can greatly simplify development in some cases?