[ANN] Suspend - Generator-based Control Flow

[jmar777]

suspend is a new control flow library that exposes a minimal API around ES6 generators, and is expressly designed to work transparently with Node's existing callback conventions.  This allows unobtrusive use of yield execution semantics that works seamlessly with existing Node code bases (no need to wrap everything in a promises/whatever layer).

Quick example:

var suspend = require('suspend'),

    fs = require('fs');

suspend(function* (resume) {  

    var data = yield fs.readFile(__filename, resume);

    console.log(data[1].toString('utf8'));

})();

Links: GitHub Repo | Blog Announcement

NPM: $ npm install suspend

suspend is extremely experimental, and I would greatly appreciate any feedback!

[jmar777]

Wow, sorry for the email formatting. I figured Google Groups could handle putting bold text and links into an email... :\

[alFReD NSH]

You just made my yesterday wish true. This is nicely done. Good job.

[Floby]

Yup, I've always told myself this would be the only kind of sync-style flow control lib I would ever use. very good work.

[jmar777]

Thanks for the feedback!  It seems to have all the right ingredients to be loved or loathed, so we'll see which it is...

I'm particularly anxious for destructuring assignment to be implemented.  Single line, zero-callback async without the array hackiness should really go a long way.  I also like the fact that you don't lose the "Hey, IO happens here!" syntactic indicator you get from callbacks, since the yield expression can serve the same purpose.

ES6 (and beyond) is definitely offering some promising features with regards to code elegance.  Glad to see the V8 team moving forward so fast once the specs hit draft :)

On Tuesday, May 28, 2013 10:20:06 AM UTC-4, jmar777 wrote:

[Jean Hugues Robert]

Hi,

This is interesting. You probably had at look at taskjs.org.  Generators are kinds of structured fibers that, yes, can be "abused" to provide some form of threading. How "unobstrusive" this can be remains to be fully explored, as you do, in good company (see also streamline implementation using generators, an interesting exploration too).

Your approach reminds me about the "await" construct in C#, the"yield" in your example really means "await".

After much thinking, I came to the conclusion that this flow control issue is not such a big deal, solutions like "promise/a" demonstrates that async activities can be orchestrated in a readable way without threads. Besides, early implementations of generators are apparently rather slow (fibers are fast, non standard... and controversial).

I am conducting an experiment about a similar control flow problem, here is one result:

  var Parole = require('l8/lib/whisper'), fs = require( 'fs' );

 

  var read = Parole(); fs.readFile( __filename, read );

  read.on( function( err, data ){ console.log( data.toString( 'utf8' ); } );

  // or: read.then( function( data ){ console.log( data.toString( 'utf8' ); } );

This is more concise, does not have the array destructuring issue, runs today everywhere and is readable once you understand that a Parole() is an extended Function suitable as a node.js callback.

See https://github.com/JeanHuguesRobert/l8/wiki/ParoleReference. You may be surprised by the P.generator() easy way of defining an async generator using today's javascript.

Yours,

    Jean Hugues

 

[alFReD NSH]

If you guys want you can put a star on the v8 issue for destructuring assignment:https://code.google.com/p/v8/issues/detail?id=811

It might help for the issue to get some attention and get implemented sooner.

[Floby]

Also, while very readable, using the yield operator forbids to run async tasks in parallel. Correct me if I'm wrong. It seems the only way to do so is to use multiple calls to `suspend` which in the end is very much like using a callbacks anyway.

On Tuesday, 28 May 2013 16:20:06 UTC+2, jmar777 wrote:

[jmar777]

Hey Jean - 

> You probably had at look at taskjs.org.

Ya, I've been tracking task.js for awhile now - just haven't delved in too deeply since node is realistically the only platform generators are going to be practical on in the near future (browser support is obviously pretty sparse still).  On the one hand, task.js is awesome and deserves the hype it's received.  On the other hand, I find the promises layer to be a deterrent against using it for Node applications, since the majority of the node ecosystem is based on passing callbacks/continuations, rather than promises.  That's probably the most fundamental difference between suspend and task.js: suspend is designed to be 

> After much thinking, I came to the conclusion that this flow control issue is not such a big deal, solutions like "promise/a" demonstrates that async activities can be orchestrated in a readable way without threads.

I agree that promises *can* improve the situation, but I think it has to be all-or-nothing.  The random hodgepodge of promises use in node modules is problematic - it's not the fault of promises, of course.  It's just confusing when/where you need to .then() vs. pass a callback.  Regardless of your feelings towards promises, though, it's important to recognize that they're still just an abstraction on top of callbacks - you're either passing a callback to the method itself, or you're passing a callback to some method on your promise/future/deferred/etc.  This is why generators are particularly interesting with regards to async code: they're the first *language* level construct in JS that allows this async behavior without relying callbacks.

> I am conducting an experiment about a similar control flow problem

Parole looks really interesting!  I'm going to give the source a more thorough reading once I have time, but on first glance I like that it thinks outside of the box, compared to 90% of the other control flow libraries out there.

> This is more concise, does not have the array destructuring issue, runs today everywhere and is readable once you understand that a Parole() is an extended Function suitable as a node.js callback.

Given where destructuring assignment and generators currently stand in V8/node, you're probably right for now, but remember that my goal is to get ahead of the curve here in anticipation of these upcoming language features.  I think I would still prefer the following:

    var resume = require('resume'), fs = require('fs');

    suspend(function* (resume) {

        var [err, buffer] = fs.readFile(__filename, resume);

        // no callbacks or .on()'s or .then()'s, just use it!

        console.log(buffer.toString('utf8'));

    });

True, you have to pay the suspend/generator block, but that's a tax you only have to pay once, and you can have as many async calls in there as you'd like with no additional "taxation".

Thanks for the feedback, and I'm looking forward to hacking around with Parole!

- Jeremy

[jmar777]

> If you guys want you can put a star on the v8 issue for destructuring assignment:https://code.google.com/p/v8/issues/detail?id=811

Yes, please do!  This adds virtually no new capabilities to the language, but from a practicality perspective, this is probably one of the niftiest ES6 features.

[jmar777]

> Also, while very readable, using the yield operator forbids to run async tasks in parallel. Correct me if I'm wrong. It seems the only way to do so is to use multiple calls to `suspend` which in the end is very much like using a callbacks anyway.

I've addressed this in the readme: https://github.com/jmar777/suspend#what-about-parallel-execution-mapping-etc

If I can arrive at a really elegant API for suspend to provide parallel and other "advanced" control flow operations, it probably will.  For now, however, it's intentionally designed to be easily interoperable with your existing control-flow libraries of choice.  Here's the example from the readme:

    // async without suspend

    async.map(['file1','file2','file3'], fs.stat, function(err, results){

        // results is now an array of stats for each file

    });

    

    // async with suspend

    var res = yield async.map(['file1','file2','file3'], fs.stat, resume); 

[Bruno Jouhier]

Look like your start function only handles one yield per function. What if you want to make several async calls from the same function? How do you handle several levels of async calls (async f1 calling async f2 calling async f3)?

I just published a module that handles these cases  (https://github.com/bjouhier/galaxy) but I had to use a much trickier `run` function for this.

Bruno

[Jeremy Martin]

Not sure I follow what you mean by "one yield per function".  The following works, if that's what you mean:

    var suspend = require('./'),

        fs = require('fs');

    suspend(function* (resume) {

        // read the current file

        var res = yield fs.readFile(__filename, { encoding: 'utf8' }, resume);

        // replace tabs with spaces

        var newContents = res[1].replace(/\t/g, '    ');

        // write back changes

        yield fs.writeFile(__filename, newContents, resume);

        // print modified file

        var modified = yield fs.readFile(__filename, { encoding: 'utf8'}, resume);

        console.log(modified[1]);

    })();

This performs 3 asynchronous operations within the same generator.  Nothing prohibits nesting, either (again, not entirely sure what you mean, but the following works as well):

    var suspend = require('./'),

        fs = require('fs');

    var readCurrentFile = suspend(function* (resume, cb) {

        var res = yield fs.readFile(__filename, { encoding: 'utf8'}, resume);

        cb(null, res[1]);

    });

    var writeToCurrentFile = suspend(function* (resume, data, cb) {

        yield fs.writeFile(__filename, data, resume);

        cb(null);

    });

    suspend(function* (resume) {

        var res = yield readCurrentFile(resume);

        var newContents = res[1].replace(/\t/g, '    ');

        yield writeToCurrentFile(newContents, resume);

        var modified = yield readCurrentFile(resume);

        console.log(modified[1]);

    })();

Congrats on getting galaxy out, looking it over now!  Curious about the fanciness you had to include in the `run` function.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/jfOq8mpaqss/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Jeremy Martin
661.312.3853

http://devsmash.com

@jmar777

[Jeremy Martin]

Bruno - 

After looking over galaxy for a bit, I'd say suspend and galaxy's philosophies are pretty different.

That is, suspend's API is designed to work directly with node's callback conventions (where `resume` acts as the continuation), whereas galaxy's API allows you to wrap code that uses node's callback conventions (using `galaxy.star()`).

For example, in galaxy, the `galaxy.star()` method does the API conversion in convertAPI():

    return Object.keys(api).reduce(function(result, key) {

        var fn = api[key];

        result[key] = (typeof fn === 'function' && !/Sync$/.test(fn.name)) ? converter(fn, idx) : fn;

        return result;

    }, {});

It appears, then, that in order to use galaxy.js with existing code, you need a shim between it and galaxy. This shim makes some assumptions, such as whether or not `Sync` is in the name.  So, would this convention need to be followed by user-land modules as well to work with galaxy?  Also, it seems to only convert top-level exports.  You'll have to weigh between the performance tradeoffs there, but I've seen modules that namespace the functionality, which would appear to thwart convertAPI() at the moment.  Anyway, still digging through the source - it's interesting, but apart from the use of generators, it looks like suspend and galaxy are a bit like apples and oranges.

[jmar777]

Alex Young was kind enough to do a writeup over on DailyJS: http://dailyjs.com/2013/05/31/suspend/.

Worth reading for the overview on generators alone.

On Tuesday, May 28, 2013 10:20:06 AM UTC-4, jmar777 wrote:

[Bruno Jouhier]

Hi Jeremy,

I had completely missed the fact that you are wrapping every generator function that you write with a `suspend` call. So I concluded too quickly that your implementation was rather naive. But it's not and it's actually very clever because it does a lot with very little code (16 lines only vs. 159 for galaxy).

As you say galaxy takes a different approach. My goal was to obtain the leanest code possible when writing galaxy code that calls other galaxy code and I was happy because I managed to get 0% fat: galaxy code that calls other galaxy code is just as lean as JS that calls JS if JS had async/await keywords: async = * and await = yield. There is no galaxy API in the middle (except for parallelizing). The galaxy functions are just used at both ends of the call stack: `star` when you call low level node APIs and `unstar` when your functions are called by node.

Note that you don't need to shim entire modules to call them, you can shim individual functions. For example:

    var fs = require('fs');
    var star = require('galaxy').star;
   
    function* countLines() {
        var contents = yield star(fs.readFile)(__filename, 'utf8');
        return contents.split('\n').length;
    }  

I feel that the two approaches are rather complementary:

• Suspend seems most appropriate for libraries: you don't have to shim node.js calls and the APIs that you create are directly exposed as node callback APIs.
• Galaxy is more targetted at applications: most of the code that you write calls your own APIs and galaxy is the leanest and least intrusive in this case.

Bruno