trying to wrap my head around "promises" - async vs Q

[Andy]

Note, I am not asking which tool is better, I am simply trying to understand the differences.

I'm trying to wrap my head around promises in node. Right now I'm writing all my code in callback soup. I am researching libraries and I found async (duh) but I also found the horribly named but seemingly very popular q.

What I am trying to figure out is if these libraries are mutually exclusive. The async page mentions nothing about "promsies" and instead talks about "flow control." But it seems like both libraries are sugar for handling async function flow and callbacks. Do they both solve the same problem, or can / should they be used together?

Take this example:

async.waterfall([
    function(callback){
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback){
        callback(null, 'three');
    },
    function(arg1, callback){
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
   // result now equals 'done'    
});

vs:

Q.call(step1)
.then(step2)
.then(step3)
.then(step4)
.then(function (value4) {
    // Do something with value4
}, function (error) {
    // Handle any error from step1 through step4
})
.end();

Both libraries are doing things in a series, and both are passing their results to the next function. Is there really any difference between the two results other than Q returning a promise that you can chain to with .then?

Is async really just a more versatile q? Or are there reasons to use one and the other and they could be used together?

And can you do parallel functions with promises? Or is that not what they're used for? (And if not, should you use async + q, or is there too much overlap?)

[Axel Kittenberger]

AFAIK they should be working together, but why would you need two
times the same thing with differnt dependencies? Also note that there
isn't an established vocabulary what is a "promise", "future", "flow
control" and the differences.

There are even far more libraries just like this, and even yet more
solutions approaching the problem from different .

This is the usual experience, "node.js oh so great" --- a day later,
"OMG I can't handle all that callbacks, statemachine logics etc.". One
Week later: yet another syncronisation library has been submitted to
teh repository.

> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

[Jeff Barczewski]

Andy,

They are related by the fact that both are trying to help remove some of the pain of developing asynchronous code. They are different in the way they go about implementing this. 

Check out the articles on the How to Node blog that Tim Caswell runs, there are some good examples with explanations on why and how you can use each of these.

 - http://howtonode.org/control-flow

 - http://howtonode.org/control-flow-part-ii

 - http://howtonode.org/control-flow-part-iii

 - http://howtonode.org/step-of-conductor

 - http://howtonode.org/promises

After you have tried some of these different approaches, you will probably find that one or the other works best for you, so you will want to do most of your development that way. 

It is not to say that you might not use both from time to time, but you probably will want to minimize switching mindset from one to the other since they work different.

All the best,

Jeff

[Mariusz Nowak]

@Andy Async is just sugar for control flow written with plain callbacks and promises address asynchronicity from very different angle.

In promises approach asynchronous state is represented with the object, so instead of registering single callback, you get the object, which you can pass to many different functions, or on which you can listen for value with many different listeners. Promises also provide clean separation of success and error flows. It's much more powerful than plain callbacks, but also takes some time to get familiar with that. Once you get it, you will never go back ;-)

I've once done introduction to promises speech. See http://www.medikoo.com/asynchronous-javascript/ (promises starts at 16 slide) 

-- 
Mariusz Nowak

https://github.com/medikoo

http://twitter.com/medikoo

[Domenic Denicola]

I'd like to submit my promises talk for consideration as well :). I agree with Mariusz that once you get it you will never go back.

http://www.slideshare.net/domenicdenicola/callbacks-promises-and-coroutines-oh-my-the-evolution-of-asynchronicity-in-javascript

[Alexey Petrushin]

There are some good use cases for control flow libraries, but, sadly, in most cases the end result is even worse than without it. 

It seems there's no good solution to this problem - code looks ugly no matter what You do - with or without control flow libraries.

So, I believe currently the "fuck that" approach is the best option. Just write ugly waterfall code and don't care about it very much ;)

[Mark Miller]

On Sun, Jul 1, 2012 at 3:08 PM, Ken Whitesell <kenwhi...@comcast.net> wrote:

On Sunday, July 1, 2012 2:55:05 PM UTC-4, Mariusz Nowak wrote:

n promises approach asynchronous state is represented with the object, so instead of registering single callback, you get the object, which you can pass to many different functions, or on which you can listen for value with many different listeners. Promises also provide clean separation of success and error flows. It's much more powerful than plain callbacks, but also takes some time to get familiar with that. Once you get it, you will never go back ;-)

Superficially, this looks a lot like Python's Twisted framework with how they work with deferreds and callbacks. Are you familiar with it? Are the parallels appropriate, or is there something significantly different that I'm missing here? (I've been working with twisted for over 5 years now, so I'm _really_ comfortable with it.)

Both Twisted deferreds and the Q promise library are based on E's promises[1]. Twisted is only about support for local asynchrony, whereas E's and Q's promises, for the same cost, also provide for distributed secure object programming.

[1] http://erights.org/talks/promises/paper/tgc05.pdf 

--
Text by me above is hereby placed in the public domain

  Cheers,
  --MarkM

[Mark Miller]

Thanks Domenic, this is indeed an excellent presentation. I especially like your emphasis on getting the exception propagation right. It's something I should emphasize more as well.

--

Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[Bruno Jouhier]

Hi Dominic,

I looked at your slides. I think you should also take a look at streamline.js.

An interesting benchmark of async solutions would be to take the little tutorial that I just wrote (https://github.com/Sage/streamlinejs/blob/master/tutorial/tutorial.md - or any similar kind of app) and try to write it in different styles: pure callbacks, async libraries, promises, fibers, harmony generators, etc.

My guess is that fibers and its futures library (and generators with the proper twist - see http://bjouhier.wordpress.com/2012/05/18/asynchronous-javascript-with-generators-an-experiment/) will allow you to keep the same simple logic with just a little more code; but that pure callbacks and library based solutions (even the most clever ones like async and promises) will lead to heavier code.

IMO, asynchronism must be dealt with at the language level. Libraries can help but they will solve part of the problem. For a real fix, you need a compiler (or a preprocessor).

Bruno

[P. Douglas Reeder]

The article
http://www.infoq.com/articles/surviving-asynchronous-programming-in-javascript
compares seven libraries (including asynch, but not q).

[Alan Gutierrez]

I've come to depend on two helper functions...

function validator (callback) {
return function (forward) { return check(callback, forward) }
}

var __slice = [].slice;
function check (callback, forward) {
return function (error) {
if (error) {
callback(error);
} else {
try {
forward.apply(null, __slice.call(arguments, 1));
} catch (error) {
callback(error);
}
}
}
}

When I start a function I create a check function like so...

var check = validator(callback);

... or else if I only call it once, I use `check(callback, func)`.

Then I use hoisted functions to name the steps in my procedure. This is
more verbose than a control flow library, but it is pretty easy for to
read and therefore maintain. Calling hoisted functions by name prevents
the formation of a template of doom.

The `check` function saves the trouble checking errors, which saves an
if/else block and therefore saves a layer of doom.

It's verbose, but I don't feel that it's ugly.

var fs = require('fs');
function deltree (directory, callback) {
var files, count = 0, check = validator(callback);

fs.readdir(directory, extant);

function extant (error, $1) {
if (error) {
if (error.code != 'ENOENT') callback(error);
else callback();
} else {
list($1);
}
}

function list ($1) {
(files = $1).forEach(function (file) {
stat(path.resolve(directory, file));
});
deleted();
}

function stat (file) {
var stat;

fs.stat(file, check(inspect));

function inspect ($1) {
if ((stat = $1).isDirectory()) deltree(file, check(unlink));
else unlink();
}

function unlink () {
if (stat.isDirectory()) fs.rmdir(file, check(deleted));
else fs.unlink(file, check(deleted));
}
}

function deleted () {
if (++count > files.length) fs.rmdir(directory, callback);
}
}

With a control flow library, I find that code gets cryptic when it gets
complicated; precisely the time it is best to be clear. The real work is
being done behind a curtain. I miss the names when they aren't there.

This is complexity I want to see.

--
Alan Gutierrez - http://twitter.com/bigeasy - http://github.com/bigeasy

[José F. Romaniello]

I saw your slides and I cant agree more with you. The other day I did some thinking about all the javascript code I have been writing and I came to this conclussio (bear with me, please):

- for me the problem with CPS (continuation passing style) for asynchronous code is not the cascade of nested callback. The code gets ugly when you mix CPS constructs with non-CPS constructs. Think when you have to do something async while looping something, or when you have to do something async IF some condition is true, and then something else even when the condition is not met.

- javascript has ifs, for, and lot of constructs that are non-CPS. In Clojure and LISPs everything could be CPS, so in this regard we can say is "better".

-tamejs, iced coffee script, f# (with its async workflows) etc has constructs to avoid CPS for asynchronous code and one of the best things of this is that non-CPS IF statements play nice with ... non cps async code.

I wrote an example in my blog about this http://joseoncode.com/2012/06/24/messing-with-cps-in-js/ where i explore a way to create a CPS IF statement and it makes the code easier than with the standar IF statement. Another thing Id like to have is a language with nice syntax for asynchronous code (as your await example) that compiles down to LISPish javascript.

--

[Bruno Jouhier]

Hi José

Transforming *every* construct of Javascript is exactly what streamline.js does. Here is a typical piece of code:

function mongoSearch(_, q) {
    var t0 = new Date();
    var db = new mongodb.Db('tutorial', new mongodb.Server("127.0.0.1", 27017, {}));
    db.open(_);
    try {
        var coln = db.collection('movies', _);
        if (coln.count(_) === 0) coln.insert(MOVIES, _);
        var re = new RegExp(".*" + q + ".*");
        return coln.find({
            $or: [{
                title: re
            }, {
                director: re
            }]
        }, _).toArray(_).map(function(movie) {
            return movie.title + ': ' + movie.director;
        }).join('<br/>') + '<br/>completed in ' + (new Date() - t0) + ' ms';;
    } finally {
        db.close();
    }
}

It contains 6 asynchrounous calls to the mongodb API: db.open, db.collection, coln.count, coln.insert, coln.find and toArray mixed with standard javascript constructs: if, return, chaining (a(_).b()), even try/finally.

There is a lot of talking with lots of opinions on this topic, but not enough facts. I think that a bench with a realistic application use case would bring clarity to the debate.

Bruno

nodejs+unsubscribe@googlegroups.com

[Andy]

To reply to my own very old thread after getting some solid experiences with promises, the answers to my questions are:

1. async is a library that passes callbacks around. it's ugly and it sucks.

2. a promise is just an object. Don't let anyone tell you differently, they are just trying to confuse you. It's an object that has method names that everyone's agreed on, like then and done which will magically trigger your callbacks for you. With promises, you include Q and you just pass around Q objects (called deferreds and promises). It's just that everyone agreed that the promise object will have a .then method, which you can call and your function jumps next in line onto the promise chain magically.

The libraries should not be used together. Once you get your head around promises and use them in field you won't want to use anything else.

If you are exposing an API, you should still take/call a callback. If you're writing a database client you don't want to give the user a promise and force them into your model. Just call their callback at the end of your own, internal beautiful promise chain. It will be our terrible secret.

Promises aren't perfect and can get strange when doing some complex composition (lots of return statements to return promise chains), but they make writing async code soooooo nice.

[Mikeal Rogers]

It's great that you have a strong opinion. Here's some numbers:

Here's the number of modules that depend on async and Q.

async: 975

Q: 109

Here are the numbers of downloads in the last month.

async: 120,271

Q: 33,242

Some people clearly like promises, but the dominant pattern in node is not promises, it's callbacks, with complex callback issues managed with async.

Stating your opinion strongly does not make it a fact. This is your preference, and many others, but not the majority.

If you write a library, it better use callbacks if you want people to use it. Using callbacks in your own application code is the path of least resistance for using the majority of value in the node ecosystem. That's a fact, there are numbers. It's not everyone's preference, but it's the most popular by far.

-Mikeal

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+un...@googlegroups.com

For more options, visit this group at

http://groups.google.com/group/nodejs?hl=en?hl=en

[greelgorke]

my approach to handle callback sallad? use the language features. just wrote down a script that would have a 7-10 callbacks nested. The problem with the callbacks is not the callback approach. it's simple and beautiful. the problem is naive and blind utilization of the closure scope. Just an example:

function lolAssert(){

  var forClojure = 'lol'

  require('child_process').exec( 'echo lol', function(err, stdout,stderr){

    require('assert').ok(~stdout.indexOf(forClojure))

  })

}

exec callback utilizes a clojure scope directly. in most cases you can easily avoid this:

function execCB(err, stdout,stderr){

    require('assert').ok(~stdout.indexOf( this ))

}

function lolAssert(){

  var forClojure = 'lol'

  require('child_process').exec('echo lol', execCB.bind(forClojure))

}

you can write it this way all way down. in most cases you even need the clojure scope/ this-binding. The script i wrote is nothing more that a number of small functions with no cb-nesting, no dependencies to 3rd party libs. this approach is perfectly usable for most of use cases, since the node callback pattern does not utilize this bindings. jQuery is different.

[Mariusz Nowak]

If I would choose which tool is best for my project just by popularity factor I wouldn't go far.

It's understood that Async is more popular as it doesn't introduce much learning factor and most developers are looking for quick solutions. To get promises you need to devote a while understand it. When you'll do it, you'll see that abstraction provided by promises gets you level higher, where you can easily produce complicated flows which are difficult to configure just using callbacks style.

Callbacks are base, and simple asynchronous function should definitely speak that convention. Moment where promises become significantly helpful is when you want to bind tens of asynchronous operations.

Other important thing, while Q plays significant role in bringing promises to Node.js world (it's also library that inspired many others), it's good to know there are other promise solution which I think in many areas play better. So judging promises just by trying Q implementation, is also limited approach.

[Bradley Meck]

Choose either, I do things in callback style personally to match the ecosystem when I write libraries, thats the expectation of the ecosystem.

For application logic, do whatever makes it faster. There are exactly 0 absolute wins on either side. I can go on at length about the reason both have pitfalls, but they both do. Programming is about compromise generally, this is no different. Readability at the code of some interesting shared state, go promises. Explicit state at the cost of repetitive error handling (much alleviated with domains), go callback.

Just remember for small cases of N terrible sorts are faster than sorts that have better O notation. For different problems, use different solutions.

[Nuno Job]

> If I would choose which tool is best for my project just by popularity factor I wouldn't go far.

I have a way to solve this issue that emulates pattern matching in javascript `github/dscape/p` while maintaing valid JS semantics.

Its sooooo unpopular even I dont use it :) (but it works.... I think!)

Still i think pattern matching is the best thing since sliced break. But will it be the way people write programs in node? (Unfortunately) No.

In a more serious tone, this is about our module ecosystem and node core. We settled on streams and callbacks. We have tried promises plus a million other crazy things but as in everything in callbacks and streams were the choice of the community. @mikeal's point was about that, everyone uses async in their modules in npm showing what our ecosystem adopted. Is it not evident that these choices are hugely correlated to the success of npm and why its so easy to use modules?

You can do other choices but will people the use your modules? Will this trend change in the future? Probably not.

Nuno

[Andy]

One of them problems even with this approach is that you write and read code backwards. It executes bottom to top because you have to define named callback functions first. I find it much more natural with promises because not only is your code organized, but it reads top to bottom and is MUCH easier to trace flow.

[Andy]

On Sunday, November 11, 2012 10:46:05 PM UTC-8, Mikeal Rogers wrote:

Some people clearly like promises, but the dominant pattern in node is not promises, it's callbacks, with complex callback issues managed with async.

Stating your opinion strongly does not make it a fact. This is your preference, and many others, but not the majority.

I'm aware of the much larger async userbase on github. In my opinion this is because promises are a more complicated concept to learn and thus scare people off and reduce adoption. It's different than native javascript so there are people who do not want to explore the option.  

If you write a library, it better use callbacks if you want people to use it.

I agree, as I said if you use expose an api you should still handle user responses with callbacks. but internally non promised code feels alien to me. I wouldn't give a user a promise as a response from my npm library.

[Domenic Denicola]

An interesting alternative is to use Q’s new “nodeify” function:

 

function returnsPromiseOrCallsback(callback) {

  return Q.resolve(5).nodeify(callback);

}

 

If callback is a function, it will call callback(null, 5); otherwise it will return the promise for 5. Similarly it will pass rejection reasons as the first parameter (i.e. as an error).

 

This allows you to create libraries that expose both promise-returning and callback-accepting APIs at the same time.

--

[Matt]

On Mon, Nov 12, 2012 at 5:48 PM, Andy <delva...@gmail.com> wrote:

One of them problems even with this approach is that you write and read code backwards. It executes bottom to top because you have to define named callback functions first. I find it much more natural with promises because not only is your code organized, but it reads top to bottom and is MUCH easier to trace flow.

In JavaScript the order you define things doesn't matter.

[Mark Hahn]

> because you have to define named callback functions first.

No you don't.  I always name then in order ...

func1 = ->  do something; func2()

func2 = -> do something; func3()

func3 = -> do something

func1()

--

[Andy]

 

In JavaScript the order you define things doesn't matter.

 

It does if your function declaration style is var funcName = function() { }; which is the style I use. Personal preference obviously.

[Mark Hahn]

Read the message before this.  I explained how it works in a forward way with your preferred method of function definition.  That is what I always use.

On Mon, Nov 12, 2012 at 3:46 PM, Andy <delva...@gmail.com> wrote:

 

In JavaScript the order you define things doesn't matter.

 

It does if your function declaration style is var funcName = function() { }; which is the style I use. Personal preference obviously.

--

[Mark Hahn]

P.S.  Coffeescript only supports that way of defining a function.

[greelgorke]

it's not backwards. first function is on top, last at the bottom, functions, which may appear multiple times in the callback chain are at the top most position by convention. i use the function declaration statement for this, because they get hoisted and the definition order is irrelevant then. There is no problem with named functions, they are good for stack-traces anyway. here just a snippet:

function rebuild(){

  checkErr.apply(null,arguments)

  vLog('rebuild dependencies...')

  exec("npm rebuild", dbJSON)

}

function dbJSON(){

  checkErr.apply(null,arguments)

  vLog('database.json...')

  exec("cp -f config/database.shared.json config/database.json", createLogFolder)

}

function createLogFolder(){

  checkErr.apply(null,arguments)

  vLog('create log folder...')

  if (fs.existsSync('./log')) prepareDistribution()

  else exec("mkdir log && touch log/dummy_file", prepareDistribution)

}

function prepareDistribution() {

  checkErr.apply(null,arguments)

  vLog('prepare distribution...')

  exec('git add . -Af && git commit -m "deployment script done preparation"', deploy)

}

its a part of a script that just runs on bare node. it's an approach you always can be used, in every node environment. it just use functions with well defined scope, they can be easily reused. the disadvantage of this approach is that you need to use bind or shared variables in outer scope instead of clojure scope. bind is much slower in v8, and shared variables may be evil.

[Baz]

Using callbacks in your own application code is the path of least resistance for using the majority of value in the node ecosystem. 

I am new to Node and trying to decide between promises, asynch and vanilla, there are so many good arguments for each. Mikeal, do you mind expanding further how using promises in your own, non-shared, code could hinder use of node? Do a lot of the libs that depend on asynch require you to use asynch as well (or make things easier if you do)?

Best,

Baz

On Sunday, November 11, 2012 10:46:05 PM UTC-8, Mikeal Rogers wrote:

[Matt]

On Thu, May 23, 2013 at 3:29 AM, Baz <b...@thinkloop.com> wrote:

I am new to Node and trying to decide between promises, asynch and vanilla, there are so many good arguments for each. Mikeal, do you mind expanding further how using promises in your own, non-shared, code could hinder use of node? Do a lot of the libs that depend on asynch require you to use asynch as well (or make things easier if you do)?

I think the problem with promises is that the core node libraries aren't implemented with promises, so if you need to do a series of e.g. fs calls (typical example might be open, then stat, then create a readStream from the fd) you wind up using async anyway, or manually coding up the nested callbacks, which can feel ugly once you've started using a particular flow control library. And then once you start using async for one thing, it just starts to feel natural. It just fits so perfectly with the Node ecosystem.

In short: There's nothing wrong with the promises model (although I find it overly verbose), it's just that async fits better with node libraries.

Matt.

[Baz]

Thanks Matt, so just to be crystal clear, if I'm using promises in my own code, then I happen to have a flow that depends on fs, for example, and since fs doesn't return or use promises, I would have to drop out of the promises paradigm and manage that particular part of the flow control with callbacks or something like asynch?

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 

---

You received this message because you are subscribed to the Google Groups "nodejs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Domenic Denicola]

That's overstating it a bit. It's very easy to convert to promise-based code, e.g.:

var fs = require('fs');

var Q = require('q');

var readFile = Q.denodeify(fs.readFile);
var writeFile = Q.denodeify(fs.writeFile);

readFile('src.txt').then(function (result) { writeFile('dest.txt', result); }).done();

Since this conversion process is pretty mechanical, there are several libraries that have done it for you, e.g. https://npmjs.org/package/pr

---

From: nod...@googlegroups.com [nod...@googlegroups.com] on behalf of Baz [b...@thinkloop.com]
Sent: Thursday, May 23, 2013 13:28

To: nod...@googlegroups.com
Subject: Re: [nodejs] Re: trying to wrap my head around "promises" - async vs Q

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/mWtqFE0spWk/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[// ravi]

On May 23, 2013, at 1:28 PM, Baz <b...@thinkloop.com> wrote:
> Thanks Matt, so just to be crystal clear, if I'm using promises in my own code, then I happen to have a flow that depends on fs, for example, and since fs doesn't return or use promises, I would have to drop out of the promises paradigm and manage that particular part of the flow control with callbacks or something like asynch?
>

Q provides you mechanisms to wrap non-promise returning calls (for both async/blocking/callback scenarios and — this is very useful — for functions that return immediately as well). If you are looking at these sort of libraries, I also heartily recommend Flow, which has the easiest learning curve IMHO: https://github.com/willconant/flow-js.

—ravi

[Baz]

Thank you gents, perfectly got it now.

[Jean Hugues Robert]

Le jeudi 23 mai 2013 19:32:25 UTC+2, Domenic Denicola a écrit :

That's overstating it a bit. It's very easy to convert to promise-based code, e.g.:

var fs = require('fs');
var Q = require('q');

var readFile = Q.denodeify(fs.readFile);
var writeFile = Q.denodeify(fs.writeFile);

readFile('src.txt').then(function (result) { writeFile('dest.txt', result); }).done();

Since this conversion process is pretty mechanical, there are several libraries that have done it for you, e.g. https://npmjs.org/package/pr

A different approach:

  var read = Parole(); readFile( "dest.txt", "utf8", read );

  read.then( function( r ){....} );

 

Parole() is an "npm l8" subset. See https://github.com/JeanHuguesRobert/l8