Wrapping My Head Around Promises

[Paul Tiseo]

I am definitely not new to software development, but I am new to extensive asynchrony and Node.js. I generally have a superficial understanding of Promises and their uses, but have yet to find a complex examples that put it all together for me in a tangible way. I'd like to try the when.js library, but I'm having a hard time getting practical with it. Hoping someone can assist?

Here's my live situation. I set myself a goal to create a REST endpoint using restify.js. The resource endpoint is Signups (from a website, let's say). When one posts an email to /signups, the handler must validate the inbound email, read a "thank you" email template from a file (using fs), send it (using nodemailer) and record the signup in a MySQL (using mysql) database. The first validation step is not asynchronous, but the file, DB and email steps are. I'm looking to avoid the callback pyramid and also create robust code that doesn't fall all over itself with these three libraries.

How does one code this up with when.js?

[Charilaos Skiadas]

I would probably structure it something like this (mind you I'm rather new to it all too):

Say you have a validateMessage method. You're saying it's synchronous, and that's fine, when.js does not mind. What it would expect is that validateMessage will either return the validated value or throw an exception (not 100% sure if throwing is the right thing in this case, maybe someone can confirm). If you then change it to a promised based method, it will all still work. So you can start with something like this:

when(validateMessage())

This returns a promise that will be fulfilled with the return value of validateMessage, i.e. right away in this case, or be rejected if validateMessage throws an exception. Wrapping it in "when" will turn the value into a promise basically.

Now next up you'll want to use fs.readFileSync. To make it promise-aware, you want to use nodefn.call and friends. So you should be able to read the template with something like:

var readTemplate = nodefn.lift(fs.readFileSync, templateName);

This doesn't exactly read the template, but it turns it into a function that when called reads the template returns a promise instead of expecting a callback.

Ok so now we have two functions, validateMessage and readTemplate that return for us the two strings we'll use to form the response, and to record things in database. Say now you have a function:

function sendThankYou(message, template) { ... }

and a function:

function recordSignup(message) { ... }

Let's say you want to run then in order. You can then do (there are probably lots of ways to do this part, perhaps more elegant ones too):


when.join(validateMessage(), readTemplate())
.then(function(values) {
return when.sequence([sendThankYou, recordSignup], values[0], values[1]);
})
.then(success, error);

The "success" function at the end would run if everything runs smoothly, and will receive the return values of sendThankYou and recordSignup as arguments. the "error" function will I believe be triggered if any of the 4 functions involved threw an exception.

You can also probably use when/apply to make that look a bit more elegant.

Hope someone corrects me if I'm wrong.

Now what I've left out is the code for sendThankYou and validateMessage. If the corresponding modules follow one of the cases described in https://github.com/cujojs/when/blob/master/docs/api.md#interacting-with-non-promise-code then you should be able to hook it up similarly to what I did with nodefn.lift earlier. It looks like nodemailer's sendmail method does use the node callback convention, so nodefn.lift should work. mysql should be similar.

>

[Brian Cavalier]

Hi Paul,

Welcome to the world of promises :)

Very good information and ideas on different approaches from Charilaos.  I'll just add a few thoughts and a few more helpful (hopefully) links.

In general, promise-based async programming can end up looking quite similar to sync code, and allowing you to reason about it, test it, and refactor in many of the same ways.  That is really the kernel of why promises are a good thing: they give you back the ability to reason about async code in a way that is more like sync code.  They give you back `return` and `throw`-like semantics and error propagation.

You may find the Learning Promises and Async Programming tutorial sections at know.cujojs.com useful: http://know.cujojs.com/tutorials

Charilaos mentions when/node/function, and with good reason.  It's purpose is to adapt node APIs to work with promises.  It can take the *asynchronous* versions of APIs like fs.readFile (rather than fs.readFileSync) and make them return promises.  For example:

fs.readFile(fileName, function(e, fileContents) {

    // callback soup commences

});

becomes like synchronous code, but still executes asynchronously:

var nodefn = require('when/node/function');

var readFile = nodefn.lift(fs.readFile);

var promiseForFileContents = readFile(fileName);

Another cujoJS team member, Scott Andrews, has even promisified then *entire* fs API: https://npmjs.org/package/fs-then.  Basically he just used nodefn.lift() to lift all the async functions.

There are plenty of other APIs that can help you, such as when.map and when.reduce, and the task executors when/sequence, when/pipeline, and when/parallel, again as Charilaos points out.

So, I would check out the tutorials, and the when.js documentation, specifically the portions for adapting node APIs.

Hope all this information helps, and please do feel free to continue asking questions here as you work through things using when.js!

[Brian Cavalier]

Great write-up, Charilaos, thank you for jumping in!  I've added a few notes inline.

On Saturday, June 22, 2013 3:06:29 PM UTC-4, Charilaos Skiadas wrote:

Now next up you'll want to use fs.readFileSync. To make it promise-aware, you want to use nodefn.call and friends. So you should be able to read the template with something like:

var readTemplate = nodefn.lift(fs.readFileSync, templateName);

Just a note that nodefn.lift will lift the *asynchronous* node APIs, thus should be used with fs.readFile() rather than fs.readFileSync().  If you want to lift non-async APIs, you can use the version of lift in when/function (rather than when/node/function).  See docs here: https://github.com/cujojs/when/blob/master/docs/api.md#synchronous-functions

 

This doesn't exactly read the template, but it turns it into a function that when called reads the template returns a promise instead of expecting a callback.

Yes, exactly.  There are also nodefn.call() and nodefn.apply() which are useful when you simply want to invoke a node-style async function once, without lifting it.

 

when.join(validateMessage(), readTemplate())
        .then(function(values) {
                return when.sequence([sendThankYou, recordSignup], values[0], values[1]);
        })
        .then(success, error);

The "success" function at the end would run if everything runs smoothly, and will receive the return values of sendThankYou and recordSignup as arguments. the "error" function will I believe be triggered if any of the 4 functions involved threw an exception.

You can also probably use when/apply to make that look a bit more elegant.

Yep, when/apply can help here, however, I would recommend the newer promise.spread(), which does basically the same thing and is a bit easier to use.  Here's Charilaos' example refactored to use promise.spread().  Note how it automatically spreads the results onto the callback functions arguments.  In general, I've found that when.join().spread() is a very useful pattern:

when.join(validateMessage(), readTemplate()) 
        .spread(function(message, template) { 
                return when.sequence([sendThankYou, recordSignup], message, template); 
        }) 
        .then(success, error); 

 

 

[Charilaos Skiadas]

On Jun 22, 2013, at 3:34 PM, Brian Cavalier wrote:

> You may find the Learning Promises and Async Programming tutorial sections at know.cujojs.com useful: http://know.cujojs.com/tutorials

Indeed it was those tutorial sections on promises, starting from http://know.cujojs.com/tutorials/async/async-programming-is-messy that got my hooked on when.js. I found it a very clear description of the problem with asynchronous calls and how promises help solve it.

Haris Skiadas

[Brian Cavalier]

Cool, glad to hear they were helpful!

[Charilaos Skiadas]

On Jun 22, 2013, at 3:40 PM, Brian Cavalier wrote:

> when.join(validateMessage(), readTemplate())
> .spread(function(message, template) {
> return when.sequence([sendThankYou, recordSignup], message, template);
> })
> .then(success, error);
>

Very cool! On that note, I was thinking, do the `when…` methods expect `when` as their "this"? I'm thinking we could do the above using Function.prototype.bind like so (probably need to make sure you have bind first):


when.join(validateMessage(), readTemplate())
.spread(
when.sequence.bind(when, [sendThankYou, recordSignup])
)
.then(success, error);

using bind to set the first argument to when.sequence, and let the rest of the arguments through. Would that work? Is "when" required there as the first argument to bind to set the "this" object, or would say "null" work just as well?

Though probably your version is more clear to read anyway.

Haris Skiadas

[Paul Tiseo]

So, I usually get stuck at the start: creating promises. Now, most tutorial are unfortunately very vague and missing on tangible examples. Let's take two of the first promises we need: validateEmail() and readTemplate().

For validateEmail(), I am using the validator (https://npmjs.org/package/validator) library. The call to isEmail() returns an exception and is not async, so is this how we promisify things?

var validateEmail = function (email) {
  var d = when.defer();
  try {
    check(email).len(6,320).isEmail();
    d.resolve(email);
  }
  catch(e) {
    d.reject();
  }
  return d.promise;
}

And, for readTemplate, I simply use lift() as in:

var readTemplate = whennode.lift(fs.ReadFile);

to promisify things.

So, technically, this restify call should print out the POSTed email and the content of the twop template files, correct? Unfortunately, it doesn't.

function signup2(req, res, next) {
    console.log(req.params.email);
    when.join( validateEmail(req.params.email), readTemplate(__dirname + '/templates/basic_email.html'), readTemplate(__dirname + '/templates/basic_email.txt') )
      .then(
        function(values) { console.write(values[0]); console.write(values[1]); console.write(values[2]); }
      );
    return next();
}

[Charilaos Skiadas]

On Jun 22, 2013, at 6:41 PM, Paul Tiseo wrote:

> So, I usually get stuck at the start: creating promises. Now, most tutorial are unfortunately very vague and missing on tangible examples. Let's take two of the first promises we need: validateEmail() and readTemplate().
>

> For validateEmail(), I am using the validator (https://npmjs.org/package/validator) library. The call to isEmail() returns an exception, so is this how we promisify things?

>
> var validateEmail = function (email) {
> var d = when.defer();
> try {
> check(email).len(6,320).isEmail();
> d.resolve(email);
> }
> catch(e) {
> d.reject();
> }
> return d.promise;
> }

I've found most of the time it's simpler than you would think, and actually you usually don't need to create a promise and a deferred object yourself.

What you have there is possibly a valid correct way to "promisify" a method, but if you look into the documentation at: https://github.com/cujojs/when/blob/master/docs/api.md#synchronous-functions you should be able to do it more straightforwardly. Basically if you have a synchronous method that either returns a value or throws, you can promisify it using when/function. With the setup described on that page you should be able to do something as simple as:

fn.call(validateEmail, email)

or promiseValidate = fn.bind(validateEmail)

where:

function validateEmail(email) {
check(email).len(6,320).isEmail();
return email;
}

I imagine the first line in this function call would just throw on an error, and otherwise you get the returned email. Passing this function to fn.call or fn.bind takes care of the "try-catch" code you were writing for you.

Then you would be starting your snippet with:

when.join(promiseValidate(email), … )


In general, "when.js" offers a number of common ways to obtain/combine promises without having to construct defer objects yourself. The only case I believe when I felt some need to add a defer object was in some code for returning a promise on a database object while attempting to connect to it.

> And, for readTemplate, I simply use lift() as in:
>
> var readTemplate = whennode.lift(fs.ReadFile);
>
> to promisify things.

Looks about right, except I think the method is called fs.readFile, lowercase r.

> So, technically, this restify call should print out the POSTed email and the content of the twop template files, correct? Unfortunately, it doesn't.
>
> function signup2(req, res, next) {
> console.log(req.params.email);
> when.join( validateEmail(req.params.email), readTemplate(__dirname + '/templates/basic_email.html'), readTemplate(__dirname + '/templates/basic_email.txt') )
> .then(
> function(values) { console.write(values[0]); console.write(values[1]); console.write(values[2]); }
> );
> return next();
> }

One thing that I found took me a while to realize is that a thrown exception will simply get lost if I don't add a handler to it at some point. For instance in this case your code could be barfing because of the "ReadFile" typo and you wouldn't know it because you haven't added anything to handle the case of a rejected promise, you are only handling the case of a fulfilled promise. Basically:

"Once you are in a promise setting/chain, your code will not be throwing exceptions any more. It will be rejecting promises. And those rejections are silent unless they are listened for."

What promises offer you is a way to handle all the exceptions from all the intermediate steps in one place at the end, if you so choose.

To catch those errors and print a message on them you have two main options, I believe:
1. Pass a second argument to ".then". It would be a function expecting an error object, and it is what would get called if any of your promises along the way was rejected.
2. Chain to ".then" a ".otherwise", with one argument being that same error handler.

So the code in general looks like one of these:

when( ….)
.then(successFunction, errorFunction);

or

when(…)
.then(successFunction)
.otherwise(errorFunction);


These are both discussed in the "main promise API" and "extended promise API" sections here: https://github.com/cujojs/when/blob/master/docs/api.md#promise


Haris Skiadas

[Brian Cavalier]

Excellent answers again, Haris.

Paul, I've added a few more comments inline to Haris's excellent info.  Also, if you'd like to discuss anything in a more realtime format, you can join us in #cujojs on freenode IRC.  The cujojs team typically hangs out there during the week.

On Saturday, June 22, 2013 7:11:38 PM UTC-4, Charilaos Skiadas wrote:

On Jun 22, 2013, at 6:41 PM, Paul Tiseo wrote:

I've found most of the time it's simpler than you would think, and actually you usually don't need to create a promise and a deferred object yourself.

We certainly strive to make it as simple as possible, so glad to hear we're doing ok!

As Haris says, the apis in when/function, when/callbacks, and when/node/function are good for *starting* a promise chain, in addition to promisifying existing code.  Typically, you won't need to promisify functions yourself.  In fact, once you've started a promise chain, plain old synchronous functions (i.e. ones that either return a value, or throw an exception), you can simply pass them to .then() without promisifying them.  For example, we can uppercase file contents:

function uppercase(anything) {

   // This is just a plain old, vanilla call-and-return function

   // It'll either return the uppercased value of anything, or throw an exception

   // Make sure it's a string, then uppercase it

   return String(anything).toUpperCase();

}

var promiseForUppercaseContents = nodefn.call(fs.readFile, 'filename').then(uppercase);

At this point promiseForUppercaseContents will be a promise.  It will either fulfill successfully with the uppercased contents of filename, *or* it will reject with any error that occurs while reading the file *or* while uppercasing the file's contents.  Either way, we can observe the outcome:

promiseForUppercaseContents.then(logToConsole, logError);

 

"Once you are in a promise setting/chain, your code will not be throwing exceptions any more. It will be rejecting promises. And those rejections are silent unless they are listened for."

Yes, exactly. Very well said.

The key here is that exceptions are not a viable error handling mechanism in asynchronous code.  The reason is that they span multiple call stacks, and the call stack where the async operation is called (i.e. where you call fs.readFile, for example) is never the same call stack as where an exception would be thrown.  Thus, exceptions are impossible for the caller (the caller of fs.readFile, for example) to catch.

For that reason, promise machinery catches them for you, and transforms them into promise rejections.  Once you get accustomed to this, it is an incredibly powerful aspect of promises, and allows async errors to propagate in an analogous way to synchronous exceptions.

However, one unfortunate aspect of this is that, since host environments like browsers and Node, don't have any native understanding of promises, they cannot detect "unhandled rejected promises".  This is also tricky since promises have a temporal component.  When a promise becomes rejected, at any instant you might deem that rejection fatal, in the very next instant, someone might add a rejection handler to it, thereby making it no longer an unhandled rejection!

To that end, we will be releasing an experimental promise monitoring tool in when.js 2.2.0.  If you want to try it out now, you can grab the dev branch and check out the docs: https://github.com/cujojs/when/blob/dev/docs/api.md#debugging-promises

All you really need to do is load when/monitor/console, and it will start monitoring when.js promises for unhandled rejections and will report them to the console.  It will also report when those unhandled rejections become handled.  When it does report unhandled rejections, it will also attempt to stitch together a "long" stack trace.  That is, it will provide a stack trace that crosses many call stacks and tracks the lifetime of the promise in question: where it was created, to where the rejection "escaped" the end of the promise chain, and also the point at which the rejection happened

 

What promises offer you is a way to handle all the exceptions from all the intermediate steps in one place at the end, if you so choose.

Yep, and you can think of this as being similar to synchronous exceptions: You can let them propagate up the stack until they reach a `catch` statement at the point in your application where handling the error is appropriate.

 

2. Chain to ".then" a ".otherwise", with one argument being that same error handler.

when(…)
        .then(successFunction)
        .otherwise(errorFunction);

There is a reason that using `otherwise` can be a better option in some circumstances: In the example above, the `otherwise(errorFunction)` will even catch errors that happen within `successFunction`.  It all depends on the situation, of course, and .then(success, error) may be exactly what you need.

[Brian Cavalier]

On Saturday, June 22, 2013 5:00:18 PM UTC-4, Charilaos Skiadas wrote:

Very cool! On that note, I was thinking, do the `when…` methods expect `when` as their "this"? I'm thinking we could do the above using Function.prototype.bind like so (probably need to make sure you have bind first):

Great question.  All of when's methods can be detached and called without a thisArg.  Even the ones in the secondary modules, like when/node/function.  when.js tries never to rely on `this` in its internals.  In fact, even the `then()` method of when.js promises can be detached and used, and it will work correctly ... although I don't recommend it, and it's not true for other methods like `promise.ensure()` and `promise.otherwise()`.

 

when.join(validateMessage(), readTemplate())
        .spread(
                when.sequence.bind(when, [sendThankYou, recordSignup])
        )
        .then(success, error);
 
using bind to set the first argument to when.sequence, and let the rest of the arguments through. Would that work? Is "when" required there as the first argument to bind to set the "this" object, or would say "null" work just as well?

In the particular case of when/sequence, it's a module and not a method on when.  So yeah, you can sequence.bind(null, ...) and that'll work:

var sequence = require('when/sequence');

when.join(validateMessage(), readTemplate()) 
        .spread(sequence.bind(null, [sendThankYou, recordSignup])) 
        .then(success, error); 

And in the case of say, when.map, you're right, the thisArg won't matter when using bind().  For example, these two will be equivalent:

when.map.bind(null, ...);

when.map.bind(when, ...);

[Paul Tiseo]

On Saturday, June 22, 2013 7:11:38 PM UTC-4, Charilaos Skiadas wrote:

I've found most of the time it's simpler than you would think, and actually you usually don't need to create a promise and a deferred object yourself.

Thank you for highlighting that for me. Although I did take a quick read through the docs, I did not go back to it, trying to do it at a lower level myself for my own education. I did use your approach and it did work. However, my original code, where I called defer and returned a promise, did not. I am curious why?

 

One thing that I found took me a while to realize is that a thrown exception will simply get lost if I don't add a handler to it at some point. For instance in this case your code could be barfing because of the "ReadFile" typo and you wouldn't know it because you haven't added anything to handle the case of a rejected promise, you are only handling the case of a fulfilled promise. Basically:

That was a good catch on my draft code!  Once I added an otherwise, I saw that I ended up there, but not sure how to report that kind of error?

Thank you both. Your efforts have seriously short-circuited my learning!

[Charilaos Skiadas]

On Jun 23, 2013, at 9:41 PM, Paul Tiseo wrote:

>
> On Saturday, June 22, 2013 7:11:38 PM UTC-4, Charilaos Skiadas wrote:
> I've found most of the time it's simpler than you would think, and actually you usually don't need to create a promise and a deferred object yourself.
>
>
> Thank you for highlighting that for me. Although I did take a quick read through the docs, I did not go back to it, trying to do it at a lower level myself for my own education. I did use your approach and it did work. However, my original code, where I called defer and returned a promise, did not. I am curious why?

Can you elaborate? The following, which is very similar to what you had, works:

when = require('when');

var validateEmail = function (email) {
var d = when.defer();
try {

if (email === "hi") { throw "error!"; }
d.resolve(email);
}
catch(e) {
d.reject(e);
}
return d.promise;
}

validateEmail("hello").then(function(email) {
console.log("Success: ", email);
}, function(error) {
console.log("Error:", error);
}); // Success

validateEmail("hi").then(function(email) {
console.log("Success: ", email);
}, function(error) {
console.log("Error:", error);
}); // Error

> One thing that I found took me a while to realize is that a thrown exception will simply get lost if I don't add a handler to it at some point. For instance in this case your code could be barfing because of the "ReadFile" typo and you wouldn't know it because you haven't added anything to handle the case of a rejected promise, you are only handling the case of a fulfilled promise. Basically:
>
>
> That was a good catch on my draft code! Once I added an otherwise, I saw that I ended up there, but not sure how to report that kind of error?

Report it to whom? Ideally whatever module is responsible for knowing about this is the one who should be receiving this promise, and then can register to listen for the failure.

Or do you mean how to handle the error? Because that's so much dependent on your situation.

Haris Skiadas

[Paul Tiseo]

I'm already looking forward to this console functionality! :) I've already encountered times when it's hard to know from the propagated error where it came from.

[Brian Cavalier]

If you'd like, you can try it now by grabbing the dev branch.