Promises vs Async vs Stepdown vs etc... Request for non-trivial implementation ideas.

[Adam Crabtree]

With all the back and forth and prostelatizing on why Promises are better, etc... I'd like to put together an example of non-trivial implementations of each to help cut through some of the noise and make it easier for people to decide which they prefer.

Ideally, the implementation would be something of sufficient complexity to show off the strengths and weaknesses of each, but easily contained and comprehended in a single file.

Suggestions?

Cheers,

Adam Crabtree

--
Better a little with righteousness
       than much gain with injustice.
Proverbs 16:8

[Bruno Jouhier]

You can take streamline's tutorial: https://github.com/Sage/streamlinejs/blob/master/tutorial/tutorial.md
source: https://github.com/Sage/streamlinejs/blob/master/tutorial/tuto7-parallel._js

It does the following:

* HTTP server which generates a small form with a search box
* web service call: search string is passed to google
* file system calls: search is also performed recursively against local files
* database calls: search string is run against a collection of movies inside mongodb
* exception handling: exceptions occurring in any of the 3 search operations are trapped and reported to the user
* parallel execution: the 3 search operations are performed in parallel.

The streamline version is only 126 lines of code. So it should not be a major endeavor to implement it with different libraries.

Bruno

[Mikeal Rogers]

How do you express compatibility with the majority of the node.js ecosystem as a code sample?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bruno Jouhier]

We would probably need two examples then:

* An "application" example to compare how the tools let you write a simple yet non trivial application on top of existing libraries.
* A "library" example to test compatibility with the rest of the ecosystem.

The streamline tutorial falls in the first category. It tests the ability of the tool to deal with 3 types of common operations (web service calls, fs calls, database calls) and it tackles common problems that every application developer will face (exception handling, parallelization).

For the second category, what about a basic SMTP client? Something that would handle the low level dialog described in http://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#SMTP_transport_example and would expose a simple API to send messages. Of course the API should be aligned on node's standards and the example would also include a piece of "standard" node code that exercises the API.

[Adam Crabtree]

@Bruno

Thanks. I appreciate the example. That sounds like a good demo touching the key features. Also, your delineation of lib vs app code I think is a good one, but for now I'll be happy just to have the former. Certainly of course the goal of this is for individuals motivated by their library of choice (or authorship) to implement these solutions themselves, so a well defined set of expectations would probably go far in soliciting contributions.

@Mikeal

I agree completely with the spirit of your comment, but promises are by definition compatible with the node ecosystem via the many wrapping APIs, etc... which is of course a primary reasons for their perceived benefit, not having to deal with callbacks. My desire is to have a crowd-sourced canonical source for these sorts of comparisons similar to an idiomatic.js to better inform teams making decisions on whether tradeoffs rather then be swayed by the prevailing noise.

FWIW, I personally don't know anyone that enjoys JavaScript as a language and has been programing in node.js for longer than 6 months that prefers promises. From what I can tell, most of the pro-promise zeitgeist originates from client-side implementations like jQuery's deferreds. 

Cheers,

Adam Crabtree

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bradley Meck]

When we have been making large scale software a few things have come up.

1. Callbacks suck for linear workflows (a->b->c)

This kind of workflow results in what I like to call the mudslide.

Your code really wants to expand into some minor nesting once you have shared state (sometimes immutable state and idempotentcy is a perf issue in resource starved JS [rare use case generally]).

But often when composing parallel workflows this is better for composability (using `.bind` or `npm i async`) vs many `.then` calls on a promise.

2. Promises suck for branching workflows (a->(b&&c)->d)

When promises enter the parallel (not async) workflow we start to see some fun things happen.

`.then` is called, a lot , which gets confusing to read.

Promises on their own need some help when making a join after a fork (see `npm i q`). This parallels to the composability fixes using `async.waterfall` above.

3. Domain all the things

IDC what it is, if you are serious, both need domains.

That being said, there are wrappers both ways, but mostly: determine what kind of workflow you do most often. Most of my things would be awesome if they were linear, but often im grabbing multiple things at once using `async.parallel` combined with `async.waterfall`.

[Adam Crabtree]

> idempotentcy is a perf issue in resource starved JS

Could you expand on this?

I agree completely on your point about branching, excessive then and being harder to read. Ultimately, it's my personal belief that given equivalent examples people will choose what they feel is most readable (and it would not be promises, though that remains up to the individual).

Would you care to fomarlize these a bit more with examples of where you feel promises suffer most?

> Domain all the things

For the near future, I feel it's appropriate to bring to people's attention that domains are not meant for catching errors (See http://nodejs.org/docs/latest/api/all.html#all_warning_don_t_ignore_errors). Unfortunately error coalescing is listed as a major benefit of promises, which exacerbates the issue a bit for various reasons.

Cheers,

Adam Crabtree

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Bradley Meck]

Domains are useful to categorize and debug the error, even if you shutdown, don't just throw them away.

For performance starved JS (talking working under 50MB in a production area) idempotentcy / stateless arch generally results in memory usage or offloading to a single authority. Single authority requires well thought out recovery if the app dies from something bad happening. Statelessness if in a resource constrained system that cannot offload to say a remote DB is just going to be a TON of book keeping and memory management in JS is difficult to predict (generally we move to C/C++ at that point).

Promises vs nested branch (sans q):

```

//make a promise to start the workflow

onuser=db.getUser(x)

//branch 1

onuser.then(db.isUserActive)

//branch 2

ongroups= onuser.then(db.getGroupsForUser)

//branch 2.1

ongroups.then(db.getPermForGroups.bind(db, 'dotask'))

//branch 2.2

ongroups.then(db.getGroupsActive)

ongroups.fail(onuser.fail.bind(onuser))

//branch 3

onuser.then(db.getPermForUser.bind(db, 'dotask'))

//join

// don't feel like writing the .then joins by hand...

onuser.fail(joined.fail.bind(joined))

joined.then(win).fail(blah)

```

With Q fixing the anger; to look a lot like async.parallel

```

onuser=db.getUser(x)

onuser.then(Q.all([

  db.isUserActive,

  db.getGroupsForUser.then(Q.all([

    db.getPermForGroups.bind(db, 'dotask'),

    db.getGroupsActive

  ]))

  db.getPermForUser.bind(db, 'dotask')

])).then(win).fail(blah)

```

[Bruno Jouhier]

The problem is not limited to linear block of statements and parallel branches:

What about async calls inside if/else branches, inside loops, inside try/catch or try/finally?
What about async calls in call chains (f1().f2().f3()), in functional composition (f1(f2(f3()))?
What about async calls in operands (f1() + f2(), f1() && f2()), in object or array initializers (arr = [f1(), f2(), f3()],
etc.

From my experience with a lot of code sitting on top async APIs (mongodb, web services, streams, etc.), there are lots of cases where async calls are naturally nested in such constructs. For an example, take a look at the streamline tutorial around line 100: return coln.find(..., _).toArray(_).map(...).join(...)

Libraries, whether they are callback based or promise based, don't help with these situations: you have no choice but introduce extra variables and split the expressions artificially on async boundaries. To me this makes JavaScript look like a broken language. I want the ability to embed async calls anywhere, chain them, etc. This is why I attacked the problem with a compiler tool rather than a library.

A bench with a non trivial real world scenario would be really helpful. Otherwise discussions tend to be too focused on "blocks of async statements" and they miss important aspects of coding around async APIs.

On Sunday, March 31, 2013 8:28:33 PM UTC+2, Adam Crabtree wrote:

[Bradley Meck]

Manually combining state is something I do not want to give up. Passing by argument is my preferred method rather than introducing variables (async.waterfall). All compiler based syntaxes I have seen give me concerns about shared cache if multiple things are done in parallel and you use shared variables.

If you feel a full benchmark (I think reference problem would be a a more appropriate term) feel free to spec out a simple task like MVC has for todo applications or a full spec for something to compare against. Tooling out an extendable task like a build service like: https://github.com/nodejitsu/module-smith/blob/master/lib/builder.js is painful and a real world example that utilized a ton of different techniques. Feel free to base the spec off that.

[Bradley Meck]

https://gist.github.com/bmeck/5287456

Feel free to spec out a reference problem set by forking and modifying.

[Adam Crabtree]

Ideally, we'd keep this thread to the OP. I know it's easy to begin discussing the merits of async vs alternative environments like streamline, but I'd prefer to break new ground.

On Mon, Apr 1, 2013 at 12:55 PM, Bradley Meck <bradle...@gmail.com> wrote:

Manually combining state is something I do not want to give up. Passing by argument is my preferred method rather than introducing variables (async.waterfall). All compiler based syntaxes I have seen give me concerns about shared cache if multiple things are done in parallel and you use shared variables.

If you feel a full benchmark (I think reference problem would be a a more appropriate term) feel free to spec out a simple task like MVC has for todo applications or a full spec for something to compare against. Tooling out an extendable task like a build service like: https://github.com/nodejitsu/module-smith/blob/master/lib/builder.js is painful and a real world example that utilized a ton of different techniques. Feel free to base the spec off that.

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Alexey Petrushin]

> error coalescing is listed as a major benefit of promises

None of control-flow helpers (steps, asyncs, promises, futures, tamejs, ....) really helps with callbacks. None of them can intercept non-async errors.

As far as I know it's impossible to reliably catch both sync and async errors, even code generators like tamejs can't do that. The only solution is to patch JavaScript engine - like with Node.js Domains or Fibers.

[Alexey Petrushin]

> None of control-flow helpers (steps, asyncs, promises, futures, tamejs, ....) really helps with callbacks.

I misspelled, sorry it should be "with errors" not "with callbacks", i.e.

None of control-flow helpers (steps, asyncs, promises, futures, tamejs, ....) really helps with errors. None of them can intercept non-async errors (or more exactly nested sync/async/sync errors).

[Adam Crabtree]

You first have to define "helps with errors". I don't know about async, but step and promises wrap all their callbacks in try/catch and bubble errors automatically. Stepdown uses my async trycatch library, which before I switched to domains in 0.2, would wrap every callback to core in a try/catch before executing providing truly async try/catch. In fact, it worked so well, it actually fixed some issues later found in node core: https://github.com/joyent/node/issues/5114 (albeit at a slight hit to performance, which is nothing core couldn't due if it was willing to take the hit)

Was that what you were thinking?

[Bruno Jouhier]

On Tuesday, April 2, 2013 4:01:59 AM UTC+2, Alexey Petrushin wrote:

> error coalescing is listed as a major benefit of promises

None of control-flow helpers (steps, asyncs, promises, futures, tamejs, ....) really helps with callbacks. None of them can intercept non-async errors.

As far as I know it's impossible to reliably catch both sync and async errors, even code generators like tamejs can't do that. The only solution is to patch JavaScript engine - like with Node.js Domains or Fibers.

Streamline.js does intercept all errors (sync and async) and propagates them through catch/finally clauses (that have been transformed). This is validated by a suite of unit tests that exercises the different combinations. The transformation patterns are given in http://bjouhier.wordpress.com/2011/05/24/yield-resume-vs-asynchronous-callbacks/

The only case where this does not work is if the exception occurs inside the node.js low level code, before reaching a continuation that has been transformed by streamline; in short if there is a bug in node and node does not even call your callbacks.

So this is not impossible, just non-trivial.

Bruno

[Eldar]

You can have a look at https://github.com/eldargab/make-flow

[Eldar]

Or if your call to third party. And may be there is no bugs, but you passed wrong args...