Making asynchronous coding easier

[rtweed]

Came across this yesterday:

http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchronous-programming-easy.aspx

Would something similar be possible in Node.js?

Rob

[Tane Piper]

http://github.com/creationix/step

You could use step to do something similar

> --
> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com.
> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.
>
>

[rtweed]

Ah ok, Step looks interesting!

On 29 Oct, 08:56, Tane Piper <piper.t...@gmail.com> wrote:
> http://github.com/creationix/step
>
> You could use step to do something similar
>
>
>
> On Fri, Oct 29, 2010 at 8:52 AM, rtweed <rob.tw...@gmail.com> wrote:
> > Came across this yesterday:
>

> >http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchron...

[Dominic Tarr]

what are you talking about? asynchronous programming IS EASY

[Georg Tavonius]

It isn't exactly easy, it just is another progamming paradigma that works well. 

[rtweed]

Sure, it's easy once you get the hang of it, but it's laborious and
all those nested call-backs get tricky to keep track of once you're
several levels deep.

Anyway it's interesting that other technologies, such as the example I
provided, are aiming to provide a way of simplifying the way you can
abstract what you're trying to do into a perhaps more intuitive set of
statements.

Just wondered if anyone had been thinking along the same lines in
Node.js - clearly so, given step.

Rob

On 29 Oct, 12:29, Georg Tavonius <g.tavon...@googlemail.com> wrote:
> It isn't exactly easy, it just is another progamming paradigma that works
> well.
>

> On Fri, Oct 29, 2010 at 12:53 PM, Dominic Tarr <dominic.t...@gmail.com>wrote:
>
>
>
> > what are you talking about? asynchronous programming IS EASY
>

> > On Fri, Oct 29, 2010 at 9:05 PM, rtweed <rob.tw...@gmail.com> wrote:
>
> >> Ah ok, Step looks interesting!
>
> >> On 29 Oct, 08:56, Tane Piper <piper.t...@gmail.com> wrote:
> >> >http://github.com/creationix/step
>
> >> > You could use step to do something similar
>
> >> > On Fri, Oct 29, 2010 at 8:52 AM, rtweed <rob.tw...@gmail.com> wrote:
> >> > > Came across this yesterday:
>
> >> > >http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchron.
> >> ..
>
> >> > > Would something similar be possible in Node.js?
>
> >> > > Rob
>
> >> > > --
> >> > > You received this message because you are subscribed to the Google
> >> Groups "nodejs" group.
> >> > > To post to this group, send email to nod...@googlegroups.com.
> >> > > To unsubscribe from this group, send email to

> >> nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> >> .

> >> > > For more options, visit this group athttp://
> >> groups.google.com/group/nodejs?hl=en.
>

> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "nodejs" group.
> >> To post to this group, send email to nod...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/nodejs?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > nodejs+un...@googlegroups.com<nodejs%2Bunsu...@googlegroups.com>
> > .

[AJ ONeal]

Futures is another library helpful for this issue and works in the browser also.

HTTP://github.com/coolaj86/futures

Check out the examples directory.

AJ

Sent from my Google Android

[AJ ONeal]

There is a major advantage to the asynchronous style, btw. It helps you to know where your functions should naturally be splitting up.

Less temptation to write horrendously long and difficult to read functions.

Also, your data becomes more clearly divided into the groups it belongs.

Read "how to think about oo" by Google testing blog. It's helped me to better organise my modules and such.

AJ

Sent from my Google Android

[rtweed]

Well, having figured out how to implement a recursive set of Node.js
functions to successfully walk a DOM tree (on a GT.M database),
nothing in the asynch paradigm fazes me now :-)

My head hurts though!!

Rob

On 29 Oct, 15:32, AJ ONeal <coola...@gmail.com> wrote:
> There is a major advantage to the asynchronous style, btw. It helps you to
> know where your functions should naturally be splitting up.
>
> Less temptation to write horrendously long and difficult to read functions.
>
> Also, your data becomes more clearly divided into the groups it belongs.
>
> Read "how to think about oo" by Google testing blog. It's helped me to
> better organise my modules and such.
>
> AJ
>
> Sent from my Google Android
>

> On Oct 29, 2010 1:52 AM, "rtweed" <rob.tw...@gmail.com> wrote:> Came across this yesterday:
>
> http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchron...

>
> > Would something similar be possible in Node.js?
>
> > Rob
>
> > --
> > You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to
>

> nodejs+un...@googlegroups.com <nodejs%2Bunsu...@googlegroups.com>.> For more options, visit this group at
>
> http://groups.google.com/group/nodejs?hl=en.
>
>
>
>

[Bradley Meck]

AJ++,

Also, if you get more than 3 levels deep of nesting, you should
probably split up that function. If you do not need a closure above
your current function, why not make the function outside of the
nesting structure to make it reusable (if done right can save quite a
bit of memory and construction time for hot areas).

Async evented coding often trips people up because they want to write
*straight forward* code. Threads/Coroutines(fibers) have taught many
people that code can always be done in a linear fashion and it can
hide the blocking/wait looping nature of code. This unfortunately
comes at a terrible price... wait loops for every thread at times.
Instead of thinking of your code as a waterfall / list (blocking/
threaded), think of it as a tree / errands list. Since topics like
these come up quite often on the mailing list I would like to go into
greater detail the strengths and weaknesses I find in them. Note:
*Node.js encourages the actor models*.

Evented Async (Single Actor Model):
You can gather up all the errands (callbacks/handlers) you need and do
them when it is possible for each, this sometimes means you must
complete one before the other (cant pick up groceries before the bank
etc.), which you might want to split up errands if you need 8hrs doing
a single one so that you can do others.
* encourages splitting up of temporal logic, easily visible splitting
up of code branching
* you must know when your program is blocking and the requirements of
each errand (and if you need to split it up), you lose your stack
trace (you'll get used to it, just avoid anonymous functions)

Worker Async (Multiple Actor Model):
You gather up all of the errands, the head of the household tells
however many people (number of workers) to go do those errands with a
specific one given to each. Once they are done and return the head of
household crosses them off of the list.
* scales well, still no need for semaphores, still quite clear of what
code is being run where, workers can often be reused without
respawning
* workers (generally as processes) have a cost for creation/memory
consumption/cpu/etc. workers may fail for some reason (unlikely if
done right) and backups of their entry parameters should be kept.

Threaded Async:
Stopping halfway done to go to another errand (like a thread), and I
think we can agree that is a little odd if you just talk about what
you are doing. And on top of that, lets gather up all the resources we
need to perform all the errands we are on and keep them with us at all
times, even when we are not working on them.
* hides most knowledge of when to do what, except when you need to use
shared data, or communicate between threads, scales well if threads
have no shared resources (unlikely), threads can often be reused
without spawning/despawning
* wasteful memory and computation wise, and you need semaphores for
any shared data (shudder), threads are sometimes left for reuse and
never are

Cooperative Async:
And lastly, cooperative programming (coroutines) it would be like the
async evented approach except you would have to plan out every step in
advance and decided sometimes you don't want to finish an errand
because another one may be able to be finished.
* much of the knowledge of the scheduler is hidden with no need for
semaphore madness
* can be extremely hard to schedule some things in order, stack /
context regeneration has a cost

Cheers,
Bradley

On Oct 29, 9:32 am, AJ ONeal <coola...@gmail.com> wrote:
> There is a major advantage to the asynchronous style, btw. It helps you to
> know where your functions should naturally be splitting up.
>
> Less temptation to write horrendously long and difficult to read functions.
>
> Also, your data becomes more clearly divided into the groups it belongs.
>
> Read "how to think about oo" by Google testing blog. It's helped me to
> better organise my modules and such.
>
> AJ
>
> Sent from my Google Android
>

> On Oct 29, 2010 1:52 AM, "rtweed" <rob.tw...@gmail.com> wrote:> Came across this yesterday:
>
> http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchron...

>
> > Would something similar be possible in Node.js?
>
> > Rob
>
> > --
> > You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> > To post to this group, send email to nod...@googlegroups.com.
> > To unsubscribe from this group, send email to
>

> nodejs+un...@googlegroups.com <nodejs%2Bunsu...@googlegroups.com>.> For more options, visit this group at
>
> http://groups.google.com/group/nodejs?hl=en.
>
>
>
>
>
>
>
>

[rtweed]

A quick sanity check if I may. I realise I could use step for this,
but I like to get my head round the basics.

Suppose I need to get two properties from a database and do something
only once I've got the both. Provided there was no dependency on the
order in which the properties were fetched, would the following kind
of pattern be a reasonable approach?

var data = [];
var collector = function(error,results) {
if (!error) {
data.push(results);
if (data.length == 2) {
console.log("ok processing can now continue");
}
}
};

mdbx.getNodeType(8,1,collector);
mdbx.getNodeName(8,1,collector);

It all seems remarkably simple - almost too good to be true. Each
function that fetches a property from the database calls the same
callback function - collector. It counts the number of responses it's
received, and when it gets two (or however many it's waiting for), it
runs the rest of the code in its callback function. Of course the
nodeType and nodeName properties may be in data element 0 or 1,
depending on which came back first.

Rob

[Ryan Gahl]

Perfectly sound for cases where the number of async processes is known ahead of time. The generic term for this type of synchronization technique is called a semaphore.

I have a very basic implementation of a reusable semaphore primitive here: http://gist.github.com/540427

To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

[rtweed]

Thanks Ryan! I appreciate the help. The deeper you get into Node,
the cooler it gets :-)

Rob

> > nodejs%2Bunsu...@googlegroups.com<nodejs%252Bunsubscribe@googlegroups.c om>>.>

[Corey Hart]

You can also setup a tracking module to handle concurrent(in the node
sense) requests:

Src: http://github.com/codenothing/Nodelint/blob/master/lib/nodelint/Tracking.js
Docs: http://github.com/codenothing/Nodelint/blob/master/doc/Tracking.md

> > > nodejs%2Bunsu...@googlegroups.com<nodejs%252Bunsubscr...@googlegroups.c om>>.>

[rtweed]

Let me throw in a bit more complexity....so I'm making 4 parallel
requests to the back-end database using my semaphore-based pattern.
Unfortunately the database connection goes down mid-way.....so the
call-back will never complete.

Can/How do you gracefully handle such a situation...ie you'd need some
timer event I guess that checks to see whether, after, say 1 minute,
you'd not had all 4 responses back...otherwise you assume there's been
a problem and return an error in the callback. I'm struggling to
figure out how you'd do something like that in Javascript, but I'm
sure there's a way! Some clever use of setTimeout...?

Rob

On 29 Oct, 21:20, Corey Hart <coreyah...@gmail.com> wrote:
> You can also setup a tracking module to handle concurrent(in the node
> sense) requests:
>

> Src:http://github.com/codenothing/Nodelint/blob/master/lib/nodelint/Track...

[Leonardo]

rtweed, most async libs provide to you a way to send those 4 parallel
and independent callbacks as a batch callback.

In most cases the problem aren't async calls, but the way that they're made.

2010/10/29 rtweed <rob....@gmail.com>:

> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

[Kris Zyp]

On Oct 29, 1:52 am, rtweed <rob.tw...@gmail.com> wrote:
> Came across this yesterday:
>

> http://blogs.msdn.com/b/somasegar/archive/2010/10/28/making-asynchron...

>
> Would something similar be possible in Node.js?

To address this original post, Node actually had virtually this same
exact operator (was called wait() in Node) in earlier versions. It was
dropped due to the interleaving hazards it introduced. Node's wait()
operator used loop stacking, which somewhat simulates coroutines (or
fibers as another recent library calls them, but similar end).

While there are plenty of ways to sugar callback-based functions, I
believe language researches focused on asynchronicity have largely
converged on promises as a robust mechanism for composition of
asynchronous code flow. Promises are essentially functional
programming (side-effect free) applied to asynchronicity, and provide
encapsulation, separation of concerns, without side-effects or non-
determinism (from callback registration, timing, etc) making easy to
build large programs without fragility. Here is a promise library for
Node, which also includes wrappers around the Node I/O operations to
provide clean consistent promised-driven flow.
http://github.com/kriszyp/promised-io

Thanks,
Kris

[rtweed]

Leonardo - understood. Just trying to figure out the basic Node.js
programming techniques really. As you say, you'd be better making a
single round-trip to the database and running a batch set of commands
there in one go.

Rob

On 29 Oct, 21:27, Leonardo <sombr...@gmail.com> wrote:
> rtweed, most async libs provide to you a way to send those 4 parallel
> and independent callbacks as a batch callback.
>
> In most cases the problem aren't async calls, but the way that they're made.
>

> 2010/10/29 rtweed <rob.tw...@gmail.com>:

[rtweed]

To answer my own question, the following adaptation of my original
semaphore example seems to work well:

var counter = 0;
var nodeType = '';
var nodeName = '';

var collector = function(error,results) {
if (!error) {

if ((counter<2)&&(results.timedOut)) {
console.log("timed out - didn't get the nodeName and nodeType
within 5 seconds!");
// abort logic here if required...
return;
}

if (results.nodeName) nodeName = results.nodeName;
if (results.nodeType) nodeType = results.nodeType;

counter++;
if (counter == 2) {
console.log("ok processing can now continue - nodeName and
nodeType are now known");
// etc.....
}
}
};

var collectorTimedout = function() {
collector(false,{timedOut:true});
};

mdbx.getNodeType(8,1,collector);
mdbx.getNodeName(8,1,collector);
setTimeout(collectorTimedout,5000);

As various people have indicated, there are already modules out there
that abstract this kind of logic, but I wanted to figure out the basic
underlying principles of how you do this kind of stuff.

Rob

[Eric Fong]

Hi

I am working on async functions for file system tree.
So I have my own function for handle recursive async calls.

Demo Core for that:

Searcher = function(){
this.root = __dirname;
}

/**
* Demo Sync Method:
* Search file for matching the filename recursively
*/
Searcher.prototype.searchSync = function(name, fromPath) {
var self = this;

fromPath = fromPath || self.root;
var regExp = new RegExp(name);

var rets = [];
var filenames = Fs.readdirSync(fromPath);
filenames.forEach(function(filename){
var filePath = Path.join(fromPath, filename);
var stat = Fs.statSync(filePath);
if (stat.isDirectory()) {
var subRets = self.searchSync(name, filePath);
rets = rets.concat(subRets);
} else if (regExp.exec(filename)){
rets.push(filePath);
}
});
return rets;
}

/**
* Demo Async Function:
* Search file for matching the filename recursively
*/
// wrap the whole method with 'method' func and it will add the
calling scope to be the first arg like python
Searcher.prototype.search = method(function(self, name, fromPath,
callback) {
fromPath = fromPath || self.root;
var regExp = new RegExp(name);

var rets = [];
// use 'this' to wrap async func so that onEnd will work
Fs.readdir(fromPath, this(function(err, filenames){
filenames.forEach(function(filename){
var filePath = Path.join(fromPath, filename);
// second level also wrap with 'this'
Fs.stat(filePath, this(function(err, stat){
if (stat.isDirectory()) {
// all level also wrap with 'this'...
self.search(name, filePath, this(function(err, subRets){
rets = rets.concat(subRets);
}));
} else if (regExp.exec(filename)){
rets.push(filePath);
}
}));
}, this);
}));

/*
* Can also use 'this' like this
* this(
* function a(err, data){
* this(
* function a1(){},
* function a2(){}
* );
* this.onEnd(function a3(){
* // call when a, a1 and a2 is finished
* });
* },
* function b(err, ret, childRets){
* // call when a, a1, a2, a3 is finished
* },
* function c(err, ret, childRets){
* // call when a, a1, a2, b is finished
* }
* );
*/

// when this level and all the lower levels async function end
this.onEnd(function(err, ret, childRets){
callback(err, rets);
});
});


var searcher = new Searcher();
console.log('Sync:');
console.log(searcher.searchSync('png'));

searcher.search('png', null, function(err, files){
if (err) throw err;
console.log('Async:');
console.log(files);
});


method src:
https://gist.github.com/662014

I put it in gist as original method.js is a RequireJs async module
instead of commonjs module.
Thanks

Eric

[Jak Sprats]

Hi All,

this is semi on topic.

I am new to Node.js but an experienced programmer, but async
programming is harder for some things and sometimes it is frustrating
to not be able to simply do nested loops

Here is an example: http://bit.ly/bx03SN
I wanted to loop thru a bunch of redis keys and then depending on
types convert them to relational tables and back them up into mysql.
This is done so I can datamine my redis data.

At the end of the file, i punked out and did a setTimeOut because I
struggled to get the "quit()"s to happen AFTER all the transactions. I
could figure out how to do it, but the point is, it is frustrating at
the beginning.

So I thought about this a while and decided that since my data-store
Redisql has Lua embedded, I didnt need to do nesting in Node.js, I
would just do it in Lua.
Here is the Lua code: https://gist.github.com/671885 - its simple

So I got to thinking, a good way to simplify async logic is to push
nesting out of it when possible. Async coding has to be lightweight in
nature, so nesting is just inherently not a good idea.

I am envisioning using Node.js as a VERY thin application logic layer
and pushing logic into the client (browser side javascript libraries)
and data-store (embedded lua in Redisql) ... is anyone doing anything
like this ... does anyone have an opinion on whether this is the right
way to architect Node.js into a WebApplicationPlatform

- Jak

[Troy]

I've had very similar thoughts, although in your case (or for the case
of looping) it sounds like an event emitter is a cleaner option. But
back to your question, I'm looking forward to getting plv8js to
compile on OS X (I'm not good enough with C to figure out why it
won't), as it will let me use V8 and js inside of postgres. Once I
can do this, I can just pass JSON back and forth to the database, for
example to update multiple rows/tables in a transaction with just a
single async call from node to postgres. And as you mentioned, node
becomes a little lighter and the async nesting flattens out when I can
make multiple sync requests for data where the data lives. If my
needs were a good fit, I would use node and couchdb and be in HTTP-js
heaven, but I'm tied to postgres and with plv8js I'll be set.

-- troy

On Nov 10, 6:34 pm, Jak Sprats <jakspr...@gmail.com> wrote:
> Hi All,
>
> this is semi on topic.
>
> I am new to Node.js but an experienced programmer, but async
> programming is harder for some things and sometimes it is frustrating
> to not be able to simply do nested loops
>
> Here is an example:http://bit.ly/bx03SN
> I wanted to loop thru a bunch of redis keys and then depending on
> types convert them to relational tables and back them up into mysql.
> This is done so I can datamine my redis data.
>
> At the end of the file, i punked out and did a setTimeOut because I
> struggled to get the "quit()"s to happen AFTER all the transactions. I
> could figure out how to do it, but the point is, it is frustrating at
> the beginning.
>
> So I thought about this a while and decided that since my data-store
> Redisql has Lua embedded, I didnt need to do nesting in Node.js, I
> would just do it in Lua.

> Here is the Lua code:https://gist.github.com/671885- its simple

[Jak Sprats]

plv8js looks interesting. Now you can program javascript on client,
server, and database :) Also javascript is 1000 times more intuitive/
robust than stored procedures, which suck.

I am gonna benchmark my two implementations, the first in node and the
second in datastore side lua, I have the feeling the 2nd one will
greatly outperform as there are no network hops (node->Redisql, node-
>mysql) involved in a large loop.

My script is probably something that would be better implemented in
Ruby, because it is not really a frontend script, so it does not
benefit from event driven's strengths as much.

Does anyone have any good links to articles on when to node and when
not to ... I know node's performance far outperfoms ruby/php/python/
etc... for certain workloads, but it seems like anything involving
modest complexity (i.e. nesting) may be so much harder to write that
it is only worth doing when the script is bottlenecking.

where can i find a good howto for doing complicated nesting in
Node.js?

> > Here is the Lua code:https://gist.github.com/671885-its simple

[Tim Caswell]

howtonode.org has a few articles. Look for things with "control-flow"
in the title.

> To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.

[Kris Zyp]

On Nov 11, 10:58 am, Jak Sprats <jakspr...@gmail.com> wrote:
> [snip]

> Does anyone have any good links to articles on when to node and when
> not to ... I know node's performance far outperfoms ruby/php/python/
> etc... for certain workloads, but it seems like anything involving
> modest complexity (i.e. nesting) may be so much harder to write that
> it is only worth doing when the script is bottlenecking.
>
> where can i find a good howto for doing complicated nesting in
> Node.js?

http://www.sitepen.com/blog/2010/09/20/promised-io/ discusses some of
the considerations of sync vs async, and when it is appropriate to use
different async mechanisms for robust composition of large
asynchronous programs.
Thanks,
Kris

[rtweed]

On Nov 11, 5:58 pm, Jak Sprats <jakspr...@gmail.com> wrote:

> Does anyone have any good links to articles on when to node and when
> not to ... I know node's performance far outperfoms ruby/php/python/
> etc... for certain workloads, but it seems like anything involving
> modest complexity (i.e. nesting) may be so much harder to write that
> it is only worth doing when the script is bottlenecking.
>
> where can i find a good howto for doing complicated nesting in
> Node.js?
>

Perhaps this is useful? :

https://groups.google.com/group/mdb-community-forum/browse_thread/thread/cf7f77b092a94712?hl=en

Rob