Re: Retrieving values asynchroniously

[greelgorke]

you could use the async library or similar.

other solution is to restructure your code:

var num1, num2, num3

function sum(){

  if(num1 != null && num2 != null && num3 != null) return (num1 + num2) /  num3

}

Object1.retrieveNum1Async(sum);

Object2.retrieveNum2Async(sum);

Object3.retrieveNum3Async(sum);

waterfalls are easy to refactor, as well parallels with full sync on completion like yours here.

Am Montag, 22. April 2013 21:54:50 UTC+2 schrieb Slobodan Blazeski:

Hi All

I'm looking for suggestions of how to retrieve values asynchronously:

In the synchronous world I have 

var num1 =  Object1.retrieveNum1();

var num2 = Object2.retrieveNum2();

var num3 = Object3.retrieveNum3();

var result = (num1 + num2) /  num3;

but since functions  Object1.retrieveNum1,Object2.retrieveNum2 &  Object3.retrieveNum3

retrieve data from the database or represent long calculation I need to pass callbacks,

that leads me to below

 

 Object1.retrieveNum1Async(function(num1){

      Object2.retrieveNum2Async(function(num2){

            Object3.retrieveNum3Async(function(num3){            

                 var result = (num1 + num2) /  num3;                  

            });

      });

});

is this the idiomatic way of doing this or there is something better

thanks

Bobi

[Mariusz Nowak]

Take a look into promises concept. e.g. with Deferred library you can configure it that way:

deferred(Object1.retrieveNum1(), Object1.retrieveNum2(), Object1.retrieveNum3()).match(function (num1, num2, num3) {

   var result = (num1 + num2) /  num3;
});

[Bruno Jouhier]

With streamline.js (https://github.com/Sage/streamlinejs), you just need one line of code:

var result = (Object1.retrieveNum1(_) + Object2.retrieveNum2(_)) /  Object3.retrieveNum3(_);

[Azer Koçulu]

I wrote a guide for defining async values previously;
https://github.com/azer/declarative-js

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[greelgorke]

where's the fibers guy? :D

[Isaac Schlueter]

Slobodan,

You can use a counter for this:


function ifYouNeedToGetThreeNumbersAsynchronouslyAndThenDivideTheSumOfTheFirstTwoNumbersByTheThirdNumberAndPassTheResultOrErrorToTheCallbackThenCallThisFunctionBecauseItDoesThat(cb)
{
var num1, num2, num3;
retrieveNumOne(function(er, n) {
num1 = n
then(er)
})
retrieveNumTwo(function(er, n) {
num2 = n
then(er)
})
retrieveNumThree(function(er, n) {
num3 = n
then(er)
})
var i = 0, errState = null
function then(er) {
if (errState) return;
if (er) return cb(errState = er)
if (++i === 3) cb(null, (num1 + num2) / num3);
}
}

Of course, yes, this is more lines of code, and a somewhat more
verbose function name. But it's very explicit, and there's no
question about what it's doing.

"Call these functions and get the result, or error" is a pretty common
pattern, so there are some libraries that abstract it away. Check out
async.

Personally, I prefer to not use control flow libraries like async.
There's nothing wrong with them, per se, but I find that leaving that
out requires me to have a lower tolerance for complexity in my
programs, which is on the net a good thing. I definitely think that
you should *write* a control flow library, even if you then throw it
away, because there are few ways to get as familiar with these
concepts than writing a library that implements them.

[Mikeal Rogers]

I feel really bad for people that ask incredibly simple questions on this list.

The solution nearly everyone in this community uses is the async library:

https://npmjs.org/package/async

https://github.com/caolan/async

Its currently depended on by 1901 other packages. It's the defacto-standard for flow control in node.js. The only people that don't use it are people that prefer to do everything by hand and people that wrote their own pet library, anyone asking this question is unlikely to fall in to one of those categories.

This list fills up with people advocating the library they wrote that almost nobody uses, which confuses new users in order to further their own agenda. If this is to be a place for new users to ask simple questions we need to find a way to discourage this behavior.

-Mikeal

[Herman Junge]

+1

Sent from my iPhone

[Mark Hahn]

You are asking to censor replies.  This is obviously not acceptable.

[Nathan White]

You are asking to censor replies.  This is obviously not acceptable.

Relax. No one is stopping anything... just encouraging best practice for communication to the whole audience.

In a technical forum I think it's best to disclose affiliation to a project. A good habit when suggesting packages is to use an objective source like npm stats like Mikeal has done. Or at least a compare/contrast on why I might use your package.

[Mikeal Rogers]

Exactly, I said "discourage," which isn't sensor. I'm not saying we should delete their posts, I'm saying we should turn them in to social pariahs :P

[Mark Hahn]

> Or at least a compare/contrast on why I might use your package.

Giving a link to a solution is a perfectly acceptable forum answer.

[Forrest L Norvell]

As someone who's recently become a huge fan of Q, but mostly writes code that eschews dependencies altogether, I totally agree with Mikeal. The strength of Node is at least in part its huge ecosystem, but that strength is a liability when it comes to getting newcomers up and running. async, express, mongoose, and winston are all far from perfect, but they're the most commonly used tools for what they do, and they get the job done. Once you're oriented and know what's going on, it's a lot easier to pick and choose from what's available (or write your own thing) to find something that works better for you.

There's a big difference between trying to help newcomers get established by giving them a simplified menu of choices and censorship. This isn't about "omakase" or even best practices, IMHO, it's just keeping people from drowning.

F

On Wed, Apr 24, 2013 at 2:50 PM, Mikeal Rogers <mikeal...@gmail.com> wrote:

[Peter Rust]

Slobodan Blazeski,

As others have said, the idiomatic way to do this in node is to use the async library and to have callback functions that take an error ("err") as the first argument.

The reason why it's important to have each asynchronous function pass an error object to your callback is because it is impossible for the asynchronous operation to throw an error in such a way that your callback function can catch it, like you can with traditional, synchronous programming. For instance, if there is a database error or a calculation error, you probably want to short-circuit and display the error.

So, with your example modified to include error handling, it would look like this:

Object1.retrieveNum1Async(function(err, num1){

      if (err) throw err;

      Object2.retrieveNum2Async(function(err, num2){

            if (err) throw err;

            Object3.retrieveNum3Async(function(err, num3){            

                 if (err) throw err;

                 var result = (num1 + num2) /  num3;                  

            });

      });

});

You may not want to throw the error -- you may want to write it to the console or ignore it or pass it back to a caller further up the chain -- but in any case you'll want to do *something* with it and you'll probably want to short-circuit the operation at whatever point the error occurs. Handling errors at each level like this adds a lot of noise, so typically node developers turn to the async library, which allows you to consolidate error handling and callbacks by short-circuiting by default and calling a single callback with any errors from any of the async calls:

var async = require('async');

async.parallel({

    'num1': Object1.retrieveNum1Async.bind(Object1),

    'num2': Object2.retrieveNum2Async.bind(Object2),

    'num3': Object3.retrieveNum3Async.bind(Object3)

}, function(err, r) {

    var result = (r.num1 + r.num2) / r.num3;

});

One annoyance with this approach is the need to .bind() the methods to their respective objects because passing a method in Javascript effectively "disembodies" it from the object it was attached to. One way around this is to pass anonymous functions to async, as James Basco demonstrated above -- this way you're passing anonymous functions to async and you're able to run the methods right off of their corresponding objects.

Or, if you have the option, it can often be simpler and cleaner to write callback-heavy code by using closures and nested functions to maintain state, rather than the more traditional object-oriented style, due to the need to constantly bind methods. These two styles (object-oriented and closure-based) can be mixed and matched for optimum simplicity.

[Peter Rust]

Addendum: Caolan McMahon (the author of async library) provides a nice example and more details regarding prefering closures over object methods in the first point of his article http://caolanmcmahon.com/posts/nodejs_style_and_structure/. Many of his other points are also helpful to someone starting out with Node (especially "Always check for errors in callbacks" and "Stick to the callback convention"):

There is a vocal minority on the Node.js mailing list and elsewhere which promotes the use of other strategies for handling asynchronous code ... [but] The vast majority of Node developers follow this style

[Marcel Laverdet]

I'm rolling in my grave.

--
Sent from My iPhone

[Bruno Jouhier]

FWIW my response to Mikeal: http://bjouhier.wordpress.com/2013/04/27/nodes-social-pariahs/

[Jorge]

On 24/04/2013, at 23:50, Mikeal Rogers wrote:

> I feel really bad for people that ask incredibly simple questions on this list.

I feel really bad whenever I read *your* posts dismissing other people's work.

> The solution nearly everyone in this community uses is the async library: <snip>

How can you know?

> Its currently depended on by 1901 other packages.

Cool, but there's many many more than that users of node.

> It's the defacto-standard for flow control in node.js.

How can you know? (You *can't*)

> The only people that don't use it

The "only" people? Are you implying they're just a few? If not, you wouldn't say 'only'. You can't know how many people don't use it.

> are people that prefer to do everything by hand

How can you know? (You *can't*)

> and people that wrote their own pet library,

Why "pet" library? Your prejudices? Dismissing other people's work?

> anyone asking this question is unlikely to fall in to one of those categories.

People asking this question want to hear valid answers for this question.

> This list fills up with people advocating the library they wrote

Which is fine. They're proud of their work and that's fine.

> that almost nobody uses,

How can you know? (You *can't*)

> which confuses new users in order to further their own agenda.

++prejudices

> If this is to be a place for new users to ask simple questions we need to find a way to discourage this behavior.

It is your behaviour what ought to be discouraged.

Bye,
--
( Jorge )();

[Mikeal Rogers]

On Apr 27, 2013, at 6:46AM, Jorge <jo...@jorgechamorro.com> wrote:

> On 24/04/2013, at 23:50, Mikeal Rogers wrote:
>
>> I feel really bad for people that ask incredibly simple questions on this list.
>
> I feel really bad whenever I read *your* posts dismissing other people's work.

Nobody is telling you not to publish what you like, nobody is telling yo not to promote it, responsibly. But, it is disrespectful to the community to steer new users in a direction that is not consistent with what the majority of node users are using to solve these issues. Many people posting to this list don't know that there are a few people who respond to, literally, every post about flow control to promote their favorite library, and that they do not represent the majority of node users, which one would expect the response to be on **the nodejs mailing list**.

>
>> The solution nearly everyone in this community uses is the async library: <snip>
>
> How can you know?

With actual numbers, which I provided.

>
>> Its currently depended on by 1901 other packages.
>
> Cool, but there's many many more than that users of node.

Roughly 6.7% of the registry is using async, it's significantly larger than any of the libraries posted repeatedly by you and others in response to simple questions. Async is the second most depended on module in the registry, second only to underscore. If every functional programming question was flooded with alternatives to underscore I'd be writing the same email.

>
>> It's the defacto-standard for flow control in node.js.
>
> How can you know? (You *can't*)

We can look at actual numbers and metrics, which I've done.

>
>> The only people that don't use it
>
> The "only" people? Are you implying they're just a few? If not, you wouldn't say 'only'. You can't know how many people don't use it.

You're right, we don't know the exact number of users, but we do have enough metrics to have a good idea of the disparity in usage between different libraries.

>
>> are people that prefer to do everything by hand
>
> How can you know? (You *can't*)

Many people prefer to do everything by hand, isaacs is one of them, i was for years. These people exist :)

>
>> and people that wrote their own pet library,
>
> Why "pet" library? Your prejudices? Dismissing other people's work?

"pet" because you make it your job to reply to every question on the list promoting a library that, by all accountable measures, it not very widely used.

>
>> anyone asking this question is unlikely to fall in to one of those categories.
>
> People asking this question want to hear valid answers for this question.

People asking questions want to know what the community is using to solve them. A portion of that community, which you eagerly represent, will always use libraries that differ from the most widely accepted ones. The answers on this list, if we were to help new users achieve their goals, would represent usage and acceptance in the community at large not the loudest opinions of people that choose to use or write alternatives.

Go ahead and build what you love, that's great! But try to be a little responsible for the experience of people new to node and recognize that most people probably want to use what *most* of the other node developers are using.

>
>> This list fills up with people advocating the library they wrote
>
> Which is fine. They're proud of their work and that's fine.

That would be great is this list was a marketing tool for libraries, it isn't, it's the support structure for people new to node. Please treat it respectfully.

>
>> that almost nobody uses,
>
> How can you know? (You *can't*)

Why do you hate numbers?

>
>
>> If this is to be a place for new users to ask simple questions we need to find a way to discourage this behavior.
>
> It is your behaviour what ought to be discouraged.

Let me ask you a question: do you think that when people ask a simple flow control question on the nodejs mailing list that what they really want isn't an answer consistent with what most node developers use but what you, in your opinion, think is better?

Many of us have strong opinions about flow control and don't flood the list with them and instead direct people toward what most other people seem to be using. Myself and others on this list are probably not the "typical" node users and we point new people in the direction of modules they will probably find easier than what we would personally choose. That's because we want to help people find what they are looking for, not what we *want* them to choose. I'm asking that you do the same.

This isn't about *your* library and *your* work, it's about trying to help people new to node find what they are looking for. If someone asks about generators nobody is going to fault Bruno for mentioning that his library supports both generators and callbacks, that's entirely relevant and being that generators aren't widely adopted elsewhere Bruno, and his library streamline, probably have more experience working with them than anyone else. But, when someone asks how to do async iterations they should get a simple answer that is consistent with what most people, especially new people, are doing.

>
> Bye,
> --
> ( Jorge )();
>

[Mikeal Rogers]

That's a good response, and a better characterization of the streamline compatibility/incompatibility than I state in my article.

I'd like to make on thing clear. 

While I think that some of your ideas might be better served as a less if not entirely incompatible fork I don't intend to disparage the work.

I think that most people who have reacted negatively to you on this list, myself included, take little or no issue *with your work* but in how you choose to promote it.

When new users ask how to accomplish a simple async operation they aren't asking "can someone tell me about every flow control library" and even if that is the answer to their question they should at least see a response of libraries representative of their usage.

There are many other discussions that happen here about es6, generators, alternative concurrency patterns, even discussions about approaches in general to flow control. In all of those discussions your input is valued and I can't remember a time when someone was negative about it.

-Mikeal

[Herman Junge]

In shorter words: we should converge, not diverge.

H.-

Sent from my iPhone

[Bruno Jouhier]

I searched into google groups and my previous mention of streamline on this mailing list dates back from October 4th. It was an exchange with Fedor Indutny about testing his spoon library. This is what you call excessive promotion!

And I usually limit my posts to a one-liner (except this one).

The OP was not asking how to perform a simple async operation. He had figured it out by himself. He was asking whether there was a better way. I answered his question.

We should be allowed to talk about the control flow library and tools that we have written when people ask specifically about this topic. Your statement that we are "confusing new users" is pure dogmatism. I don't see anything wrong in giving pointers to people. Rather the opposite.

Bruno

[Isaac Schlueter]

Ok, gentlemen, let's take the intensity down a notch here... :)


Streamline is great. It's super useful to Bruno and some other
people. All software is awesome, and you all do wonderful work.

As Bruno pointed out in his blog
http://bjouhier.wordpress.com/2013/04/27/nodes-social-pariahs/ he's
not in the business of writing npm modules, and if he was, he'd think
twice before writing them with Streamline. Though, to be fair, as
long as you *publish* JavaScript, it's perfectly same. In that
regard, Streamline is no different than CoffeeScript or any other
compile-to-js tool.

Async is incredibly widely used. If you ARE in the business of
writing npm modules, then you'll have lots of company if you're using
async.

Mikeal is not the node police. He curates NodeConf, is incredibly
outspoken, and has been around Node for a long time, and is the author
of a few very widely used npm modules. We're lucky to have him around
here. But he gets to be somewhat boisterous and freely outspoken with
his opinions, because he is not in power.

I actually AM the node police. (At least, insofar as Node even HAS
police.) However, even my opinions are not necessarily the opinions
of Node, and in many cases, my opinions about opinions as such
override my actual opinions about code! (Being human is so
complicated!)

When we make recommendations to new users, it is important to
communicate not just the options, but something *about* those options
as well. Saying "Most node users use async for this sort of thing" is
about as substantiated by evidence as we could want. It's not just
the most popular control flow module in npm, it's the ONLY popular
control flow module in npm. Every other one vanishes next to it.
We'd probably be fine answering questions like this by saying "Here's
the concept. Here's the module-free pure-js way to do it. Most
people who use a library for this use async."

Personally, I don't care how you write your JavaScript, whether you
use JSLint or Streamline or IcedCoffee or tea leaves or whisper the
contents of your heart into the ear of a fairy who then sprinkles the
JavaScript into the file with a liberal mix of extraneous comments,
sourcemaps, semicolons, and pre-hoisted vars. Just don't make ME have
to deal with that crap when I'm using your module. Publish
JavaScript, not fairy whispers.

Also, as I prefer not to whisper to fairies, I'll be less likely to
contribute to your module in that case. These are the facts of the
tradeoffs. You choose the pants policy in your own house, and this
choice affects who feels comfortable there. Same as semicolons or
other inconsequential things that people nonetheless have opinions
about.

I find control flow utilities objectionable on principle, Promises and
Streamline and Async included. I've used a few, and I think everyone
should write one. It's not that I think you should do control flow
"by hand" -- I think that you should write such simple and focused
programs that these techniques are largely *unnecessary*. If your
module depends on async (or for that matter, underscore) then I
consider that a code smell. If your module has 10 levels of
indentation, and is doing 30 things using a hand-rolled async.map,
well.. that's probably even worse.

Anyone who has seen the npm source code will know that I am a
hypocrite when I say that you should not use control flow libs, as I
use "slide" all over the damn place in there. But part of learning
from your mistakes is making mistakes, and npm is nothing if not a
collection of all the biggest mistakes you can make in a Node program,
going back several years.

Note that even though Isaac The Node Developer gives a crap about
these things, Node The Community regards these specific opinions as
flat-out wrong, and Node The Platform has no opinions on this matter
at all!

Wearing the hat as Node Project Leader and Node Mailing List Admin,
the important thing is that we are nice to each other, and give new
users facts and truth. If a new user asks a question about doing 3
things asynchronously, and they get these answers:

1. use async
2. use streamline

then it looks like the two answers are representative, and roughly
equivalent, which is a bit misleading. It rubs (at least) Mikeal the
wrong way because it seems like Streamline is being promoted as
equivalently popular or relevant in the community as Async, which,
according to literally all the data we have on this question, it
certainly is not.

What's worse, this equivalent representation is often followed up by a
lengthy spiteful debate where we all complain about each other. So
the communicated answer seems to be:

1. You should use ether Async or Streamline (and they're about the same).
2. Don't ask, unless you want to see Node people fight and get mad.

I don't think anyone wants that. (From my point of view, both points
are terrible advice!)

Bruno, I appreciate that you're trying to be a good citizen etc.
Towards that point, I think it would actually be much better if your
"one liner" came along with a link to a longer post or something that
very objectively explains what Streamline is, what it's good for, who
it's (un)popular with, and what the drawbacks are. Ironically,
perhaps, the shortness of the reply may be what causes some of the
problems here. (Certainly, a long explanation of how great and
perfect Streamline is for everything and how everyone uses it, would
be inappropriate, but I see that you grok that already.)

I can see that you both very much want to help new users be
successful, and that is great. The fact that the debate is over HOW we
should help newcomers most effectively is *awesome*. This question
comes up often enough, and is enough of a source of disruptive
disagreement, that maybe we three should just hash out a single wiki
page or something that we can all be moderately happy with, and agree
to send users there when they have this question.

Does that sound like a reasonable idea?

[Mark Hahn]

>  that maybe we three should just hash out a single wiki
page or something that we can all be moderately happy with, and agree
to send users there when they have this question.

+1, however I predict a troublesome, contentious process to reach agreement.  :-)

[Bruno Jouhier]

Hi Isaac,

Thanks for taking the time. It helps. Just replying to selected fragments

Personally, I don't care how you write your JavaScript, whether you

use JSLint or Streamline or IcedCoffee or tea leaves or whisper the
contents of your heart into the ear of a fairy who then sprinkles the
JavaScript into the file with a liberal mix of extraneous comments,
sourcemaps, semicolons, and pre-hoisted vars.  Just don't make ME have
to deal with that crap when I'm using your module.  Publish
JavaScript, not fairy whispers.

I cannot resist but repost the secret recipe of streamline that I had posted a long time ago on the streamline mailing list

... it really is black magic:

The streamline compiler sends the code (asynchronously of course) to a
haitian vodou.
The vodou kills 3 animals at random, mixes their blood, drinks it,
spits 7 times on the code.
Then he dances in circles and invokes Papa Legba (guardian of the
callbacks) who whispers the result of the transformation to his left
ear.
And, as you would expect, the compiler gets the vodou's reply through
a callback.

Hope this helps :-)

 

Bruno, I appreciate that you're trying to be a good citizen etc.
Towards that point, I think it would actually be much better if your
"one liner" came along with a link to a longer post or something that
very objectively explains what Streamline is, what it's good for, who
it's (un)popular with, and what the drawbacks are.  Ironically,
perhaps, the shortness of the reply may be what causes some of the
problems here.  (Certainly, a long explanation of how great and
perfect Streamline is for everything and how everyone uses it, would
be inappropriate, but I see that you grok that already.)

Well I had included the link to streamline's GitHub repo.
The README page gives a good overview of what it is.

I can see that you both very much want to help new users be
successful, and that is great. The fact that the debate is over HOW we
should help newcomers most effectively is *awesome*.  This question
comes up often enough, and is enough of a source of disruptive
disagreement, that maybe we three should just hash out a single wiki
page or something that we can all be moderately happy with, and agree
to send users there when they have this question.

Does that sound like a reasonable idea?

Why not? I think that the most important is to try to clarify the different approaches: there are several library flavors (async, promises, others), CPS transforms (streamline, IcedCoffeeScript, wind.js), coroutines (fibers) and of course raw callbacks, and probably more to come (just saw this today: http://koush.com/post/yield-await-v8). This is probably difficult to grok for a newcomer. A technical explanation of the rationale of each solution with factual pros and cons would be very helpful, much more than psychodrama on the mailing list.

Bruno

[greelgorke]

On original topic:

I truly believe that a good and clean software design paired with a small switch in the mindset, is more than enough to handle most of complexity in  a node program. That's why a almost always advise to go pure js way, it forces you to actually learn how to do good software design and to think in evented/callbackdriven way. this is also valid in topics about code reusage, coffeescript etc. But i'm realist enough to not asume that everyone is enthusiastic to reach this standards. many of us, me sometimes including, just want to get the job done. or just have troubles to think different. nothing's wrong with that, there are tools to support this approach. this is where async, streamline, coffeescript etc. are valuable.

The problem with tools is always, that they add another level of complexity to grog. promises even worse, because they need another switch in the mindset, to follow a different paradigm.

It's a good advice, to write you own control-flow lib. but not everyone has the time or ambitions to do it. and you don't have to. you just cant try to solve your current flow problem without libs and magic, just pure js, first. when it's done, then you could decide on the value of such libs for you.

my advice on such questions will be allways: think about your code, your architecture first, think, how the decisions you made about it influence your problem. improve that first, it not only solves your flow problem, but probable improves other criteria for code quality, as testability, maintainability, coupling and reusability etc. If the design is good enough and the flow control is still a problem, then don't be shy on tooling. just know, that every tool you chose will have impact on your productivity in short and long range.

I think a common and clean way to find and make decisions on a module, which refelects popularity is still a problem in node comunity, that is solved not good enough. npm search must provide more infos. download rates are good but only a beginning, dependency statistics and contributions frequency may be a good next step. then, it's easier for a node novice to decide on opinions posted here. at least regarding concrete modules.

greet  

[Nuno Job]

Node means a certain set of things, and part of those things are callbacks and streams.

Callbacks and streams are hard to learn because its a new programming paradigm you don't learn in school. It was hard for me too and I did it when this was still up for discussion. And I had good friends to teach me that really knew node.

Developers will always try to fast track to their already understood paradigms; but that is not node. This existing paradigm makes sense to build network proxies; which node is awesome for. To node programs are proxies.

A person should always start with callbacks and streams until he understands them, closures, and the event loop. Then he can make informed decisions on what to use when.

However node is not the solution to all problems and if you wanted to build a general purpose replacement for Ruby on Rails in JS you probably wouldn't use callbacks and streams by something else as discussed. Personally i think this would be great, an would follow closely such a project. It could even interopt with node and npm but its not node!

Given my opinions above it makes sense that i would advocate that a node beginner should start with the standard idioms: callbacks and streams. It might even make sense to make these documented as "the standard" on the site.

However its possible that this is not the best solution for the person that asked the question, but then maybe that person shouldn't be using node and instead the brand new super cool thing someone will build. Or maybe he needs to learn how node works and then decide, by himself, that something else makes more sense.

The ecosystem exists, it has an interoperability model, and works great for what node is for (check node success). Improvements on this model are possible (like streams2) but even those aren't without backlash. Breaking changes are also awesome, but its not node

Nuno

Ps. Cause this doesn't seem obvious to everyone, these are just my personal opinions. And I expressed them here mostly in respect to Bruno who did a blog post i thought was respectful and thoughtful, and i felt compelled to respond in the same manner. We have a great community and i respect a lot the work and passion community members put into their projects

[Mark Hahn]

> if you wanted to build a general purpose replacement for Ruby on Rails in JS you probably wouldn't use callbacks

How can you code in javascript without callbacks?  It doesn't support threads.  What would you do with setTimeout?

[Alexey Petrushin]

I guess the answer for novices should be - don't use any async library, write ugly but simple code that works.

The debate about callbacks libraries are because none of it solves the problem completely, everyone has one or other tradeoffs. And to be able to pick a right solution - you need to understand those tradeoffs, novice can't do that.

So, maybe answer for novices would be - don't use control flow libraries, or if you really wish - go to github and learn about some of them and decide by yourself what you want.

[Liam]

Slobodan,

Sadly the node.js list sometimes generates more heat than light. Here's a bit of the latter:

DB.retrieve('id1', cb); // start all retrieves in parallel, a key benefit of Node
DB.retrieve('id2', cb); // providing the object name in the callback arguments may require passing it to retrieve
DB.retrieve('id3', cb);

var vals = { length:3 }; // could also vals.length++ on each retrieve line

function cb(err, result) { // result is { name:string, value:number }
  if (err) throw err; // or something more useful
  vals[result.name] = result.value;
  if (--vals.length > 0)
    return;
  // process vals here
}

Hope that helps.

[Bruno Jouhier]

Hi Nuno,

Your post is helpful. I'll try to challenge it in a constructive way. I think that most of disagreement comes from a different perspective on means and ends.

From your perspective, node is an end in itself: you **do** node; you build streams; you write callbacks; etc. For you, adherence to node's core philosophy is paramount.

From my perspective, node is just a means to an end: we **use** node. For us, the end is to more our product forwards. Node is just one piece of the puzzle we are assembling (an important one). For us, node's core philosophy is "interesting" but it is not binding.

Our team is growing and when I'm bringing new people on board I'm not teaching them callbacks and streams; I'm teaching them streamline. This is because I need them to get up to speed quickly and I also need to have a code base which is simple, robust, easy to maintain and homogeneous. They can still learn callbacks and streams on their own and I'm encouraging them to be curious but our internal coding standard is streamline.

Also, I considered node's philosophy when architecting our new system. But I have difficulties to fit what we are doing into the streams and pipes paradigm. It does not feel "natural" and I don't see what benefits I would gain. For me, it is much more natural to architect our system as classical modules that **consume** streams than as streams that would be piped into each other. I find the pure asynchronous nature of node very appealing but I find the streams & pipes philosophy less appealing. Maybe there is also a question of maturity: the asynchronous APIs are something we can very easily build upon today (in part thanks to streamline); the streams & pipes looks more like an experiment in progress, which may turn out to be a real breakthrough but which would be introducing too much complexity for us today.

I also have a different perspective on node's APIs. I followed the discussions on the new streams API and I was always amazed by the very strong focus on the "building streams" side. I find this very complex and fragile. In our system, we very rarely build a stream but we consume a lot of streams. So we have a small wrapper that gives us a very simple API to consume streams. We are not following the core philosophy but it does not really matter for us. What matters is that we can build our product.

Now, when a person comes the the mailing list with a question, I wonder what his perspective is. Does he want to **do** node, or is he just trying to **use** node to build something? If his goal is to **do** node, then yes, Mikeal is right with his hard line: people should not blur the message and give him advice that may distract him from node's philosophy. But what if all he wants is to **use** node, maybe in ways that are not completely aligned with node's core philosophy, like we do? should we stick to the hard line, or should we let him explore "other ways" that may be a better answer to his needs.

The node ecosystem is growing fast, with people that **do** node, and I don't want to disrupt this. But I feel that there is a great opportunity for node to go more mainstream (like PHP and RoR). I don't think that node will go mainstream by trying to convert mainstream developers to the callbacks and streams paradigm, simply because it is too hard. I think that it can do so by being open and letting people just **use** node, and this is where alternate tools may help.

I don't have a good answer to what the line of conduct should be on the mailing list. But I don't think that excluding discordant voices is the right solution (in this specific case but also in general).

Bruno.

[Isaac Schlueter]

> I don't think that node will go mainstream by trying to convert mainstream
> developers to the callbacks and streams paradigm, simply because it is
> too hard.

This is a judgement that is not substantiated by any evidence whatsoever.

The evidence suggests that most node users, beyond the initial
learning curve, have a pretty easy time with streams, events, and
callbacks. Compared to the learning curve of, say, Haskell or Clojure
or Erlang or even C (which are all pretty popular, and at least C is
as "mainstream" as it gets), it's a trivial hurdle. When users come
from the front-end, where these concepts have already been around for
ages in various forms, it's even less of a hurdle. While the
peculiarities of system internals and TCP are a bit new for many, of
course, Node's API is much easier to pick up quickly than any web
browser's.

The evidence I'm referring to is the increasing rate of growth of our
module ecosystem, Node developer jobs being filled, and companies
using node in their products, vs the comparatively low rate of these
questions on the mailing list, irc, issues, and stackoverflow.

It is a minority that has serious trouble with this, and they tend to
learn quickly, and never have trouble with it again. (As shown in
this case by this thread, where Slobodan grokked the answers quickly
and moved on while we all debated the best way to answer his question
:)

Node is pretty obviously on the path to mainstream-ness, and patterns
that deviate from its minimal set are pretty fringe. (The least
fringe alternative control-flow pattern is Promises, by a long shot,
helped by the fact that it has considerable uptake in the front-end JS
world.)

[Bruno Jouhier]

Fine, but I would not call Haskell, Clojure or Erlang mainstream! I was more thinking of PHP, Visual Basic and the likes.

[Marcel Laverdet]

I think the distinction between *doing* node and *using* is good call. I wrote a tampering HTTP proxy server in node; I used streams and callbacks. I wrote a web application with lots of waiting on database and filesystem chatter; I used fibers w/ futures.

[greelgorke]

well, we should distinguish a bit. 

Devs coming from web-frontends probably never used or even heard about the concept of streams, and it is causing troubles for them.

on the other side, backend devs are used to shared state, threads and all the magic stuff around isolating concurrency in to more abstract models, i.E. actors, green threads, stm etc. many of them never used or heard about the concept of the single-threaded event loop. 

then, there are devs, which managed to not bother about threads, callbacks, etc, mostly because they used one "silver-bullet" framework, that hid every thing away from them. i guess they will have the hardest way to use or do node. ever wondered about questions like "how do i structure my project in express?" 

i think node makes it easier to learn it's api by reducing itself to a few core modules. i think it's a nice good way how node exposes relatively low-leve api's in a relatively high level manner. but the thing with "let userland handle the rest" makes it harder to enter the eco-system.

I don't think that we have real problems with nodes philosophy or it's api. the biggest problem is the entry path to it. RoR success is mainly related to the opinions it have in it. it leads the way. Express does it too, but it's way less opinionated. I do not know many mvc-frameworks on top of RoR, but there are several on top of express. 

here is it. there is no "userland" out there, that handles the rest. there is a "doersland". the node philosophy is for the "do"ers, and it offers too few to them who are "use"ers. i think it's worth to think about how to bridge the node-philosophy into the "use"ers-land, because the ideas of this philosophy still valid in userland as well.

[Alexey Petrushin]

> I think the distinction between *doing* node and *using* is good call.

+ 1, in most cases I don't care about interoperability of my npm modules or complying to some ideologies at all. I just want to get things done with simplest and fastest way. 

[Mark Hahn]

Is "doing node" like "doing lunch"?  

This discussion left me behind some time ago.  I flunked philosophy in college.

[greelgorke]

i think the distinction is made roughly by creating a npm-package (do node), where your code is supposed to be consumed by modules of others vs. creating an app for endusers (use node), where you don't expose many node-interfaces to other modules and mostly consume other modules. 

[Floby]

My Opinion on that matter is that people who think node should provide opinionated utilities and structures should make a "distro" with their opinions. That's how it works with the rest of minimalist software.

for example, the debian creators thought that their system should provide a linux base with GNU utilities and manage its software with fully-tested .deb packages.

[Slobodan Blazeski]

Thanks to everybody for their replies, I tried few options and settled on async for time being and so far it works fine.

If I have special needs in the future I'll take a look at the other options.

best

Slobodan

[Mark Hahn]

> My Opinion on that matter is that people who think node should provide opinionated utilities and structures should make a "distro" with their opinions. 

That is an awesome idea. 

[greelgorke]

strong loop does

[Floby]

Yes, and I find it awesome that they do ;)

[Baz]

I'm a node newb and I don't know enough to make my own judgments between various (generally very professionally written) frameworks/libs. I rely on the wisdom of the community and generally my first step in researching a solution is to search github and order by most stars. I then google to be sure or search the mailing list, which is how I got here. I want to find and use tools that embody the spirit of Node, that most people use, that stay as close to the metal as possible, and with minimum abstractions to get what I need done. The answer I was unequivocally looking for is to use asynch - regardless of whether it's the best or not. 

I ended up reading a lot of Bruno's Ramblings (that's the name of his blog), and a lot about streamline, and it's clear that Bruno's not only hyper-intelligent but an excellent communicator. Streamline seems really cool, but it's way more than an asynch tool, and an entire paradigm shift on how to write node. One that would be irresponsible of me to take on before learning the fundamentals of pure Node, and especially to solve a localized asynch issue. I don't mean to pick on Bruno, especially since all he did was give a simple suggestion several replies below the original post, but he just happens to be really good and prolific at making his case, and if a side had to be chosen I think Mikeal's philosophy on how to answer these non-sophisticated, clearly newb questions is spot on.

That aside, is there a resource somewhere that lists the most popular libs by useful categories? Something like:

Web App Framework: express

Flow Control: asynch

Templating: xxx

etc.

I would love to see a "Mikeal's List" page maintained somewhere.

Best,

Baz

--

[Peter Rust]

> is there a resource somewhere that lists the most popular libs by useful categories?

 

The closest thing I know of is to work down NPM’s “most depended on” list (https://npmjs.org/browse/depended), for each package, you can look at the description and keywords to get a sense of what the category it belongs in.

 

-- peter

[Alexey Petrushin]

The best rating I know https://github.com/languages/JavaScript/most_watched

By the way, not only for JS but for other langs too, usually, when I need something and don't know where to start - I start there, and after reading 3-10 pages found all major solutions for my problem.

I don't like "mentored/opinionated" ratings, usually they are either too small and don't include all the possibilities or outdated or not objective or your opinion don't match with the opinion of those who mentor it :)

[Mark Hahn]

The most_watched list is awesome. Thanks for the tip.

--

[Peter Rust]

There’s also nipster (http://jiyinyiyong.github.io/nipster/), which combines github rankings with npm packages.

 

-- peter