Synchronous?

[Kevin Burton]

I notice that simply leaving off the callback or making a callback null doesn't always make a node function synchronous. What is a good general pattern for making a node method synchronous (assuming the source for the method is available)?

[Gary Katsevman]

You can’t. Some methods have a sync counterpart, usually, they’re paired functions kind of like foo and fooSync.
You could use a control flow library to make async functions look and feel slightly more sync.

Gary Katsevman
gkatsev.com

On Fri, Apr 18, 2014 at 11:24 AM, Kevin Burton <ronald.ke...@gmail.com> wrote:

I notice that simply leaving off the callback or making a callback null doesn't always make a node function synchronous. What is a good general pattern for making a node method synchronous (assuming the source for the method is available)?

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Kocharin]

 

18.04.2014, 19:24, "Kevin Burton" <ronald.ke...@gmail.com>:

I notice that simply leaving off the callback or making a callback null doesn't always make a node function synchronous. What is a good general pattern for making a node method synchronous (assuming the source for the method is available)?

 

 

A usual advice about making a node method synchronous is "don't do that".

 

[Alexey Petrushin]

Actually you can, with Fibers. Making asynchronous function to behave like synchronous is as simple as

    fn = sync(fn)

More details http://alexeypetrushin.github.io/synchronize

[Kevin Burton]

Alex, I think in this case I need to. I have about 20 functions to execute, each of which depends on the side-effects of the previous function. I can basically make it synchronous by making what looks like a big 'V'

a(function(err) {

   if(err) {

   . . . . .

   } else {

      b(function(err) {

          if(err) {

          } else {

              c(function(err) {

                  if(err) {

                  } else {

                  }

              });

          }

      });

  }

});

[Aria Stewart]

On Apr 18, 02014, at 14:52, Kevin Burton <ronald.ke...@gmail.com> wrote:

> Alex, I think in this case I need to. I have about 20 functions to execute, each of which depends on the side-effects of the previous function. I can basically make it synchronous by making what looks like a big ‘V'

Check out http://callbackhell.com

You can also write a little glue (or use libraries) like async to do this a bit more tidily, or consider using Promises.

[// ravi]

On Apr 18, 2014, at 2:52 PM, Kevin Burton <ronald.ke...@gmail.com> wrote:

Alex, I think in this case I need to. I have about 20 functions to execute, each of which depends on the side-effects of the previous function. I can basically make it synchronous by making what looks like a big 'V'

Consider a library like FlowJS (easy) or a more complete idea like Promises (Q, Bluebird). Many folks like using Async - I find it non-intuitive but it might help you. To state what might be obvious, It is important to keep in mind that you are not making the function synchronous by using any of these methods (i.e., your process is not going to block on the call), but rather, you are just structuring your code to avoid the callback hell / pyramid of doom.

—ravi

[Bruno Jouhier]

a(_);
b(_);
c(_);

Bruno

[Bruno Jouhier]

[Kevin Burton]

What is the '_' is that some special character to get it to run synchronously?

[Bruno Jouhier]

Yes. Sorry guys, I could not resist the temptation of advertizing: https://github.com/Sage/streamlinejs

Bruno

[Alex Kocharin]

 

It's someone trying to sell another javascript-like language as an answer to this simple problem. :)

 

If you have 20 subsequent functions, you can use generators, that's what they are good at.

 

 

19.04.2014, 01:46, "Kevin Burton" <ronald.ke...@gmail.com>:

[Alexey Petrushin]

There are following solutions to fight callbacks (my personal opinion in brackets):

0. callback hell (simple and good to for working with concurrency, but too verbose to be usable in day to day work when in 99% of cases you work with plain sequential logic)

1. statemachine hell (when you trade callbacks hell to bunch states & named functions, just a little better than callback hell, most frequently advised approach, a little better than callback hell)

2. async helpers (just a little better, but still far from being simple)

2. generators (simple, and seems to be good one, although, seems there are some gotchas with robust error handling)

3. code generation (some tools are good, but most are not, also there may be gotchas with error handling).

4. fibers (resulting code is simplest, it is exactly the same as plain synchronous, works in 99.99% of cases, handles errors almost perfectly, but, in 0.01% you need to be careful and know what you are doing).

[Kevin Burton]

Thank you very much for all the replies. This has been most enlightening. Based on all of the replies and my situation I would like to explore Promises more. I think I understand the basic idea behind promises but lack specific implementation details. Does anyone have a good example of using Q and Node.js like the readme indicates:

```javascript

step1(function (value1) {

    step2(value1, function(value2) {

        step3(value2, function(value3) {

            step4(value3, function(value4) {

                // Do something with value4

            });

        });

    });

});

```

With a promise library, you can flatten the pyramid.

```javascript

Q.fcall(promisedStep1)

.then(promisedStep2)

.then(promisedStep3)

.then(promisedStep4)

.then(function (value4) {

    // Do something with value4

})

.catch(function (error) {

    // Handle any error from all above steps

})

.done();

```

I get lost with denodify and deferred which the above simplification doesn't address. Any clarification for the newly initiated?

Thank you.

[Bruno Jouhier]

I probably sounded a bit weird yesterday. A quick explanation: I've been advertising a bit loudly for my own solution on this mailing list some time ago. I'm trying to be a bit more discrete these days.

But just for the fun of it, I'll do it one more time. Here is what your last example becomes:

var value4 = step4(step3(step2(step1(_), _), _), _);

So if you are ready to go with a little (transparent) preprocessor, this is what you can expect your code to look like.

A quick comment on Alexey's list (callback hell, statemachine hell, etc.): If you pick the right tools, the last 3 solutions (preprocessor, generators, fibers) become almost equivalent: they all give you plain synchronous code with robust error handling, meaningful stack traces and good debugging experience (thanks to sourcemaps). What matters then is performance and here it depends on your code patterns.

Bruno

[Floby]

Can everybody be adult about this once and for all?

Your problem is not asynchrony vs synchrony. Please forget these words. Nothing ever happens synchronously on a computer. You've been lied to all your life, sorry.

Your problems is about flow control. Flow control is a set of patterns for scheduling atomic actions. Some actions can be gathered together as a single operation. These are functions in javascript. Some operations need to wait for something else to finish. These are async operations. In some environments, async operations are made to look synchronous using things such as threads, processes and fibers. But they are not synchronous. If you're comfortable with hiding this complexity, you can use python or ruby (or anything else, really)

The problem you are having can be solved by choosing a method of scheduling. In node the following methods exist.

Pipelines : streams and pipe(). These are core core methods. Several stream modules exist.
Callbacks scheduler : async, flowjs, step, etc.
Promises : rsvp, futures, etc.
Transpilers : streamline, iced coffeescript (these are not strictly javascript)
Async/wait : fibers, galaxy, etc. (these need a special version of node)

The names on the right hand side are npm modules.

It is up to you to choose what method you prefer. They all have their pros and cons. My advice is to read up on each at callbackhell.com.

My second advice is to ask people trying to sell you their stuff in which category their stuff belongs.

My third advice is to use async.waterfall from the async library. And this is my favourite way of doing things.

Hope this makes things a bit clearer.

Floby.

[Bruno Jouhier]

Floby, some quick remarks/corrections:

• streams and pipes are a high level concept. They don't really help with sync-style in low level code (dealing with sequence of calls, conditionals, loops, exception handling, etc.)
• fibers don't need any special version of node. Fibers is a module that you install with npm, like any other node module.
• generator based solutions (suspend, galaxy, co, genny, etc.) require node 0.11 or higher. Calling that a "special" version of node is a bit misleading because it seems to imply that node has been tweaked.

Kevin, I guess that the tone of some of these exchanges may surprise you a bit. The explanation is that there have been many sour discussions around this topic in the past (and I've been one of the most active participants). Things are more relaxed these days but people tend to be still a bit edgy. To clarify where I stand: I wrote on of the transpiler solutions (streamline) as well as one of the generator-based libraries (galaxy). I'm also supportive of fibers because it is one of the runtime options behind streamline and this is the one that we chose to release our own product.

My only advice to you is to *not* trust what you will read from a single source like callbackhell.com but to try things out and make up your own mind. Also google around. Look for benchmarks and real-world experiences with the solutions, not just opinions.

Bruno

[Daniel Rinehart]

Just wanted to interject a quick reminder that "flow control" is not the same as "control flow". Important to keep in mind especially if talking about using streams and pipes to handle "control flow" as the stream pipe() method internally implements "flow control".

-- Daniel R. <dan...@neophi.com> [http://danielr.neophi.com/]

[Alessandro Pellizzari]

Il Sat, 19 Apr 2014 03:43:41 -0700, Floby ha scritto:

> Your problem is not asynchrony vs synchrony. Please forget these words.
> Nothing ever happens synchronously on a computer. You've been lied to
> all your life, sorry.

I think you are confusing async and sync.

Nowadays, both sync (serial) and async (parallel) operations can happen
on a PC, as every single core in the CPU can work in parallel with the
others.
Normally node runs on a single process (and core) so it's not
multithreaded (unless you use some specific modules), and simulates
preemptive multitasking like the OS scheduler do. Async operations in
node are necessary to avoid blocking the "core" (the whole app) while
waiting, for example, for a file to be read or a socket to receive data.

> Some operations
> need to wait for something else to finish. These are async operations.

No. those are sync operations.

> In some environments, async operations are made to look synchronous
> using things such as threads, processes and fibers.

And those (parallel) are async.

> My third advice is to use async.waterfall from the async library. And
> this is my favourite way of doing things.

async.waterfall is simply a method to transform async operations in sync
ones (executing them one after another).

Bye.

[// ravi]

On Apr 19, 2014, at 12:28 AM, Kevin Burton <ronald.ke...@gmail.com> wrote:

Thank you very much for all the replies. This has been most enlightening. Based on all of the replies and my situation I would like to explore Promises more. I think I understand the basic idea behind promises but lack specific implementation details. Does anyone have a good example of using Q and Node.js like the readme indicates:

<snip happens>

I get lost with denodify and deferred which the above simplification doesn't address. Any clarification for the newly initiated?

Kevin,

broadly, in the Q/promises world, it may help to think of functions as one of four types: (1) something that returns a value immediately, (2) something that is asynchronous and returns a value later to a callback using the node convention callback(err, result), (3) something that is asynchronous and returns a value later to a callback in a manner different from the node convention (or via an event, which I’ll ignore here), and finally, what we want: (4) something that returns a promise (whose return value can be obtained using its then() method).

To structure your code using promises, you may want some or all such functions to return promises. Of course only the last one does, and so, Q provides a few mechanisms to coerce the others types to do so as well: denodify/ninvoke/nbind/etc are examples for type (2). Q.fcall() lets you wrap a return value as a promise. And Q.defer() lets you create and return your own promises for #4 or solve #3.

Say you want to connect to a DB using a node-db module that uses standard Node conventions, and you want to insert a record, and finally write a log message using an async log routine that does not follow node conventions. You might do this (perhaps a bit contrived, to demonstrate the various Q-bits):

var Q = require(‘q’);

var db = require(‘db’);

var logger = require(‘logger’);

Q.ninvoke(db, “connect”, { host : ‘localhost’, user : ‘notroot’, password ; ‘super secret’ })

.then

(

function(dbconn)

{

return( Q.ninvoke(dbconn, ‘insert’, { first : ‘Johnny’, last : ‘Appleseed’, device : ‘5s’ } ));

}

)

.then(write_log)

.fail(function(err) { console.error(err); })

.done(process.exit);

function write_log(result)
{
var deferred = Q.defer(); // we are going to wrap the async logger in a promise
var logmsg = “DB :: INSERT :: new ID “ + result.id;
logger.write
(
logmsg,
function(rc)
{
if( rc === -1 ) // logging failed
deferred.reject(new Error(“Write to DB log failed.”));
else
deferred.resolve();
}
);
// we return a promise which will be fulfilled when logger.write() returns
return(deferred.promise);
}

Note that all function references passed to .then() are to promise-returning functions. In write_log() you see an example of how you can create your own promise and return it, in this case, wrapping an async routine that uses a non-node style callback.

In the above, we handle errors that can occur at any step in the chain at the end in the fail() method, but Q will let you catch them in the then() if you provide a second function reference: promise.then(success_func, fail_func). In either case, Q is smart enough to pass the first argument (err, in node style) to fail_func (or the function referenced in fail()) and the second argument (the result of the async operation in node style) to success_func.

If you have more questions specifically about Q promises, you may also wish to join the Q mailing list (https://groups.google.com/forum/#!forum/q-continuum) and ask there.

Regards,

—ravi

[Tim Caswell]

I would like to bring some experience and history to this conversation.  First I've been writing node libraries and programs since 2009.  Many of you know me and most of you probably don't because the node community has grown exponentially over the past few years.

Shortly after node was made, the concept of the event emitter was added.  We didn't have the current callback-last style, but had a convention that non-blocking functions would return a special emitter known as a "Promise".  Usage was something like this: <https://github.com/creationix/node-router/blob/5d731874a210829ac389d7c978daa97f74a3c7e3/http_server.js#L176-L190>

    var promise = node.fs.cat(filename, encoding);

    promise.addCallback(function (data) {
      body = data;

      headers = [ [ "Content-Type"   , content_type ],
                  [ "Content-Length" , body.length ]
                ];
      headers.push(["Cache-Control", "public"]);
       
      callback();

    });

    promise.addErrback(function () {
      notFound(req, res);
    });

As you can see it was quite verbose.  After almost a year of this we (the smallish node community) decided to find a better control-flow pattern.  My personal vote was for what I called shotgun continuables.  These were a type of promise, but much easier to use than the event emitter based promise in node (and *much* simpler than the A+/ES6 promise that's popular today) <http://howtonode.org/do-it-fast>

In the end, Ryan decided to go for callback-last with error as the first argument (current node style).  This was the most minimal interface and supposedly the fastest.  Performance was a *big* deal because that was how we were getting people excited about node to grow the community.  Node still avoided functions that blocked on I/O of any kind, but did include blocking versions of most the fs operations.  The pattern was to leave out the callback, add "Sync" to the end of the name and return the value or throw the error.  This is still the suggested style if you want to make a blocking version of a function.

Shortly after this the concept of using streams wherever possible was baked into node, sys.pump and later Stream.prototype.pipe were added to be able to treat streams as first-class values.  This cleaned up a lot of code that involved streams of data that needed layers of processing transforms applied.

Node grew and grew, but the primitives for control-flow stayed the same.  New people complained that callbacks were hard because they didn't understand them.  Large companies started adopting node, etc.  Eventually the A+ crowd started using node more and were evangelizing their view of how control-flow should be done.  Bruno adopted node at his large company, but decided to create streamline (a source to source transform) to handle the complexity of callbacks in logic heavy code.  I later tried to push for continuables again, but this time closer to the node style with a combined (err, value) function.

Node was still growing at an amazing rate and the ratio of new people to experienced people was getting pretty high.  I decided to port node.js to lua <http://luvit.io> to experiment with using coroutines in this style of non-blocking I/O.  It was an interesting experiment and I learned a lot of techniques that are used today by libraries like gen-run, co, and galaxy.  At the same time, node-fibers was created to give full coroutines to node.js as a C++ addon.  It works really well and lets you do all the same things I was doing in lua, but do them in node.  ES6 (the next version of JavaScript) added generators and during the node 0.11 dev cycle (which we're *still* in) they were implemented in V8 and added to node behind a flag.  <http://howtonode.org/generators-vs-fibers>

Also thanks to the hard work of the A+ crowd, promises became part of ES6 and many browser APIs are moving that direction.

So today we have several years of experience using the node-style callbacks with many sites like howtonode.org and callbackhell.com giving tips to be more effective at writing your own control-flow logic.  We have multiple competing promise libraries (all converging on A+) and the language itself including promises soon.  We have full coroutines in node-fibers, light more verbose coroutines in generators, source transforms like streamline, etc.  This question of what is best is far from settled and there have historically been quite the heated debate on this list over the years.

Personally I write most my code in node-style callbacks because it's simple, I'm experienced in it, and I just want to get stuff done.  I don't know where the future lies.  I expect to see more of promises since it's gotten into the language itself.  I expect to see source transforms used more and more as build tools mature and all browsers have sourcemaps.  Generators are getting more widespread.  Stable Firefox has them today without a flag.  node 0.11.x has it today with a flag, meaning you can use it as long as you control the server (so app authors, not library authors).

-Tim Caswell

aka @creationix

[Tim Caswell]

Oh, and as far as terminology goes.  I prefer blocking and non-blocking.  A blocking function waits for the result and returns it.  A non-blocking function returns right away and gives you the result later through a function you pass-in somehow.  Node draws the line to block on anything that's quick and non-block on slow things like network I/O, disk I/O, timers and some expensive CPU operations like crypto and compression.

The efficiency in systems like node are because it's able to combine I/O waits in parallel, but still be single-threaded for the main logic.  Generators still have this property, but change the syntax so that non-blocking operations don't block the entire process, but do block your local logic.  As long as you're aware of how this works, it's the best of both worlds.  If you're not aware and it's not implemented well, it's the worst of both worlds.

In summary, we're still learning.  Welcome and have fun!

[// ravi]

On Apr 19, 2014, at 1:02 PM, Tim Caswell <t...@creationix.com> wrote:

I would like to bring some experience and history to this conversation.

Great post Tim, thank you for giving us a glimpse of the past.

Re: your closing comment, I too was going to suggest use of blocking/non-blocking earlier in this thread (and in fact, I sort of adopted such a usage, myself, in my earlier comment), but on second thought, I think it is equally misleading (at least to me). Whether a call is blocking or non-blocking often depends on how it is used (such as I/O system calls which leave it to the caller to decide whether to block or not). Of course I realise we are speaking of node module calls not system calls.

Regards,

—ravi

[Tim Caswell]

Ravi, I don't think it's misleading to say that something blocks a thread or blocks a generator or blocks a function.  To me, all blocking means is that the logic can't continue till the thing has finished.  A non-blocking function will return immediately and let you run other code while waiting for the result.

So to me, a yield in a generator that suspends the generator till some non-blocking operation is complete is "blocking" in the context of the generator function, but not blocking for the entire process.  The single threaded node server can still handle and process other requests.

Though I could see how saying "this yield blocks on the non-blocking function" would confuse someone.

 

[Floby]

It seems to me that `yield` instructions block the same way that `wait` blocks with thread or processes. async/wait is more or less threading (cooperative threading).

[Bruno Jouhier]

On Thursday, April 24, 2014 10:11:21 AM UTC+2, Floby wrote:

It seems to me that `yield` instructions block the same way that `wait` blocks with thread or processes. async/wait is more or less threading (cooperative threading).

Yes. The main difference is that you know where yield will transfer control (to the generator.next() call which started or resumed the generator) whereas a wait() call issued from a thread will just yield to the scheduler.
 

[Bruno Jouhier]

Tim, your historical perspective is great. I got involved later than you, in 2010, a couple months before Ryan decided to drop promises and go with callbacks. So in our project, we did start with promises, then we went back to callbacks for a few months, and then to streamline.

I noticed one small error in your chronology: fibers appeared only a few days after streamline. They were both released in January 2011.

Also, streamline was not the first transpiler. narrative.js and stratified.js were there before. I got my syntax inspiration from narrative but I went with a different algorithmic approach.

And as you say, the debate is still not settled today. It's a bit of a jungle.

Bruno