When fibers beat handwritten callbacks by 10 to 1

[Bruno Jouhier]

https://gist.github.com/2362015

fibers: 10 times faster!
callbacks generated by streamline: almost 2 times faster!!! (find out why!)

Bruno

[Jann Horn]

On Wed, Apr 11, 2012 at 01:22:42PM -0700, Bruno Jouhier wrote:
> https://gist.github.com/2362015
>
> fibers: 10 times faster!

Uhm... ok, what?

> callbacks generated by streamline: almost 2 times faster!!! (find out why!)

Erm... I like streamlinejs, but I don't see how it can be faster than the hand-written JS version... tell me, please! :D

[Mark Hahn]

Anything is faster than code with a process.nextTick().  The callback example should be rewritten to not use it.

[Jann Horn]

On Wed, Apr 11, 2012 at 02:11:24PM -0700, Mark Hahn wrote:
> Anything is faster than code with a process.nextTick(). The callback
> example should be rewritten to not use it.

I thought so, too - but that gives you a stack overflow. :D

Seems like you'd have to do a process.nextTick() depending on the stack size or so... sounds ugly.

[Mikeal Rogers]

node includes a bunch of great benchmarks for fairly real world performance issues, this is not one.

if you want to try and be faster than callbacks i would suggest you use a real benchmark.

[mscdex]

On Apr 11, 5:22 pm, Mikeal Rogers <mikeal.rog...@gmail.com> wrote:
> node includes a bunch of great benchmarks for fairly real world performance issues, this is not one.
>
> if you want to try and be faster than callbacks i would suggest you use a real benchmark.

+1

[Mark Hahn]

>  I thought so, too - but that gives you a stack overflow. 

That code would give an overflow, that's why I'm saying it should be rewritten.  You can't argue it is impossible because streamline does it without process.nextTick() or stack overflow.

[Marco Rogers]

I would've hoped you had more fortitude against the attacks on streamline then to start throwing around hastily designed benchmarks to prove your point. It sucks that the "sync-looking" advocates have to deal with the level of aggression they do here. I've expressed that to some people myself, even though I don't advocate these solutions either. But this is almost certainly not the way to win hearts and minds.

When you create an abstraction layer, it's almost always slower. That's just something you're not going to get away from. Even if the benchmark wasn't flawed, simply because you can contrive a scenario where it wins doesn't make up for the fact that it stills loses most of the time. Then on top of that, it becomes clear that streamline does some trickery in transforming to js that injects more cleverness. Hence the fact that your example doesn't require an explicit nextTick. So then you leave people with the feeling that if it is ever faster, it's because streamline has done some tricky optimizations in the background.

In my opinion, people who are interested in streamline and fibers will already be prepared to take the performance hit in exchange for ease of use. A better argument would be that you're committed to making that hit as small as possible. And assuring people that a 2x difference won't often make a huge difference in practice. I agree with that, and I think it's easily defensible. This benchmark is not.

:Marco

[Oleg Podsechin]

In my opinion, people who are interested in streamline and fibers will already be prepared to take the performance hit in exchange for ease of use. A better argument would be that you're committed to making that hit as small as possible. And assuring people that a 2x difference won't often make a huge difference in practice. I agree with that, and I think it's easily defensible. This benchmark is not.

When benchmarking Common Node, which uses fibers, I found it to have 60-80% of the throughput of pure Node and use 10-15% more memory. More details here:

https://github.com/olegp/common-node#readme - scroll down to "Benchmarks"

http://www.slideshare.net/olegp/js-everywhere-2011/55

Keep in mind that Common Node adds additional overhead due to to the use of wrapper objects & adding/removing a bunch of listeners on every Stream "data" event.

Oleg

[Jorge]

On Apr 12, 2012, at 1:24 AM, Marco Rogers wrote:

> I would've hoped you had more fortitude against the attacks on streamline then to start throwing around hastily designed benchmarks to prove your point. It sucks that the "sync-looking" advocates have to deal with the level of aggression they do here. I've expressed that to some people myself, even though I don't advocate these solutions either. But this is almost certainly not the way to win hearts and minds.
>
> When you create an abstraction layer, it's almost always slower. That's just something you're not going to get away from.

That's wrong, StreamLined code does not have to be any slower because it simply writes the JS for you:

> On Apr 10, 2012, at 10:35 PM, Bruno Jouhier wrote:
>
>> Streamline translates its special syntax to vanilla JS with callbacks. So, if you implement a function like
>>
>> function foo(x, y, _) { ... }
>>
>> Normal JS modules (not written with streamline) will be able to call it as:
>>
>> foo(x, y, function(err, result) { ... });
>>
>> Streamline is just a tool that helps you write your code. This tool produces vanilla JS that follows the standard node callback conventions. So you can publish it to NPM, you won't be forcing anyone else to use streamline. (...)

> On Apr 10, 2012, at 11:01 PM, Axel Kittenberger wrote:
>>
>> Its not like Bruno is doing an evil plot here. Since as long your
>> benchmark does not compare apple with oranges, it will always be just
>> as fast as vanilla written code, since in the end, it translates to
>> vanilla code.

> Even if the benchmark wasn't flawed, simply because you can contrive a scenario where it wins doesn't make up for the fact that it stills loses most of the time.

Again, wrong. It simply writes for you the ~ exact vanilla JS code you would have had to write yourself.

> Then on top of that, it becomes clear that streamline does some trickery in transforming to js that injects more cleverness. Hence the fact that your example doesn't require an explicit nextTick. So then you leave people with the feeling that if it is ever faster, it's because streamline has done some tricky optimizations in the background.

Totally safe optimizations, as any other good compiler would do. Is that a bad thing too, in your opinion?

> In my opinion, people who are interested in streamline and fibers will already be prepared to take the performance hit in exchange for ease of use.

Please stop with "the performance hit" bs, thanks.

> A better argument would be that you're committed to making that hit as small as possible. And assuring people that a 2x difference won't often make a huge difference in practice.

You're making up numbers.

> I agree with that, and I think it's easily defensible.

Only if it were true, which it isn't.

> This benchmark is not.

FYI the benchmark is a reply to this:

> On Apr 10, 2012, at 4:55 PM, Joe Ferner wrote:
>> On Apr 10, 10:12 am, Bruno Jouhier <bjouh...@gmail.com> wrote:
>>> @joeferner
>>>
>>> (...) From my experience there are code patterns where fibers beats callbacks 10 to 1.
>>
>> beats 10 to 1 how? in code readability?

--
Jorge.

[billywhizz]

the benchmark is completely skewed by forcing the use of process.nextTick. i've done up a better benchmark here and streamline is three times slower than native, even with fibers enabled:

https://gist.github.com/2366955 

[billywhizz]

i also did a benchmark yesterday in response to the other thread which compares the overhead of streamline compared to just calling a native function back. it is 85 times slower on my setup, which means about 10% of every second will be lost in function overhead compared to native callbacks...

https://gist.github.com/2352874 

On Wednesday, April 11, 2012 9:22:42 PM UTC+1, Bruno Jouhier wrote:

[Naouak]

Can't you guys at least use "time" to show how much the system is used during these "benchmark".

In my opinion, with node we are not looking for single request performance but scalable performance.

If one request take about the same time most of the time, I prefer that than one request that goes super fast when there is no one but is slow as hell when there is lot of request to manage.

Please stop making useless "benchmark" about something we don't really care.

Naouak, Grade 2 de Kobal.
Site web: http://www.naouak.net

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

[alessioalex]

If you want sync code that badly, why don't you use EventMachine instead with fibers? You write sync language in Ruby, so it's natural to write "sync-style" code in EventMachine also.

JavaScript is asynchronous, Node.js is asynchronous, everything feels natural (and fibers don't).

> Please stop with "the performance hit" bs, thanks.

Just because you make a wrong benchmark doesn't mean anything.

[Axel Kittenberger]

> JavaScript is asynchronous, Node.js is asynchronous, everything feels
> natural (and fibers don't).

"Everything feels natural"... now thats a scientific argument how
something "feels"?

I wont go in this benchmark gaggle, but I agree with Marco Roger, the
level of agression from the sync-haters is ridiculous. And both sides
eider gaggle around benchmarks, or throw in baseless general
statements like: "When you create an abstraction layer, it's almost
always slower.". Thats an argument in 2012? Haven't decades of
compilers and optimizers tought us (I'm looking at the Assembler vs. C
arguments back in the 1970s-1980s) that eventually a compiler/optimize
can work way beyond human abilities to micro-optimize things, and we'd
rather work on a higher level developing solutions instead of dealing
with the minuscule things how the electrons are moved around in the
most efficient way?

So please everybody get a cool head. Syncs are not going to kill you,
neither is anybody going to be forced to use them, neither is async
everything that uber cool what you think it is, neither is it not
going to change the live of everybody.

[Mark Hahn]

>  , the level of agression from the sync-haters is ridiculous 

I can't speak for others, but I have nothing against sync.  I'm just pointing out that benchmark is flawed.  Plain and simple.

[Marco Rogers]

Thanks for the backhanded support Axel :)

Since when can we not make general statements once they've proven themselves to be generally true? How would we ever have best practices if nothing ever presented itself as best in the general sense?

:Marco

[Mikeal Rogers]

How are we defining aggression?

To me, the same 5 people offering the same answer on every related thread, representing nearly 60% of the traffic in that thread, for an approach that has by any available measure less than 2% of the community adopting it is **very aggressive**.

-Mikeal

[Tim Caswell]

Something productive came out of this thread.  I learned the name of the trampoline technique and came up with a hand-written version of the benchmark that's 4x faster than the streamline-js version.

var count = 1000000;

function bench(cb) {
  var total = 0;
  function loop(i) {
    var async;
    while (async !== true) {
      async = undefined;
      if (i === count) {
        cb(null, total);
      } else {
        load(__dirname + '/benchCallbacks.js', function(err, data) {
          if (err) return cb(err);
          total += data.length;
          if (async) {
            loop(i + 1);
          } else {
            async = false;
            i++;
          }
        });
      }
      if (async !== false) {
        async = true;
      }
    }
  }
  loop(0);
}

Here are the speed numbers I'm getting on my machine with my new trampoline style version asbenchCallbacks2.js.

tim ~/gist-2362015 $ uname -a
Linux touchsmart 3.2.13-1-ARCH #1 SMP PREEMPT Sat Mar 24 09:10:39 CET 2012 x86_64 Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz GenuineIntel GNU/Linux
tim ~/gist-2362015 $ node benchCallbacks.js 
hit=999999, missed=1, result=805000000
elapsed: 1297
tim ~/gist-2362015 $ _node benchStreamline
hit=999999, missed=1, result=805000000
elapsed: 1001
tim ~/gist-2362015 $ _node --fibers benchStreamline
hit=999999, missed=1, result=805000000
elapsed: 180
tim ~/gist-2362015 $ node benchCallbacks2.js 
hit=999999, missed=1, result=805000000
elapsed: 308

Most applications won't notice the speed differences here, but I do appreciate learning a new technique I'm sure to use in the future.

[Bruno Jouhier]

I wrote this bench because I claimed yesterday that fibers can be 10 times faster than callbacks on some code patterns and someone dismissed it with a sarcasm. I was also getting really tired of all the FUD that people spread around streamline and fibers.

This bench did not come from nowhere, it came from the experience of a real development that I did with node last year. This dev consisted in porting a complex component written in C to node. This code is doing I/O but it is also executing complex logic with lots of loops, tests, exception handling, etc. It is not "just another proxy".

As performance is a critical factor for this component I started to write it with manual callbacks (yes, I can do this, and I can even enjoy it!). I used nextTick to avoid stack overflow with loops but this was not fast enough (half the speed of the original C component). So I replaced nextTick by trampolines. This improved performance (component was only 20% slower than the original C version) but the code became really hard to debug.

Around the same time Marcel Laverdet had written a fibers backend for streamline. Fibers introduce overhead in I/O calls but they also make everything else much leaner and thus faster: loops don't need to be converted to recursive form any more, functions don't need to trampoline, values are returned by a simple "return" statement, not by a callback, you don't have to test for exceptions and propagate them with callbacks at every level, you can just set up one try/catch at the high level, etc. I did some small benchmarks which showed that fibers could be much faster on some code patterns (10 times on caching patterns but a lot more on some pathological recursive patterns).

I thought that, given the code patterns that I had in my component, fibers might give faster code than callbacks overall. So I converted the code to streamline and I executed it in fibers mode. The result is not 10 times faster than the manually written version but it got a little closer to the C version (almost on par). But the code was much simpler and debugging became really pleasant (with meaningful stack traces). So I decided to switch to streamline + fibers for this subproject.

Of course, the bench is not representative of what you will get on average with fibers but it shows that fibers can speed up some patterns (caching in this example). It also shows that manually written loops that use nextTick to avoid stack overflow are slower than streamline generated loops that use a trampoline variant instead.

I only have two comments for billywhyzz:

1) callback !== asynchronous: function(cb) { cb(); }  is *not* an asynchronous function. There is no reason to give it an _ parameter in streamline and there is no need for an _ in the function that calls it either. Your bench just shows that you don't understand streamline. Bench with a real async call. With process.nextTick streamline (callback mode) introduces a 50% overhead, with setTimeout(cb, 0), the overhead becomes 18%, if this can make you happy.

2) hacking the loop by going through nextTick once every 1000 iteration is dangerous. Exit loop after 900 iterations => loop has eaten 1800 stack frames. Do this a few times on different caches and you'll blow the stack. The streamlined loop runs with a fixed stack depth.

[Bruno Jouhier]

This pattern is fine for an async library helper but I doubt that people will be ready to write all their loops like this. I'm just thinking of the poor guy who's going to maintain this kind of code written by someone else.

nodejs+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to

nodejs+unsubscribe@googlegroups.com

[Axel Kittenberger]

It took me 30 minutes staring on those 30 lines to understand Tims code ;-)

>>> nodejs+un...@googlegroups.com

>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>>
>>>
>>> --
>>> Job Board: http://jobs.nodejs.org/
>>> Posting guidelines:
>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
>>> You received this message because you are subscribed to the Google
>>> Groups "nodejs" group.
>>> To post to this group, send email to nod...@googlegroups.com
>>> To unsubscribe from this group, send email to

>>> nodejs+un...@googlegroups.com

>>> For more options, visit this group at
>>> http://groups.google.com/group/nodejs?hl=en?hl=en
>>
>>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to

> nodejs+un...@googlegroups.com

[Jorge]

On Apr 12, 2012, at 10:51 PM, Tim Caswell wrote:

> Something productive came out of this thread. I learned the name of the trampoline technique (...)

See "Proper Tail Calls" in <http://peter.michaux.ca/articles/javascript-is-dead-long-live-javascript>

Enjoy!
--
Jorge.

[Mark Hahn]

Good article.  I learned a few things.