Benchmarking Node with Observers

[ry]

I've implemented the "addListener" observer pattern for most of Node.
I think it looks good and has cleaned up a lot of the internal
interfaces. I wanted to see how it compared to the original simple
callbacks.

To do this I ran a simple web server which when you go to /bytes/123,
it returns 123 bytes of data. When you go to /bytes/24576 it returns
24kb of data. The code for this web server is in the repository.

I tested the new code "node (events)" against the old code "node
(callbacks)". For comparison I threw in the results of two Ruby
language web servers, "ebb" and "thin".

I ran two runs with apache bench on all 4 servers: once with 123 byte
responses and once with 24 kb responses. Each run was done with 30
concurrent clients (-c 30) for 60 seconds (-t 60). I recorded the data
to a csv file with -g and made the graphs with R. (See
http://four.livejournal.com/955976.html?thread=2677320#t2677320 for
the script I used.)


Results:

123 bytes, histogram:
http://s3.amazonaws.com/four.livejournal/20090627/small_hist.png

123 bytes, time series (note, ab stops after 50,000 requests, that's
why it doesn't last for the full 60 seconds)
http://s3.amazonaws.com/four.livejournal/20090627/small_ts.png

24kb, histogram
http://s3.amazonaws.com/four.livejournal/20090627/big_hist.png

24kb, timeseries
http://s3.amazonaws.com/four.livejournal/20090627/big_ts.png

raw data: http://s3.amazonaws.com/four.livejournal/20090627/stats.tar.bz2



Conclusion:

There is a definite performance hit using more complex event
interface. This is most observable in the 123 byte benchmark. This
makes sense since events are being triggered more rapidly. Looking at
larger responses, however, the difference is nearly negligible.

[ry]

The more I think about it, the less happy I am with the results. I'm
not sure if the slowdown and additional code are worth the nicer
interface.

> summary(node_callbacks_123$ttime)
Min. 1st Qu. Median Mean 3rd Qu. Max.
4.00 22.00 23.00 24.77 23.00 103.00

> summary(node_events_123$ttime)
Min. 1st Qu. Median Mean 3rd Qu. Max.
11.00 25.00 25.00 28.56 25.00 219.00

If your interested in the observer version, checkout
http://github.com/ry/node/tree/events
src/http.js should be a good example of the new API

[Aaron Quint]

I mean, the API looks like it could be pretty useful/cool. Though the
more I thought about it after seeing your benchmarks, it wouldnt be
too hard to build an higher level event observer system. The one thing
that it would be nice to have though is the framework level events to
bind/listen to. Would a good middle ground be maybe firing the events
off, but not using them to actually trigger the different methods
internally?

--AQ

Aaron Quint
http://www.quirkey.com

[ry]

I've made some more changes. (Basically removing the possibility of
using legacy onBody = function () {} style callbacks.) And I was able
to bring the benchmark into a rather acceptable range.

http://s3.amazonaws.com/four.livejournal/20090627/ts1b.png
http://s3.amazonaws.com/four.livejournal/20090627/hist1b.png

This is tested with just a one byte response.

> summary(callbacks$ttime)

Min. 1st Qu. Median Mean 3rd Qu. Max.

7.00 23.00 23.00 24.39 23.00 102.00

> summary(events$ttime)

Min. 1st Qu. Median Mean 3rd Qu. Max.

13.00 22.00 23.00 25.07 23.00 193.00