Writing an HTTP benchmarking program

[Avi Flax]

Hi everyone,

These days my standard first program after "hello world" when I'm
learning a new language is a clone of ApacheBench, AKA "ab". (I
recently wrote one in Scala, and I'm working on one in Go.)

So now that I'm exploring Node, I've started working on "NodeBench", or "nb".

Note that this is just for my own educational purposes, I'm not trying
to build software to be released.

Here's my current code:

——BEGIN CODE——

var sys = require("sys");
var http = require("http");

var client = http.createClient(8000, "localhost");
var start_time = new Date();
var target_requests = 5000;
var finished_successful = 0;
var finished_failed = 0;

for (var started_requests = 0;
started_requests < target_requests;
started_requests++) {
var request = client.get("/");

request.finish(
function(response) {
response.addListener("complete",
function() {
if (response.statusCode < 400)
finished_successful++;
else
finished_failed++;

if (finished_successful + finished_failed >= target_requests) {
var elapsed_time_ms = new Date() - start_time;
epilogue(elapsed_time_ms);
}
}
)
}
)
}

function epilogue(elapsed_time_ms) {
var finished_total = finished_successful + finished_failed;
var elapsed_time_secs = elapsed_time_ms / 1000;
var requests_per_sec = finished_total / elapsed_time_secs;

sys.print("\n" + finished_total + " requests completed in ");
sys.puts(elapsed_time_ms + " MS");
sys.puts("Time taken for tests: " + elapsed_time_secs + " secs");
sys.puts("Total requests: " + finished_total);
sys.puts(" successful: " + finished_successful);
sys.puts(" failed: " + finished_failed);
sys.puts("Requests per second: " + requests_per_sec);
}


——END CODE——

Here's what it prints out:

5000 requests completed in 4735 MS
Time taken for tests: 4.735 secs
Total requests: 5000
successful: 5000
failed: 0
Requests per second: 1055.9662090813092


So this is working fairly well, and so far I'm pleased.


My first question is: is there anything I'm doing particularly wrong
here? I'd love some feedback!


My second question is: I don't see any obvious way to make this
faster, but ab is currently getting this done faster:

$ ab -c 1 -n 5000 http://127.0.0.1:8000/

results in:

Time taken for tests: 2.604 seconds
Complete requests: 5000
Requests per second: 1920.27

both seem to be fully saturating both my cores (C2D 2.4).

So: is there anything I can do to make this faster? Or would this have
to be tuned in the internals of Node? Any theories on why is ab so
much faster? Is it because it doesn't run in a VM?


I appreciate any feedback!

Thanks,
Avi

--
Avi Flax » Partner » Arc90 » http://arc90.com
➙ Have you tried Kindling‽ http://kindlingapp.com

[Felix Geisendörfer]

If you want to stay with -c 1 you may want to check if apache bench is
using keep-alive for its requests, that would explain a difference in
speed.

-- Felix Geisendörfer aka the_undefined

> $ ab -c 1 -n 5000http://127.0.0.1:8000/

[Schnalle]

hm. i just want to point out that - as far as i understand it - your
node-ab and the original ab work differently with those arguments (if
not, please correct me):

apache-ab does 5k requests, but only 1 at a time:

ab -c 1 -n5000 url

your node implementation just starts 5k connections as fast as
possible, so it's more comparable to

node-ab -c 5000 -n 5000 url

maybe the difference is because of that.

> $ ab -c 1 -n 5000http://127.0.0.1:8000/

[Avi Flax]

On 2009/12/2 Felix Geisendörfer <fe...@debuggable.com> wrote:

> If you want to stay with -c 1 you may want to check if apache bench is
> using keep-alive for its requests, that would explain a difference in
> speed.

Good suggestion, thanks! I checked the man page, and it says the
default is to not use keepalive.

On Wed, Dec 2, 2009 at 04:00, Schnalle <ste...@schallerl.com> wrote:

> hm. i just want to point out that - as far as i understand it - your
> node-ab and the original ab work differently with those arguments (if
> not, please correct me):
>
> apache-ab does 5k requests, but only 1 at a time:
>
> ab -c 1 -n5000 url
>
> your node implementation just starts 5k connections as fast as
> possible, so it's more comparable to
>
> node-ab -c 5000 -n 5000 url
>
> maybe the difference is because of that.

I think you're right, that makes total sense, thanks!

Wow, thinking about concurrency is tricky… or maybe I'm just not used
to event-based concurrency; when I tried Scala's Actors I believe I
clicked into their behavior immediately.

OK, so, I've adjusted my script to try to emulate ab's behavior, here it is now:

——BEGIN CODE——

var sys = require("sys");
var http = require("http");

var client = http.createClient(8000, "localhost");
var start_time = new Date();
var target_requests = 5000;
var finished_successful = 0;
var finished_failed = 0;

function start_request() {

var request = client.get("/");

request.finish(
function(response) {
response.addListener("complete",
function() {
if (response.statusCode < 400)
finished_successful++;
else
finished_failed++;

if (finished_successful + finished_failed >= target_requests) {
var elapsed_time_ms = new Date() - start_time;
epilogue(elapsed_time_ms);
}

else
{
start_request();

}
}
)
}
)
}

function epilogue(elapsed_time_ms) {
var finished_total = finished_successful + finished_failed;
var elapsed_time_secs = elapsed_time_ms / 1000;
var requests_per_sec = finished_total / elapsed_time_secs;

sys.print("\n" + finished_total + " requests completed in ");
sys.puts(elapsed_time_ms + " MS");
sys.puts("Time taken for tests: " + elapsed_time_secs + " secs");
sys.puts("Total requests: " + finished_total);
sys.puts(" successful: " + finished_successful);
sys.puts(" failed: " + finished_failed);
sys.puts("Requests per second: " + requests_per_sec);
}

start_request();

——END CODE——

Elapsed time is now around 3.5 seconds, and requests per second is
around 1300–1400. This is definitely an improvement than the first
version. But it's still slower than ab.

So: does anyone have any ideas on how to make this faster? Or is it
limited by node at this point?

Thanks!
Avi

[Ryan Dahl]

> So: does anyone have any ideas on how to make this faster? Or is it
> limited by node at this point?

Likely ab will just be faster - just since its written in C and
several years old. But maybe you can run a profile on the program and
try to find out what the bottleneck is. I'm not infront of my computer
right now, but I think if you do node --prof -- myscript.js it will
create a file called v8.log. Pass that file to
deps/v8/tools/linux_tick_counter.py to analyize it. It would be
interesting if there is any obvious things to fix.

[Avi Flax]

On Wed, Dec 2, 2009 at 10:20, Ryan Dahl <coldre...@gmail.com> wrote:

> Likely ab will just be faster - just since its written in C and
> several years old. But maybe you can run a profile on the program and
> try to find out what the bottleneck is. I'm not infront of my computer
> right now, but I think if you do node --prof -- myscript.js it will
> create a file called v8.log. Pass that file to
> deps/v8/tools/linux_tick_counter.py to analyize it. It would be
> interesting if there is any obvious things to fix.

Here you go: http://gist.github.com/250508

I'd love to hear whether this reveals anything specific which can be
improved, in Node or in my script.

Thanks,
Avi

[Tautologistics]

Check out this:

http://github.com/tautologistics/node_loadtest

I haven't really optimized anything because it was meant to test
concurrency right now but it can crank out 1 - 2K req/sec. Of course
this is apples vs. oranges until you run it on your own machine. (Just
be sure to set resDelay to 0 in server.js).

[Tautologistics]

Oh, meant to mention one place where some speed can be picked up is
the anon functions for request.finish and response.addListener.
Declare the functions ahead of time instead so there aren't two new
functions being created for every request.

On Dec 2, 10:04 am, Avi Flax <a...@arc90.com> wrote:

[Avi Flax]

On Sun, Dec 6, 2009 at 22:06, Tautologistics <cpt.o...@gmail.com> wrote:

> Check out this:
>
> http://github.com/tautologistics/node_loadtest
>
> I haven't really optimized anything because it was meant to test
> concurrency right now but it can crank out 1 - 2K req/sec. Of course
> this is apples vs. oranges until you run it on your own machine. (Just
> be sure to set resDelay to 0 in server.js).

Very cool. I've only skimmed it so far, but a bunch of things caught
my eye that I want to go back and study to really understand. Thanks!

On Sun, Dec 6, 2009 at 22:11, Tautologistics <cpt.o...@gmail.com> wrote:

> Oh, meant to mention one place where some speed can be picked up is
> the anon functions for request.finish and response.addListener.
> Declare the functions ahead of time instead so there aren't two new
> functions being created for every request.

Makes total sense, thanks! I tried it and my script went from ~3.5
secs to ~3.2 secs. Every little bit helps!

Avi