Strange bug in benchmark, Express.js is slower than Ruby on Rails

[Alexey Petrushin]

I ran a benchmark for Ruby on Rails and Express.js using `siege` - and Express for some reason performed worse than Rails.

How benchmark works - application query some text from remote HTTP service (service delays each

request for 200ms) and render HTML page using that text.

Results

- transactions rate are the same (already strange, I expected node.js to outperform rails by at least a couple of times)

- average response time in rails is 0.8s and in express 4s (siege doesn't show average time it in the report, but it can be seen from its logs).

Details

The remote service with delay (sipmle node.js server) https://github.com/alexeypetrushin/web-frameworks-benchmarks/blob/master/lorem-ipsum.js

Express.js https://github.com/alexeypetrushin/web-frameworks-benchmarks/blob/master/expressjs/app.js

Rails (with Puma server) https://github.com/alexeypetrushin/web-frameworks-benchmarks/blob/master/rails/app/controllers/posts_controller.rb

Siege command `siege -b -t10s -c100 http://localhost:3000`

Full code and instructions how to run https://github.com/alexeypetrushin/web-frameworks-benchmarks

All application has been run in production mode.

How is that possible and where is the bug in this benchmark, what's I'm doing wrong?

[Fedor Indutny]

I think you could be hitting this:
http://nodejs.org/api/http.html#http_agent_maxsockets

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups
> "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Matt]

Also important that your node code is only using one CPU. See http://nodejs.org/docs/v0.10.22/api/cluster.html

[Alexey Petrushin]

Single core shouldn't be a problem as far as I know ruby also uses only one CPU it doesn't have real threads.

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/tgATyqF-HIc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards, Alex.

[Matt]

On Thu, Jan 23, 2014 at 9:33 AM, Alexey Petrushin <alexey.p...@gmail.com> wrote:

Single core shouldn't be a problem as far as I know ruby also uses only one CPU it doesn't have real threads.

Apparently for Puma this changes if you're running under Rubinius or JRuby.

But I strongly suggest looking into what Fedor suggested.

[Alexey Petrushin]

Fedor, you are right, thanks. That was the problem, after fixing it node became more than 10 times faster.

Issue resolved.

[Fedor Indutny]

You are welcome!

On Thu, Jan 23, 2014 at 7:19 PM, Alexey Petrushin

[Reza Razavipour]

what was the fix? You changed the 5 to what?

[Alexey Petrushin]

I've changed agent.maxSockets

https://github.com/alexeypetrushin/web-frameworks-benchmarks/commit/bac85d4485809d7dca59d9a1ea64445c37af71de

[tjholowaychuk]

plus this benchmark doesn't really test express, it tests outbound requests AND express :p so that's really misleading 

[Alexey Petrushin]

Yes, but I wanted to see a test that more or less close to what's used in real projects. Express usually don't work in isolation, usually it uses some sort of database. And request seems like a good approximation for DB connection.

[Paul Vencill]

I didn't know about this either, good info.  Is there a good method for determining what the optimum # max connections should be? I assume there's some practical limit, or way of figuring out the point of diminishing returns for a given app?  for example, if you open up (as the OP did) 1000 concurrent connections, but your db connection pool is only 10, then you're probably going to run into issues.  Other than dependencies like that, any good rubrics y'all can think of?

[Stefan Klein]

2014/1/24 Paul Vencill <paul.v...@gmail.com>

I didn't know about this either, good info.  Is there a good method for determining what the optimum # max connections should be? I assume there's some practical limit, or way of figuring out the point of diminishing returns for a given app?  for example, if you open up (as the OP did) 1000 concurrent connections, but your db connection pool is only 10, then you're probably going to run into issues.  Other than dependencies like that, any good rubrics y'all can think of?

I think this will be absolutely app specific.

For your example, it could be fine to have 1000 concurrent connections (maybe even much more) and just 10 database connections if your app uses long polling to notify clients on new messages.

[Alexey Petrushin]

by the way, 100 connections work the same and give the same numbers, I set 1000 just in case :)

[Alexey Petrushin]

Update, after tuning the Rails with Puma Server ( thanks to https://github.com/antage ) its speed increased in ten times and is almost the same as Node (it uses only one process and it seems only one core).

Updated benchmark https://github.com/alexeypetrushin/web-frameworks-benchmarks

Rails

Transactions:               2258 hits

Longest transaction:            0.52

Express

Transactions:               3158 hits

Longest transaction:            0.34

[Alexey Petrushin]

Maybe there's some other resource limit that prevents node to work in full power? Node.js process uses only 50% of CPU but if I raise concurrency over 100 it starts to generate connection errors. 

What maybe the cause?