Poor performance when behind proxy and low concurrency level

[Radosław Bułat]

Hi!

Today I tested ebb with simple merb application behind nginx and
apache servers in front.
It looks like ebb has poor performance when there is low concurrency
level.

My scenario:

I created merb application and created simple rack handler.
config/rack.rb

class HelloWorldHandler
def initialize(app)
@app = app
end

def call(env)
if env['REQUEST_URI'] == '/handler_hello'
return [
200,
{"Content-type", "text/html"},
"Hello World"
]
else
return @app.call(env)
end
end
end

use HelloWorldHandler
run Merb::Rack::Application.new

(so when I get http://domain/handler_hello it returns "Hello World"
document.

First test:

$ merb -p 3000 -e production -a ebb -c 1
Starting merb server on port: 3000

$ ab -c 1 -n 10000 http://rackapp.local:3000/handler_hello
Requests per second: 4378.92 [#/sec] (mean)

So far it's very good.

I configured nginx to pass request to 3 ebb processes (nginx runs on
81 port).

$ merb -p 3000 -e production -a ebb -c 3
Starting merb server on port: 3000
Starting merb server on port: 3001
Starting merb server on port: 3002

$ ab -c 1 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 387.53 [#/sec] (mean)

$ ab -c 2 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 627.93 [#/sec] (mean)

As you see there is only 387req/s. But when I increase concurrency to
3 and later to 20:

$ ab -c 3 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 2251.51 [#/sec] (mean)

$ ab -c 20 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 2967.70 [#/sec] (mean)

There is some connection between number of processes and concurrency
level.

I tested ebb also with apache and there is similar relation with low
concurrency level and requests per seconds (but performance is much
worse):

Apache runs on 80 port:
$ ab -c 1 -n 100 http://rackapp.local/handler_hello
Requests per second: 36.45 [#/sec] (mean)

$ ab -c 2 -n 100 http://rackapp.local/handler_hello
Requests per second: 56.95 [#/sec] (mean)

$ ab -c 3 -n 200 http://rackapp.local/handler_hello
Requests per second: 73.14 [#/sec] (mean)

$ ab -c 20 -n 2000 http://rackapp.local/handler_hello
Requests per second: 556.18 [#/sec] (mean)

$ ab -c 100 -n 2000 http://rackapp.local/handler_hello
Requests per second: 791.16 [#/sec] (mean)

$ ab -c 200 -n 10000 http://rackapp.local/handler_hello
Requests per second: 2881.07 [#/sec] (mean)

I tested nginx and apache with the same application but as application
server thin and mongrel. Results for low concurrency level was much
much better than for ebb.

# thin
$ merb -p 3000 -e production -a thin -c 3
Starting merb server on port: 3000
Starting merb server on port: 3001
Starting merb server on port: 3002

# apache
$ ab -c 1 -n 10000 http://rackapp.local/handler_hello
Requests per second: 1459.45 [#/sec] (mean)

# nginx
$ ab -c 1 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 1978.73 [#/sec] (mean)

# mongrel
$ merb -p 3000 -e production -a mongrel -c 3
Starting merb server on port: 3000
Starting merb server on port: 3001
Starting merb server on port: 3002

# apache
$ ab -c 1 -n 10000 http://rackapp.local/handler_hello
Requests per second: 987.58 [#/sec] (mean)

# nginx
$ ab -c 1 -n 10000 http://rackapp.local:81/handler_hello
Requests per second: 1184.62 [#/sec] (mean)

Any comments?

[Radosław Bułat]

I think it could be related to how ebb handle connections or something
like that. When I test it with apache benchmark tool (ab) directly to
ebb application (without proxy) I have results like this:

Requests per second: 4253.11 [#/sec] (mean)

But when I use httperf results are much worse:

Request rate: 542.9 req/s (1.8 ms/req)

When I use thin or mongrel results are very similar regardless I use
httperf or ab.

--
Radosław Bułat

http://radarek.jogger.pl - mój blog

[Radosław Bułat]

Maybe someone can reproduce it? Nobody interested why it's so slow?

[ry dahl]

> Maybe someone can reproduce it? Nobody interested why it's so slow?

I am :) I think you're correct. I suspect that Ebb is not properly
closing the connections. ab is notorious for not measuring completed
requests.
ry