Benchmarks

[Gustavo Niemeyer]

Hello,

I'm researching a bit about current options to create a web
application (like everyone else, it seems :-) where there are
real speed constraints. I was quite happy to see the benchmark[1]
stating more than 500 connections per second, but I couldn't
reproduce that result in my own tests. Using the code below,
the command "ab2 -n 1000 http://localhost:8000/" states around
132 requests/second, which is way behind.

[1] http://www.cherrypy.org/wiki/CherryPySpeed

Is CherryPy slower since the benchmark was done?

The tested environment is:

* Processor: Pentium M 1.7 Ghz
* RAM: 512GB
* Linux 2.6.12 (Ubuntu)
* Load testing tool: ab2 from Apache2
* CherryPy version: 2.2.0beta (SVN Rev. 956)

The code used is:

import cherrypy
class Root:
@cherrypy.expose
def index(self):
return "OK"
cherrypy.root = Root()
cherrypy.config.update({'global': {
'server.socket_port': 8000,
'server.log_to_screen': False,
'server.environment': 'production',
}})
cherrypy.server.start()

--
Gustavo Niemeyer
http://niemeyer.net

[Remi Delon]

Did you have a look at
http://groups.google.com/group/cherrypy-devel/browse_thread/thread/4d549b755fd0b9c5/9be191f78d96ad29 ?

Regards,

Remi.

[Gustavo Niemeyer]

Hey Remi!

> Did you have a look at [...]

Humm.. interesting. No, I wasn't aware that lowercase_api would
improve the performance meaningfully. CherryPy is now answering
up to 183 requests per second. I've also benchmarked behind
mod_python, and was able to get about 337 reqs/sec. Still behind
the 500 reqs/sec, but I can probably live with that.

I wonder if there's some kind of permanent analysis on those
numbers to warn developers if a change that affects performance
badly goes in. If there's no such test, might be worth thinking
about it.

Thanks for the pointer.

[Sylvain Hellegouarch]

Hi Gustavo,

There is no such test and as indeed it would be more than welcome in the
CherryPy world. If you have ideas go ahead :)

- Sylvain

Gustavo Niemeyer a écrit :

[pytrade]

So if you run cherrypy with mod_python you get nearly twice as many
requests per second? I knew it would be higher but not this much
higher.

[Gustavo Niemeyer]

> There is no such test and as indeed it would be more than welcome in
> the CherryPy world. If you have ideas go ahead :)

I belive it would be fairly simple to put ApacheBench to run
on every committed revision with a few different CherryPy
setups, like the mentioned script. It may either send daily
analysis to the development mailing list, or have some kind
of trigger that sends a warning message if the number of
requests per second drops behind a certain limit.

> >up to 183 requests per second. I've also benchmarked behind
> >mod_python, and was able to get about 337 reqs/sec. Still behind
> >the 500 reqs/sec, but I can probably live with that.

Unfortunately I was wrong. The true value is 134 res/sec, so I'm
still looking for ways to improve the speed.

As a random point of comparison, plain mod_python does 586 reqs/sec
on the same environment, and twisted web does 407 (yes, different
frameworks yadda yadda yadda).

[Gustavo Niemeyer]

> So if you run cherrypy with mod_python you get nearly twice as many
> requests per second? I knew it would be higher but not this much
> higher.

No, I was wrong.. I broke the code without noticing, so ApacheBench
was running against a broken server (and being very fast at it ;-).
The real value is actually worse than the internal server (about 130
reqs/sec in my environment, against ~180 with internal server).

Apologies.

[PA]

On Feb 07, 2006, at 19:40, Gustavo Niemeyer wrote:

> The real value is actually worse than the internal server (about 130
> reqs/sec in my environment, against ~180 with internal server).

As a random data point, here is some recent benchmarks I ran against
cherrypy (subversion as of January 21st) and others:

[Cherrypy]
% python -V
Python 2.4.2
% python tut01_helloworld.py
% ab -n 10000 http://localhost:8080/
Requests per second: 164.92 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost:8080/
Requests per second: 164.77 [#/sec] (mean)

[Webrick]
% ruby -v
ruby 1.8.4 (2005-12-24) [powerpc-darwin7.9.0]
% ruby webrick_compare.rb >& /dev/null
% ab -n 10000 http://localhost:4000/test
Requests per second: 37.90 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost:4000/test
Requests per second: 27.58 [#/sec] (mean)

[Mongrel]
% ruby -v
ruby 1.8.4 (2005-12-24) [powerpc-darwin7.9.0]
% ruby simpletest.rb
% ab -n 10000 http://localhost:3000/test
Requests per second: 660.20 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost:3000/test
Requests per second: 386.31 [#/sec] (mean)

[LuaWeb]
% lua -v
Lua 5.1 Copyright (C) 1994-2006 Lua.org, PUC-Rio
% lua Test.lua
% ab -n 10000 http://localhost:1080/hello
Requests per second: 948.32 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost:1080/hello
Requests per second: 927.04 [#/sec] (mean)

[httpd]
% httpd -v
Server version: Apache/1.3.33 (Darwin)
% ab -n 10000 http://localhost/test.txt
Requests per second: 1218.47 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost/test.txt
Requests per second: 1186.10 [#/sec] (mean)

[lighttpd]
% lighttpd -v
lighttpd-1.4.9 - a light and fast webserver
% ab -n 10000 http://localhost:8888/test.txt
Requests per second: 3652.30 [#/sec] (mean)
% ab -n 10000 -c 10 http://localhost:8888/test.txt
Called sick today (fdevent.c.170: aborted)

Cheers

--
PA, Onnay Equitursay
http://alt.textdrive.com/

[Sylvain Hellegouarch]

Interesting.

I can appreciate CherryPy being far behind lightttpd but I wonder why so
far away and what could be the main bottleneck.

- Sylvain

PA a écrit :

[Robert Brewer]

Sylvain Hellegouarch wrote:
> I can appreciate CherryPy being far behind lightttpd but I
> wonder why so far away and what could be the main bottleneck.

You know, none of you have to wonder. ;) I wrote a lovely profiler
module and integrated it into the test suite (test.py --profile) so that
tracking down such bottlenecks would be a couple of minutes' worth of
work. Here's the significant entries from my box (cp_0003.prof):

Tue Feb 07 11:32:51 2006
c:\python24\lib\site-packages\cherrypy\lib\profile\cp_0003.prof

439 function calls (435 primitive calls) in 0.007 CPU seconds

Ordered by: cumulative time

ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.007 0.007 profile:0(<bound method
Request._run of <cherrypy._cphttptools.Request object at 0x00BCA670>>)
1 0.000 0.000 0.007 0.007 _cphttptools.py:76(_run)
4 0.000 0.000 0.005 0.001
filters\__init__.py:97(applyFilters)
8/4 0.001 0.000 0.002 0.001
_cputil.py:57(get_special_attribute)
7 0.001 0.000 0.002 0.000 config.py:101(get)
8 0.001 0.000 0.001 0.000
_cputil.py:11(get_object_trail)
12 0.001 0.000 0.001 0.000
_cputil.py:309(lower_to_camel)
1 0.000 0.000 0.001 0.001
_cphttptools.py:124(processRequestLine)
1 0.000 0.000 0.001 0.001
_cphttptools.py:164(processHeaders)
1 0.000 0.000 0.001 0.001
cachefilter.py:111(before_main)
19 0.000 0.000 0.001 0.000
cherrypy\__init__.py:41(__getattr__)

That was down to .005 the last time I looked. The lower_to_camel call
has already been dealt with, so that just leaves filters or "something
else". In my limited experience with ab+cp, that "something else" is
usually either 1) using Python 2.3 (which is significantly slower than
Python 2.4 because its threadlocals are written in Python instead of C),
or 2) you haven't turned off sys.stdout; if your test logs to stdout,
that will take a significant amount of processing time. Turn off
log_to_screen.

Robert Brewer
System Architect
Amor Ministries
fuma...@amor.org

[Gustavo Niemeyer]

Hello Robert,

[...]

> else". In my limited experience with ab+cp, that "something else" is
> usually either 1) using Python 2.3 (which is significantly slower than
> Python 2.4 because its threadlocals are written in Python instead of C),
> or 2) you haven't turned off sys.stdout; if your test logs to stdout,
> that will take a significant amount of processing time. Turn off
> log_to_screen.

I'm running under Python 2.4, with log_to_screen disabled, and it's
still around 180 requests per second. I think the "something else" is
simply cherrypy getting a bit heavier in its request dispatching.

[Christian Wyglendowski]

Robert Brewer wrote:
> Sylvain Hellegouarch wrote:
>> I can appreciate CherryPy being far behind lightttpd but I
>> wonder why so far away and what could be the main bottleneck.
>
> You know, none of you have to wonder. ;) I wrote a lovely profiler
> module and integrated it into the test suite (test.py --profile) so that
> tracking down such bottlenecks would be a couple of minutes' worth of
> work. Here's the significant entries from my box (cp_0003.prof):

So what is the significance of each cp_*.prof file? Is one generated
for each test method in each test*.py file? I had a hard time
understanding what I was looking at for each file. Maybe I just need to
read up on python's profiler.

Christian

[Robert Brewer]

Christian Wyglendowski wrote:
> So what is the significance of each cp_*.prof file?
> Is one generated for each test method in each test*.py file?

One is generated for each HTTP request.

[Christian Wyglendowski]

Robert Brewer wrote:
> Christian Wyglendowski wrote:
>> So what is the significance of each cp_*.prof file?
>> Is one generated for each test method in each test*.py file?
>
> One is generated for each HTTP request.

Thanks. That makes sense.

Christian