High CPU load for simple django view

[litterbug]

Hi -

I'm load testing a django 1.21/apache (prefork)/mod_wsgi (daemon)
configuration on an AWS small instance (ubuntu 10.04) with apache
bench, and seeing extremely high CPU load (using uptime and vmstat) at
low concurrent requests. Note that I'm using a trivial out-of-box
django project/app with a simple "hello world" view (no DBs, etc).

CPU is at 100% even with an apache bench concurrency value of 2. I'm
running apache bench from a different AWS instance in the same region/
zone. Ideas on what's the problem, or how I should continue to debug
this?

I'm running mod_wsgi version 2.8-2ubuntu1, since that's what apt-get
gave me, but I can manually compile if recommended.

I believe my mod_wsgi config is at fault, since CPU looks fine (almost
idle) when running against a django dev server. Also, CPU looks fine
running against a static file served by apache (ie. no django).

Many more details (including config files) in my serverfault post at
http://serverfault.com/questions/182672/high-cpu-load-on-django-apache-mod-wsgi-site
if useful.

Of note, I'm seeing this in my apache logs:
[Sun Sep 19 18:04:58 2010] [error] Exception KeyError:
KeyError(-1218693376,) in <module 'threading' from '/usr/lib/python2.6/
threading.pyc'> ignored

Thanks for any help!
Brian

[Graham Dumpleton]

On 21 September 2010 10:35, litterbug <briang...@gmail.com> wrote:
> Hi -
>
> I'm load testing a django 1.21/apache (prefork)/mod_wsgi (daemon)
> configuration on an AWS small instance (ubuntu 10.04) with apache
> bench, and seeing extremely high CPU load (using uptime and vmstat) at
> low concurrent requests. Note that I'm using a trivial out-of-box
> django project/app with a simple "hello world" view (no DBs, etc).
>
> CPU is at 100% even with an apache bench concurrency value of 2. I'm
> running apache bench from a different AWS instance in the same region/
> zone. Ideas on what's the problem, or how I should continue to debug
> this?
>
> I'm running mod_wsgi version 2.8-2ubuntu1, since that's what apt-get
> gave me, but I can manually compile if recommended.
>
> I believe my mod_wsgi config is at fault, since CPU looks fine (almost
> idle) when running against a django dev server. Also, CPU looks fine
> running against a static file served by apache (ie. no django).
>
> Many more details (including config files) in my serverfault post at
> http://serverfault.com/questions/182672/high-cpu-load-on-django-apache-mod-wsgi-site
> if useful.

Benchmark it with a WSGI hello world script and see whether that
behaves normally.

Your MPM settings are bit broken as well. You have:

StartServers 2
MinSpareServers 2
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 3000
ServerLimit 8
Keepalive off
HostnameLookups Off

You have said MaxClients should be 50, yet ServerLimit of 8 means that
MaxClients will actually be 8. So, not even enough capacity to proxy
as many as 10 requests to use all daemon processes at once.

If your Django application is thread safe, would also recommend not
using multiple single threaded processes, but a single multithreaded
process instead.

By using multiple single threaded processes, you are likely seeing the
startup cost of loading Django and initialising each instance when you
are doing the testing.

Suggest at least drop number of daemon processes down to 3. Even that
will likely still manage to handle adequate requests and you avoid
start up costs on extra processes you do not need.

> Of note, I'm seeing this in my apache logs:
> [Sun Sep 19 18:04:58 2010] [error] Exception KeyError:
> KeyError(-1218693376,) in <module 'threading' from '/usr/lib/python2.6/
> threading.pyc'> ignored

You can ignore it as warning only. Eliminated in mod_wsgi 3.X.

Graham

[Robert Coup]

On Tue, Sep 21, 2010 at 12:35 PM, litterbug <briang...@gmail.com> wrote:

I'm load testing a django 1.21/apache (prefork)/mod_wsgi (daemon)
configuration on an AWS small instance (ubuntu 10.04) with apache
bench, and seeing extremely high CPU load (using uptime and vmstat) at
low concurrent requests. Note that I'm using a trivial out-of-box
django project/app with a simple "hello world" view (no DBs, etc).

CPU is at 100% even with an apache bench concurrency value of 2. I'm
running apache bench from a different AWS instance in the same region/
zone. Ideas on what's the problem, or how I should continue to debug
this?

There's a bug with lucid under EC2 where it can report high loads which aren't actually there. See https://bugs.launchpad.net/ubuntu-on-ec2/+bug/574910 - I'm not sure if a fixed kernel has actually been released, maybe have a look through the ec2ubuntu group (http://groups.google.com/group/ec2ubuntu) archives over the last couple of weeks for posts by Scott Moser from Canonical. 

Rob :)

[litterbug]

I appreciate the feedback. I've tried the hello world wsgi program,
with similar results:

Using an apache bench concurrency value of 10 (ab -c 10 -n 10000
"my_url"), I'm still seeing load CPU loads (with top and uptime) above
3 and 0% idle time (with MaxClients = 3, and processes=3 threads=1)

I've established that as expected, setting MaxClients = 10, and
processes=10 threads=1 causes CPU Load to skyrocket, up to 9. I'm
running 10000 requests, so I'm not just seeing the initial startup CPU
spike.

Questions:

1)Are these simply the result I should expect with an AWS small
instance (1.7 GB memory, 1 virtual core which according to Amazon is
1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor) ?

2)Am I banging on my server too hard? I know it's all about expected
traffic, but is my apache bench test an unrealistic baseline?

3)I'm using prefork MPM since apache is also running php, which
doesn't work with worker MPM. With prefork MPM, can I still run
multithreaded wsgi (ie processes=1 threads=10)? Uncertain if they're
independent. In any case, I'm seeing similar results when I try.

Other ideas? I'm going to clone my AWS instance in the odd chance
it's a hardware issue, and also try a large instance to see relative
performance.

Thanks,
Brian

On Sep 20, 6:18 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

> On 21 September 2010 10:35, litterbug <brianggro...@gmail.com> wrote:
>
>
>
> > Hi -
>
> > I'm load testing a django 1.21/apache (prefork)/mod_wsgi (daemon)
> > configuration on an AWS small instance (ubuntu 10.04) with apache
> > bench, and seeing extremely high CPU load (using uptime and vmstat) at
> > low concurrent requests. Note that I'm using a trivial out-of-box
> > django project/app with a simple "hello world" view (no DBs, etc).
>
> > CPU is at 100% even with an apache bench concurrency value of 2. I'm
> > running apache bench from a different AWS instance in the same region/
> > zone. Ideas on what's the problem, or how I should continue to debug
> > this?
>
> > I'm running mod_wsgi version 2.8-2ubuntu1, since that's what apt-get
> > gave me, but I can manually compile if recommended.
>
> > I believe my mod_wsgi config is at fault, since CPU looks fine (almost
> > idle) when running against a django dev server. Also, CPU looks fine
> > running against a static file served by apache (ie. no django).
>
> > Many more details (including config files) in my serverfault post at

> >http://serverfault.com/questions/182672/high-cpu-load-on-django-apach...

[sste...@gmail.com]

On Sep 21, 2010, at 5:04 PM, litterbug wrote:

> I appreciate the feedback. I've tried the hello world wsgi program,
> with similar results:
>
> Using an apache bench concurrency value of 10 (ab -c 10 -n 10000
> "my_url"), I'm still seeing load CPU loads (with top and uptime) above
> 3 and 0% idle time (with MaxClients = 3, and processes=3 threads=1)
>
> I've established that as expected, setting MaxClients = 10, and
> processes=10 threads=1 causes CPU Load to skyrocket, up to 9. I'm
> running 10000 requests, so I'm not just seeing the initial startup CPU
> spike.
>
> Questions:
>
> 1)Are these simply the result I should expect with an AWS small
> instance (1.7 GB memory, 1 virtual core which according to Amazon is
> 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor) ?
>
> 2)Am I banging on my server too hard? I know it's all about expected
> traffic, but is my apache bench test an unrealistic baseline?
>
> 3)I'm using prefork MPM since apache is also running php, which
> doesn't work with worker MPM. With prefork MPM, can I still run
> multithreaded wsgi (ie processes=1 threads=10)? Uncertain if they're
> independent. In any case, I'm seeing similar results when I try.

You could run PHP with Nginx like this: http://wiki.nginx.org/NginxFcgiExample, then you're not stuck with prefork.

S

[litterbug]

The issue was that I had inadvertently installed the package apache2-
mpm-itk instead of apache2-mpm-prefork. apache2-mpm-itk is derived
from apache2-mpm-prefork, but for some reason, didn't perform well
when used with mod_wsgi.

I'm still curious if, when using prefork MPM, I can still run

multithreaded wsgi (ie processes=1 threads=10)?

Thanks!
Brian

[Carl Nobile]

Yes you can, mod_wsgi is what creates the daemons and threads not
apache in this case.

~Carl

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>

--
-------------------------------------------------------------------------------
Carl J. Nobile (Software Engineer)
carl....@gmail.com
-------------------------------------------------------------------------------