At work, we are developing a commercial website based on Django. It's a fairly dynamic site (think social networking). I am doing the initial load testing to estimate the number of servers we will need for the production site. The production site will be load balanced using a pair of BigIP boxes.
When I stress test the dynamic part of the site, I am only getting about 300 requests per second on my test setup. This is using two Dell 1950's (one for web, one for mysql database). These are very powerful machines (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
I've done all the standard performance tuning I find when reading through various websites about django tuning. Is this the performance I should expect.
When I'm running the load test, the CPU on the web server gets completely buried (even with 8 cores). The mysql server doesn't seem loaded at all. Any suggestions on how to find the bottlenecks?
I'm running Ubuntu 7.10 (server, 64 bit version). Django from a very recent trunk.
> When I stress test the dynamic part of the site, I am only getting about
> 300 requests per second on my test setup. This is using two Dell 1950's
> (one for web, one for mysql database). These are very powerful machines
> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
- How are you running your Django app? Mod_python? FastCGI?
- What web server are you using?
- What's the nature of the dynamic Django request you are measuring?
How many DB queries does it make? Is DEBUG mode off?
- What kind of numbers do you get when you serve a simple and small
HTML file statically from your Web server without Django?
- What kind of numbers do you get when the Django view you are
benchmarking goes directly to a simple template (i.e. no DB queries)?
- Are you using memcache?
> > When I stress test the dynamic part of the site, I am only getting about > > 300 requests per second on my test setup. This is using two Dell 1950's > > (one for web, one for mysql database). These are very powerful machines > > (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
> - How are you running your Django app? Mod_python? FastCGI? > - What web server are you using? > - What's the nature of the dynamic Django request you are measuring? > How many DB queries does it make? Is DEBUG mode off? > - What kind of numbers do you get when you serve a simple and small > HTML file statically from your Web server without Django? > - What kind of numbers do you get when the Django view you are > benchmarking goes directly to a simple template (i.e. no DB queries)? > - Are you using memcache?
I've found the largest memory hog to be the native way related tables
are setup.
Check class definitions with related tables and edit as such:
class ...(models.Mode):
relatedtable = models.ForeignKey(RelatedTable, core=True,
raw_id_admin=True)
The raw_id_admin=True prevents django from pulling up all related
records when the object is loaded etc. While I would have expected
this to only affect the admin side, it had a huge impact on the public
side of our site. Memory consumption went from +500mb to 20-30mb.
Joe
On Dec 11, 12:53 pm, Richard Coleman <rcole...@criticalmagic.com>
wrote:
> At work, we are developing a commercial website based on Django. It's a
> fairly dynamic site (think social networking). I am doing the initial
> load testing to estimate the number of servers we will need for the
> production site. The production site will be load balanced using a pair
> of BigIP boxes.
> When I stress test the dynamic part of the site, I am only getting about
> 300 requests per second on my test setup. This is using two Dell 1950's
> (one for web, one for mysql database). These are very powerful machines
> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
> I've done all the standard performance tuning I find when reading
> through various websites about django tuning. Is this the performance I
> should expect.
> When I'm running the load test, the CPU on the web server gets
> completely buried (even with 8 cores). The mysql server doesn't seem
> loaded at all. Any suggestions on how to find the bottlenecks?
> I'm running Ubuntu 7.10 (server, 64 bit version). Django from a very
> recent trunk.
On Dec 11, 2007 12:18 PM, Joe <joerad...@gmail.com> wrote:
> I've found the largest memory hog to be the native way related tables > are setup.
> Check class definitions with related tables and edit as such:
> class ...(models.Mode): > relatedtable = models.ForeignKey(RelatedTable, core=True, > raw_id_admin=True)
> The raw_id_admin=True prevents django from pulling up all related > records when the object is loaded etc. While I would have expected > this to only affect the admin side, it had a huge impact on the public > side of our site. Memory consumption went from +500mb to 20-30mb.
You must be using manipulators in oldforms. Try newforms.
Rajesh Dhawan wrote: >> When I stress test the dynamic part of the site, I am only getting about >> 300 requests per second on my test setup. This is using two Dell 1950's >> (one for web, one for mysql database). These are very powerful machines >> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
> - How are you running your Django app? Mod_python? FastCGI? > - What web server are you using? > - What's the nature of the dynamic Django request you are measuring? > How many DB queries does it make? Is DEBUG mode off? > - What kind of numbers do you get when you serve a simple and small > HTML file statically from your Web server without Django? > - What kind of numbers do you get when the Django view you are > benchmarking goes directly to a simple template (i.e. no DB queries)? > - Are you using memcache?
> -Rajesh D
1. mod_python 2. apache 2.2.4 3. I'm using funkload and ab to measure the requests per second of one of the base pages within the dynamic part of the website 4. When I hit a static page in the same way (using ab), I get 6500 requests per second. 5. This is without memcached, or any other caching.
>>> When I stress test the dynamic part of the site, I am only getting about >>> 300 requests per second on my test setup. This is using two Dell 1950's >>> (one for web, one for mysql database). These are very powerful machines >>> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
>> - How are you running your Django app? Mod_python? FastCGI? >> - What web server are you using? >> - What's the nature of the dynamic Django request you are measuring? >> How many DB queries does it make? Is DEBUG mode off? >> - What kind of numbers do you get when you serve a simple and small >> HTML file statically from your Web server without Django? >> - What kind of numbers do you get when the Django view you are >> benchmarking goes directly to a simple template (i.e. no DB queries)? >> - Are you using memcache?
>> -Rajesh D
> 1. mod_python > 2. apache 2.2.4 > 3. I'm using funkload and ab to measure the requests per second of one > of the base pages within the dynamic part of the website > 4. When I hit a static page in the same way (using ab), I get 6500 > requests per second. > 5. This is without memcached, or any other caching.
I forgot to mention these:
6. django debug turned off 7. mod_python debug turn off 8. django template debugging turned off 9. apache maxclient cranked up to 1000 (although it never gets close to that many processes).
Are you serving static content from the same apache instance? Also,
what kind of network connectivity do you have between your web and
mysql servers? It sounds like apache might need some tuning in terms
of thread parameters. Have you enabled caching yet? Turn on the
cache framework site-wide and set your expiration period to 1 minute
or something like that. You can go back and only enable it for views
that you want cached later.
On Dec 11, 12:53 pm, Richard Coleman <rcole...@criticalmagic.com>
wrote:
> At work, we are developing a commercial website based on Django. It's a
> fairly dynamic site (think social networking). I am doing the initial
> load testing to estimate the number of servers we will need for the
> production site. The production site will be load balanced using a pair
> of BigIP boxes.
> When I stress test the dynamic part of the site, I am only getting about
> 300 requests per second on my test setup. This is using two Dell 1950's
> (one for web, one for mysql database). These are very powerful machines
> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.)
> I've done all the standard performance tuning I find when reading
> through various websites about django tuning. Is this the performance I
> should expect.
> When I'm running the load test, the CPU on the web server gets
> completely buried (even with 8 cores). The mysql server doesn't seem
> loaded at all. Any suggestions on how to find the bottlenecks?
> I'm running Ubuntu 7.10 (server, 64 bit version). Django from a very
> recent trunk.
Firebug: http://www.getfirebug.com/ will show you how long each component of the page takes from request to response and shows you what components are returned in parallel (among other really cool features). This can help identify your bottleneck. For example, if your images are taking longer than expected while the page itself returns quickly then you may want to serve images with a dedicated web server, like lighttpd.
Yahoo's 'yslow' tool gives your page a 'grade' and offers advice on how to improve your grade. It also has other handy bits (JSLint, Empty vs. Primed cache comparisons). http://developer.yahoo.com/yslow/
> what pages are you requesting and have you profiled them to understand > what's taking long?
> -joe
> On Dec 11, 2007 10:08 AM, Rajesh Dhawan <rajesh.dha...@gmail.com> wrote: >> Hi Richard,
>>> When I stress test the dynamic part of the site, I am only getting about >>> 300 requests per second on my test setup. This is using two Dell 1950's >>> (one for web, one for mysql database). These are very powerful machines >>> (3.0ghz Xeons, 8 cores each, 16 gig of ram, 15k SAS drives, etc.) >> - How are you running your Django app? Mod_python? FastCGI? >> - What web server are you using? >> - What's the nature of the dynamic Django request you are measuring? >> How many DB queries does it make? Is DEBUG mode off? >> - What kind of numbers do you get when you serve a simple and small >> HTML file statically from your Web server without Django? >> - What kind of numbers do you get when the Django view you are >> benchmarking goes directly to a simple template (i.e. no DB queries)? >> - Are you using memcache?
Brian Morton wrote: > Are you serving static content from the same apache instance? Also, > what kind of network connectivity do you have between your web and > mysql servers? It sounds like apache might need some tuning in terms > of thread parameters. Have you enabled caching yet? Turn on the > cache framework site-wide and set your expiration period to 1 minute > or something like that. You can go back and only enable it for views > that you want cached later.
1. Static content and dynamic are on the same server (that will change on production). But they are on different apache virtual. Mod_python is turned off for the static stuff. Hitting only a static page is very fast (6500 requests per second). Hitting the dynamic side is slow (300 requests per second).
2. It's gigE between the web server and mysql server, on a dedicated switch. The database is working pretty hard (7000 selects per second), but doesn't seem to be the bottleneck. The webserver is hammered (typing any command takes a long time). 3. I'm using prefork MPM on apache with maxclients set to 1000.
We are starting to experiment with caching, but I want to improve the raw performance of the site as well.
