Performance tuning tips?

[Milton Waddams]

I'm having some issues with performance and wondering if there's
anything I can do to speed things up.

I'm using the db cache backend (memcached doesn't seem to want to work
for me and my memory is running low enough as it is).

The page I'm trying to speed up has a quite a bit of db interaction
(it's a forum index page), it isn't overly popular (around 500
uniques/day, avg 7 page views/user), though if I can avoid the major
load it would be good.

It seems that the cache is working (dynamic components aren't dynamic
for the cache time and there is corresponding data in the cache
table), though the speed gain isn't really anything much to write home
about, response times have gone from ~5 to ~3 seconds.

I haven't looked at the db cache internals though I would have
suspected that it was a matter of 2 db queries to determine/return the
cached page. Other pages I have with 5 db queries have a response time
of around 1 second.

I'm using Apache/2.0.54 (Debian GNU/Linux) mod_python/3.1.3
Python/2.3.5, django is the latest before the last set of backwards
incompatible changes. Server is P4 2.8ghz with 1 gig of ram. If it's
important the page size is ~24kb.

I've included the output from openload below, is that an Apache cache
kicking in where it drops to sub 0.1 second responses?

Any ideas that could help me would be greatly appreciated.

MaTps 0.34, Tps 0.34, Resp Time 2.965, Err 0%, Count 1
MaTps 0.44, Tps 1.35, Resp Time 3.251, Err 0%, Count 6
MaTps 0.53, Tps 1.37, Resp Time 3.971, Err 0%, Count 11
MaTps 0.82, Tps 3.46, Resp Time 3.819, Err 0%, Count 15
MaTps 0.78, Tps 0.41, Resp Time 3.428, Err 0%, Count 16
MaTps 0.99, Tps 2.80, Resp Time 3.807, Err 0%, Count 19
MaTps 1.04, Tps 1.53, Resp Time 1.899, Err 0%, Count 23
MaTps 2.67, Tps 17.34, Resp Time 0.436, Err 0%, Count 41
MaTps 2.80, Tps 3.98, Resp Time 0.919, Err 0%, Count 54
MaTps 2.78, Tps 2.65, Resp Time 4.230, Err 0%, Count 57
MaTps 4.04, Tps 15.30, Resp Time 0.176, Err 0%, Count 107
MaTps 13.70, Tps 100.70, Resp Time 0.085, Err 0%, Count 208
MaTps 24.92, Tps 125.87, Resp Time 0.101, Err 0%, Count 335
MaTps 34.55, Tps 121.27, Resp Time 0.032, Err 0%, Count 457
..........

here's another that I ran to copy the before/after loads, server load
went from "load average: 0.18, 0.25, 0.40" to "load average: 3.49,
1.55, 0.86"
MaTps 0.03, Tps 0.03, Resp Time 33.326, Err 0%, Count 1
MaTps 0.38, Tps 3.57, Resp Time 13.769, Err 0%, Count 6
MaTps 0.43, Tps 0.89, Resp Time 19.468, Err 0%, Count 8
MaTps 1.32, Tps 9.28, Resp Time 2.513, Err 0%, Count 25
MaTps 8.23, Tps 70.42, Resp Time 0.066, Err 0%, Count 105
MaTps 14.46, Tps 70.58, Resp Time 0.088, Err 0%, Count 176
MaTps 21.61, Tps 85.97, Resp Time 0.020, Err 0%, Count 263
MaTps 28.15, Tps 87.00, Resp Time 0.079, Err 0%, Count 350
MaTps 32.23, Tps 68.97, Resp Time 0.029, Err 0%, Count 420
MaTps 38.34, Tps 93.25, Resp Time 0.094, Err 0%, Count 514
MaTps 40.19, Tps 56.84, Resp Time 0.034, Err 0%, Count 595
MaTps 46.80, Tps 106.36, Resp Time 0.041, Err 0%, Count 712
MaTps 50.22, Tps 81.00, Resp Time 0.237, Err 0%, Count 793
MaTps 55.88, Tps 106.79, Resp Time 0.036, Err 0%, Count 900

[Eugene Lazutkin]

Just out of curiosity: what db do you use for cache?

Thanks,

Eugene

"Milton Waddams" <ilikep...@gmail.com> wrote in message
news:29ae23f90511211801g738...@mail.gmail.com...

[...cut...]

I'm using the db cache backend (memcached doesn't seem to want to work
for me and my memory is running low enough as it is).

[...cut...]

[Milton Waddams]

DOH!

I thought I had listed all the important server side stuff

MYSQL 4.0.24_Debian-10sarge1-log

[Eugene Lazutkin]

Could you just for kicks switch to "file:" cache and run tests again? I
wonder if there is a significant difference in response times. It may help
us to isolate the problem.

One more question: did you apply MySQL backend patch
(http://code.djangoproject.com/ticket/463)? I don't think it should make a
difference under mod_python, but checking just in case.

Thanks,

Eugene


"Milton Waddams" <ilikep...@gmail.com> wrote in message

news:29ae23f90511211942l78...@mail.gmail.com...

[Milton Waddams]

sorry for the delay, got buried under a ton of work :(

I haven't applied any patches to it (well except a couple of filters
that I put into the default filters file).

I switched to file based cache, results are below, sadly no great improvement.

It may help to know that I'm running openload with the default number
of clients which is 5.

load averages were

before: 0.64, 1.16, 1.07
after: 2.60, 1.57, 1.21

openload output:

MaTps 0.36, Tps 0.36, Resp Time 2.742, Err 0%, Count 1
MaTps 0.50, Tps 1.74, Resp Time 3.070, Err 0%, Count 6
MaTps 0.63, Tps 1.78, Resp Time 2.927, Err 0%, Count 11
MaTps 0.95, Tps 3.83, Resp Time 1.066, Err 0%, Count 25
MaTps 3.58, Tps 27.26, Resp Time 0.477, Err 0%, Count 57
MaTps 21.59, Tps 183.71, Resp Time 0.005, Err 0%, Count 242
MaTps 27.46, Tps 80.27, Resp Time 0.012, Err 0%, Count 351
MaTps 38.32, Tps 136.00, Resp Time 0.007, Err 0%, Count 487
MaTps 46.12, Tps 116.36, Resp Time 0.049, Err 0%, Count 620
MaTps 61.69, Tps 201.80, Resp Time 0.098, Err 0%, Count 822
MaTps 77.30, Tps 217.78, Resp Time 0.018, Err 0%, Count 1040
MaTps 86.47, Tps 169.00, Resp Time 0.023, Err 0%, Count 1209

[Eugene Lazutkin]

For comparison I ran openload against my blog page
(http://lazutkin.com/blog/). It is 38012 bytes at the moment. I don't know
how many db transactions required to build it, but with cache it's a moot
point anyway. Incidentally I use "file:" cache. The site is hosted with
DreamHost --- cheap community hosting company.

Over Internet using my home connection I got:

Total TPS: 12.54
Avg. Response time: 0.781 sec.
Max Response time: 1.641 sec
Total Requests: 503
Total Errors: 0

With average ping ~40ms, it correspondes to ~400kbps, which looks about
right order of magnitude.

Judging by your times you probably used "localhost" for load testing, or you
had some intermediate caching, or you have really fat connection to get 24k
in 20ms. ;-)

I don't know what the exact problem is, but I would suggest to check basic
stuff first: check footprint (maybe swapping is an issue); check database
speed (e.g., using Django's database API with some rudimentary timing
program); if you run db and apache/django on the same server, try to
separate them (sometimes there is an interference between programs); to get
realistic test results, try to run openload from separate computer (or
several copies of openload from separate computers for even more realism).
If everything works as expected, you probably found a bug in cache subsystem
and we need to find a way to reproduce it.

BTW, could you cut'n'paste your MIDDLEWARE_CLASSES? The order is important.

Thanks,

Eugene

[Milton Waddams]

Yes I am running openload on localhost, I don't trust my DSL enough to
get accurate results :(

my MIDDLEWARE_CLASSES:

"django.middleware.common.CommonMiddleware",
"django.middleware.doc.XViewMiddleware",
"django.middleware.sessions.SessionMiddleware",

I'm caching on a view by view basis (hope that's the correct
terminology) using this code.

index = cache_page(index, 180)

After looking at those middleware classes I removed the session
middleware and it has dropped response times to a very respectable:

Total TPS: 144.27
Avg. Response time: 0.034 sec.
Max Response time: 0.675 sec
Total Requests: 1951
Total Errors: 0

So I believe my problem was the session middleware. At present I don't
require the session middleware (I will move the admin). However in the
near future the forum will change to requiring user authentication,
any thoughts about the best way for me to enable caching with the
sessions enabled? If need be what would help my problems is if the
session caching time can't be improved then if sessions are enabled
only for certain views? As an example, in one of my sites I have no
session requirements, except when generating a captcha and then
verifying the captcha when the comment is submitted.

Thanks for your help!

[Eugene Lazutkin]

Interesting that the session middleware was the culprit. BTW, why did you
put it at the end of your middleware list?

The default order is:

"django.middleware.sessions.SessionMiddleware",
"django.middleware.common.CommonMiddleware",
"django.middleware.doc.XViewMiddleware",

(see
http://www.djangoproject.com/documentation/settings/#middleware-classes).

Thanks,

Eugene

[Milton Waddams]

no idea why I put it there, I tried moving it to the top, the response
times went back to around 3 seconds.