Re: Performance and scalability

[Simon Willison]

On 29 Sep 2005, at 15:06, Tau wrote:

> What you, the authors of django, can provide on the subject of
> django's
> performance and scalability. I find the framework architecture to be
> excellent but, literally speaking, what if I migrate my php sites to
> django. Will hardware upgrade be inevitable?

In theory, Django with mod_python should be faster than the
equivalent in PHP because PHP has to read and interpret the scripts
every time, while mod_python loads them once from disk and keeps the
compiled code in memory. If you were using one of the PHP
accelerators however PHP should have the same thing going for it.

How much traffic are you talking about? www.kusports.com uses Django
and gets hit with some pretty heavy traffic at times.
chacagocrime.org has weathered a slashdotting or two thanks to
Django's caching framework.

[Andreas]

Simon Willison wrote:
> How much traffic are you talking about? www.kusports.com uses Django
> and gets hit with some pretty heavy traffic at times.

Just out of curiosity: Is there a reason for KUSports using a PHP
message board? Is Django not appropriate for this kind of web
application or was it just easier to use an existing solution instead
of coding a message board from scratch?

Andras

[Adrian Holovaty]

On 9/29/05, Andreas <and...@mindpicnic.com> wrote:
> Just out of curiosity: Is there a reason for KUSports using a PHP
> message board? Is Django not appropriate for this kind of web
> application or was it just easier to use an existing solution instead
> of coding a message board from scratch?

Heh, KUsports.com has used that PHP message board for *years*, and we
just haven't had the time (or inclination, really) to convert it to
Django. There's no technical reason why it couldn't use Django --
indeed, Django is very good at forum-style stuff. Message boards on
the other World Online sites are indeed using Django:
http://www.lawrence.com/forums/ and http://www2.lawrence.com/forums/

Adrian

--
Adrian Holovaty
holovaty.com | djangoproject.com | chicagocrime.org

[Jacob Kaplan-Moss]

On Sep 29, 2005, at 10:31 AM, Adrian Holovaty wrote:
> Heh, KUsports.com has used that PHP message board for *years*, and we
> just haven't had the time (or inclination, really) to convert it to
> Django. There's no technical reason why it couldn't use Django --
> indeed, Django is very good at forum-style stuff. Message boards on
> the other World Online sites are indeed using Django:
> http://www.lawrence.com/forums/ and http://www2.lawrence.com/forums/

And to get back to the performance question: that board nearly
brings down the server when we get our post-game rushes of about 20k
hits in an hour, but when a Django page gets farked and gets 100k
hits in an hour I barely notice. Seriously - we got farked a few
weeks ago and it wasn't until someone looked at our server stats the
next day that I knew about it.

Jacob

[Jeremy Dunck]

On 9/29/05, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
>
> On Sep 29, 2005, at 10:31 AM, Adrian Holovaty wrote:
> > Heh, KUsports.com has used that PHP message board for *years*, and we

>>...Message boards on

> > the other World Online sites are indeed using Django:
> > http://www.lawrence.com/forums/ and http://www2.lawrence.com/forums/
>
> And to get back to the performance question: that board nearly
> brings down the server when we get our post-game rushes of about 20k
> hits in an hour, but when a Django page gets farked and gets 100k
> hits in an hour I barely notice.

...

For clarity, which is the board that nearly brings you down?

[Jacob Kaplan-Moss]

On Sep 29, 2005, at 1:13 PM, Jeremy Dunck wrote:
> For clarity, which is the board that nearly brings you down?

UBB -- http://boards.kusports.com/ is the site.

It's on a seperate server because because UBB is an insecure,
inefficient, bug ridden piece of shit.

Jacob

[Adrian Holovaty]

On 9/29/05, Jeremy Dunck <jdu...@gmail.com> wrote:
> For clarity, which is the board that nearly brings you down?

The one on KUsports.com, which is powered by UBB (PHP).

[Jeremy Dunck]

On 9/29/05, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:

> UBB -- http://boards.kusports.com/ is the site.
>
> It's on a seperate server because because UBB is an insecure,
> inefficient, bug ridden piece of shit.

Don't hold back now. ;-) It was mostly a question intended for the
archive-searcher, since I knew what you meant.

[Jimmie Houchin]

Can you tell us what kind of hardware is supporting that 100k?

Thanks.

Jimmie

[Jeremy Dunck]

On 10/3/05, Jimmie Houchin <jlho...@gmail.com> wrote:
> Jacob Kaplan-Moss wrote:
...

> > an hour, but when a Django page gets farked and gets 100k hits in an
> > hour I barely notice. Seriously - we got farked a few weeks ago and it
> > wasn't until someone looked at our server stats the next day that I
> > knew about it.
>
> Can you tell us what kind of hardware is supporting that 100k?

From a different thread, Jacob answered:

>On Sep 28, 2005, at 3:38 PM, Jeremy Dunck wrote:
>> On 9/28/05, Jacob Kaplan-Moss <ja...@jacobian.org> wrote:
>>> ...In case you hadn't noticed, Django has a pretty large memory
>>> footprint; between Apache, mod_python, Python, the database drivers,
>>> etc., Apache server processes tend to weigh about 10M each (at least
>>> on my servers).
>>
>> I hadn't noticed. Do you just load up your web servers with memory,
>> or do you (allow Apache to) throttle concurrent requests?
>
>Yes, and yes. Our current web servers have 2G of RAM, and our new
>ones being ordered will have at least 4G. My philosophy has always
>been that hardware is cheaper than programming time: 2G of RAM costs
>about $100 these days, which corresponds to something around 2 hours
>of programming time.
>
>> Does anyone have numbers on footprints for different servers? Any
>> pointers to comparisons of request/process/thread models between
>> various servers which support WSGI or mod_python?
>>
>> (If my tone sounds shrill, I'm just curious, not upset.)
>
>I don't have any hard numbers, but my impressions have been that
>unless you need all the amazing stuff that Apache will do for you,
>lighttpd+WSGI serves Django faster with a smaller memory footprint.
>We will, however, be launching a new website in a month or so which
>will most likely our first powered by lighty so hopefully at that
>point I'll have some numbers to share; I'll certainly keep the list
>posted.

[Jacob Kaplan-Moss]

On Oct 3, 2005, at 11:14 PM, Jimmie Houchin wrote:

> Can you tell us what kind of hardware is supporting that 100k?
>

Our main production platform is three servers: a database server, a
web server, and a media server. They're all Dual Xeons with 2G of
RAM. We've got a secondary server setup that's a single dual Xeon
server that we serve a bunch of miscellaneous smaller sites from.

In general, Django sites are happiest with lots of RAM. If your
database server has enough RAM it can cache nearly the entire
database in memory, which makes things much happier. Similarly, if
Apache doesn't have to swap, it's much faster. I've seen very little
difference in performance between different processors -- although I
wish we'd know about the performance issues with Postgres on Xeons
before we bought new hardware -- and other factors like disk speed
and L1 cache size also aren't nearly as important to Django
environments as RAM is.

Jacob (who loves to talk about hardware :)

[Jimmie Houchin]

Thanks for the reply. (and also to Jeremy)

That's good news. I've only got one server to start and its a dual
Opteron with 4gb ram and 6 disk SATA RAID. So I feel reasonable about
that. As things develop I can add servers and spread the responsibilities.

I was hoping for a little more clarification. I was on a mailing list
for a Java web app server several years ago. They were touting the
performance. One of their "spotlight" websites was said to get 1 million
hits a day. They were able to handle that load with only 7 Sun
enterprise servers and 2 Sun workstations. And they were quite happy.
Ugh! I unsubscribed immediately. (I didn't like Java anyway. :)

What you describe is an affordable and capable setup.

Thanks again.

Jimmie