What's the most scalable Django deployment out there currently?

[ldm999]

Is there any reason why Django couldn't be used to create a website
that gets Facebook-like traffic?

[Alex Gaynor]

On Wed, Mar 25, 2009 at 12:32 AM, ldm999 <malcol...@gmail.com> wrote:

Is there any reason why Django couldn't be used to create a website
that gets Facebook-like traffic?

Not really, at that point it's mostly scaling your DB(since web nodes are comparatively easy).  Django doesn't have a public API for mulitple databases so it could be harder, but there is both private APIs and external tools like PgPool.

However, what's the real question you're trying to answer?  100% of websites don't get traffic that approaches facebook.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero

[ldm616]

I guess I'm asking if there's any inherent reason to *not* use Django
for a website that you hope/anticipate will have many simultaneous
users and high levels of database reads/updates. And, related, what's
the most "extreme" Django deployment currently out there.

And on the database side, what database do folks recommend for the
more extreme apps. Can mysql go all the way or are there limitations
there too?

On Mar 24, 9:34 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:

[Adam V.]

Curse (http://www.curse.com/) manages to stay up on WoW patch day, so
that's a pretty good sign.

[Alex Gaynor]

On Wed, Mar 25, 2009 at 12:43 AM, Adam V. <fla...@gmail.com> wrote:

Curse (http://www.curse.com/) manages to stay up on WoW patch day, so
that's a pretty good sign.

Curse is no longer running on Django(it's ASP.net now I think), no idea if that was a technical decision or not.

[Graham Dumpleton]

On Mar 25, 3:45 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Wed, Mar 25, 2009 at 12:43 AM, Adam V. <fla...@gmail.com> wrote:
>
> > Curse (http://www.curse.com/) manages to stay up on WoW patch day, so
> > that's a pretty good sign.
>
> Curse is no longer running on Django(it's ASP.net now I think), no idea if
> that was a technical decision or not.

If it using ASP.net, maybe that is why none of their addresses respond
to me from where I am. :-)

Curse was using mod_python:

http://www.davidcramer.net/curse/44/what-powers-curse.html

David Cramer though has since switched to mod_wsgi and has expressed
the opinion that mod_wsgi works a lot better.

Important think though is not so much the hosting mechanism but how
well you optimise the performance of your application and database.

Graham

[Alex Gaynor]

David also did a good talk on scaling Django(based on his experiences at curse) at DjangoCon, the video is on youtube.

[Darren Mansell]

On Wed, Mar 25, 2009 at 5:09 AM, Alex Gaynor <alex....@gmail.com> wrote:

On Wed, Mar 25, 2009 at 1:06 AM, Graham Dumpleton <Graham.D...@gmail.com> wrote:

On Mar 25, 3:45 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:

> On Wed, Mar 25, 2009 at 12:43 AM, Adam V. <fla...@gmail.com> wrote:
>
> > Curse (http://www.curse.com/) manages to stay up on WoW patch day, so
> > that's a pretty good sign.
>

> Curse is no longer running on Django(it's ASP.net now I think), no idea if
> that was a technical decision or not.

If it using ASP.net, maybe that is why none of their addresses respond
to me from where I am. :-)

Curse was using mod_python:

 http://www.davidcramer.net/curse/44/what-powers-curse.html

David Cramer though has since switched to mod_wsgi and has expressed
the opinion that mod_wsgi works a lot better.

Important think though is not so much the hosting mechanism but how
well you optimise the performance of your application and database.

Graham

David also did a good talk on scaling Django(based on his experiences at curse) at DjangoCon, the video is on youtube.

Alex

As far as DBs go we are scaling MySQL quite hard. If you ensure you have the tools to support it you can use circular replication with Linux-HA to provide a single IP address that the app connects to. You can also scale out Apache with Linux-HA in this way. We've found that it's very hard to scale asp.Net/SQL platforms out but using Linux with HA it's very cheap and easy.

[Dave Snider]

We're using Django on giantbomb.com and comicvine.com . They are
pushing 25 mil or so monthly pageviews. I'm not up on the community
these days but I think we're one of the larger implementations. Most
of our team came from PHP/SMARTY backgrounds and I can't recommend
Django enough.

Dave



On Mar 24, 9:34 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:

[David Holstius]

We're migrating www.gigapan.org to Django (was: PHP/Smarty) on a page-
by-page basis. We have a high read/write ratio, so judicious caching
solves most of our scaling issues in the realm of dynamic web content.
Most of our stress comes from serving static imagery. So, let me
qualify that: we have Django running on some otherwise maxed-out
servers and it is performing extremely reliably.

[Oli Warner]

David, slightly off-topic, but you should see some impressive gains if you move your static handling off Apache onto something lighter (Cherokee/Lighttpd/nginx/etc) and reverse-proxy the dynamic stuff through to a localhost-bound Apache.

Hell, I use Cherokee for dynamic content too (via django's SCGI output). Uses a tiny amount of resources compared to the Apache+mod_python setup I was using.

[Graham Dumpleton]

On Mar 26, 12:41 am, Oli Warner <o...@thepcspy.com> wrote:
> David, slightly off-topic, but you should see some impressive gains if you
> move your static handling off Apache onto something lighter
> (Cherokee/Lighttpd/nginx/etc) and reverse-proxy the dynamic stuff through to
> a localhost-bound Apache.
>
> Hell, I use Cherokee for dynamic content too (via django's SCGI output).
> Uses a tiny amount of resources compared to the Apache+mod_python setup I
> was using.

A lot of the problems people have with mod_python and excessive memory
usage are because they don't change Apache default MPM settings,
especially if they use prefork MPM. See:

http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usage.html

It is rather obvious that if you leave Apache to be able to create
huge numbers of worker processes that you are then going to end up
with an equal number of Django instances and therefore a lot of memory
being used.

Graham

> On Wed, Mar 25, 2009 at 1:20 PM, David Holstius <david.holst...@gmail.com>wrote:
>
>
>
> > We're migratingwww.gigapan.orgto Django (was: PHP/Smarty) on a page-