nasa site on django

[Vance Dubberly]

Moved http://opensource.arc.nasa.gov to django a couple of weeks ago.  Thought ya'll might want to know that.

Also thought you might want to know the site was running on Tomcat/Mysql and the performance difference is mind blowing.   Unfortunately the actual server specs and benchmarks can't be released without going through a nasty ball of red tape but let's just say django is ALOT more resource intensive and ALOT slower. 

On the plus side it is faster than a similar PHP app ( with APC ) and faster than serving static files.

--
To pretend, I actually do the thing: I have therefore only pretended to pretend.
  - Jacques Derrida

[KpoH]

I don't understand. Static files served by django?

--
Artiom Diomin, Development Dep, "Comunicatii Libere" S.R.L.
http://www.asterisksupport.ru
http://www.asterisk-support.com

[Kai Kuehne]

Hi,

> Vance Dubberly wrote:
> > On the plus side it is faster than a similar PHP app ( with APC ) and
> > faster than serving static files.

Afaik, dynamic things _can't be_ faster than static things.

Kai

[Jeremy Dunck]

On 6/27/07, Vance Dubberly <altje...@gmail.com> wrote:
...

> ball of red tape but let's just say django is ALOT more resource intensive
> and ALOT slower.

Have you done any profiling? I haven't compared such a port, but I'd
be surprised if performance is significantly (i.e. one order)
different without some optimization being missed.

If you have, can you point to any parts of Django that we could improve?

As presented, there's not much actionable. "Java is faster than
Python" is not news, but I doubt the language is the problem here.

Thanks!

[Jacob Kaplan-Moss]

On 6/27/07, Vance Dubberly <altje...@gmail.com> wrote:

> Moved http://opensource.arc.nasa.gov to django a couple of weeks ago.
> Thought ya'll might want to know that.

Congrats :)

> Also thought you might want to know the site was running on Tomcat/Mysql and
> the performance difference is mind blowing. Unfortunately the actual
> server specs and benchmarks can't be released without going through a nasty
> ball of red tape but let's just say django is ALOT more resource intensive
> and ALOT slower.

I (and many others, I'm sure) would love to help, but without details
it's basically impossible. Tuning hardware for Java is a great deal
different than tuning it for LAMPish stuff like Django.

Like Jeremy, though, I'd suspect a software problem: most code ported
from one language to another suffers the idioms of the old language.
Python isn't Java, but if you've coding like it is you're gonna have
issues.

If you can/want to share any additional details, we can try to help you out.

Jacob

[Vance Dubberly]

Kai,

   Very very untrue.  RAM is much faster than Disk IO.  Dynamically assembling a page from query results cached in database memory  is far faster than going to disk.  Caching the rendered page  via memcached, squid, or even in a database table is so much faster than serving static disk bound files it'll make your router cry out for mercy.

Vance

[Vance Dubberly]

No dynamically assembling the pages via django is faster than serving static files off disk with Apache.

Vance

  - Jacques Derrida

[Vance Dubberly]

Jeremy,

   I don't know about Java vs Python speeds. I've little doubt Java is alot faster if for no other reason than the amount of resources Sun puts behind it.   However resource  intensity  is a very big deal.  RAM is a huge factor here. The Tomcat App was running in a VM limited 256 meg and serving roughly 8x the number of requests.  Django suffers from running under mod_python in a prefork environment  where every process loads a full copy of django. So as I'm sure you can imagine RAM gets chewed up very quickly.  I've not read good things about running django in fastcgi so we've not tried that, frankly I don't trust fastcgi.  As for optimizations neither version of the app has any to speak of.   Just get me x,y and z from the database and push it into a page.  No caching, although we'll be adding memcached in the near future in an attempt to recover some speed ( at the cost of RAM ).  

  Frankly I think a generic python application server similar to tomcat would do a world of good for python apps in general. Something similar to cherrypy... but this is beyond the responsibilities of the django community which have plenty great work left.

  I really wish I could talk more about this provide actual benchmarks and server info and code example... unfortunately, GOVERNMENT should pretty well some up why I can't and getting this licensed under NOSA takes longer than the little bit of code written to do this is really worth. 

Vance

  - Jacques Derrida

[Jeremy Dunck]

On 6/28/07, Vance Dubberly <altje...@gmail.com> wrote:
> However resource intensity is a very big deal. RAM is a huge factor
> here.

Thanks for clearing up the issue.

It's true that the process-per-request model is heavier in terms of
RAM than alternatives, but it's generally fast. Maybe your available
processes are bottlenecking on something that could be addressed.

See here for an example of what you should expect:
http://www.davidcramer.net/curse/44/what-powers-curse.html
http://www.davidcramer.net/other/43/rapid-development-serving-500000-pageshour.html

They did make some changes to django for their specific need, but
they're not too exotic, and I doubt you're pushing that hard.

For more common performance tuning, have a look here:
http://www.jacobian.org/writing/2005/dec/12/django-performance-tips/

In particular, make sure KeepAlive is off for the django domain and
serve media with something lighter than apache.

...

> Frankly I think a generic python application server similar to tomcat
> would do a world of good for python apps in general. Something similar to
> cherrypy... but this is beyond the responsibilities of the django community
> which have plenty great work left.

Similar to CherryPy but not it? How come? In any case, if you're
thinking this way, consider Aspen/Stephane:
http://www.zetadev.com/software/aspen/

It appears to be thread-per-request. I haven't used it, but Chad's
been specifically trying to gain acceptance w/ the Django community;
I'm sure your interest would be appreciated.

> I really wish I could talk more about this provide actual benchmarks

At least you've described, roughly, your beef. I still think you
should be in the same ballpark, but I'm not even sure what you're
measuring.

Cheers,
Jeremy

[Jacob Kaplan-Moss]

On 6/28/07, Vance Dubberly <altje...@gmail.com> wrote:

> Django suffers from running under mod_python in
> a prefork environment where every process loads a full copy of django. So
> as I'm sure you can imagine RAM gets chewed up very quickly.

Are you running Django behind a load balancer like Perlbal or nginx?
One side-effect of propper LB is that you can drastically lower
MaxClients (because each backend client spends less time serving a
request). Look up "spoonfeeding" for more about this problem.

Even if you've only got a single web server you should always use
Perlbal or friends.

Jacob

[Gábor Farkas]

Jeremy Dunck wrote:
>> Frankly I think a generic python application server similar to tomcat
>> would do a world of good for python apps in general. Something similar to
>> cherrypy... but this is beyond the responsibilities of the django community
>> which have plenty great work left.
>
> Similar to CherryPy but not it? How come? In any case, if you're
> thinking this way, consider Aspen/Stephane:
> http://www.zetadev.com/software/aspen/
>
> It appears to be thread-per-request.

so all the usual issues (and non-issues) with the python GIL apply...

btw. i never understood what's so nice about those pure-python (or
pure-ruby (mongrel etc.)) web-servers..

for example regarding performance/memory-consumption/whatever,

why should apache+mod_python be worse than a pure-python-web-server?

let's say with the single-process multi-threaded apache-mode...?

gabor

[Graham Dumpleton]

On Jul 4, 11:20 pm, Gábor Farkas <g...@nekomancer.net> wrote:
> Jeremy Dunck wrote:
> >> Frankly I think a generic python application server similar to tomcat
> >> would do a world of good for python apps in general. Something similar to
> >> cherrypy... but this is beyond the responsibilities of the django community
> >> which have plenty great work left.
>
> > Similar to CherryPy but not it? How come? In any case, if you're
> > thinking this way, consider Aspen/Stephane:
> >http://www.zetadev.com/software/aspen/
>
> > It appears to be thread-per-request.
>
> so all the usual issues (and non-issues) with the python GIL apply...
>
> btw. i never understood what's so nice about those pure-python (or
> pure-ruby (mongrel etc.)) web-servers..
>
> for example regarding performance/memory-consumption/whatever,
>

> why should apache+mod_pythonbe worse than a pure-python-web-server?

>
> let's say with the single-process multi-threaded apache-mode...?

It may be of interest for you to read my recent blog post at:

http://blog.dscpl.com.au/2007/07/web-hosting-landscape-and-modwsgi.html

This talks about mod_wsgi, but various of the issues discussed there
with running mod_wsgi in embedded mode apply equally to mod_python.
Some of the things discussed are the balance between performance vs
security, memory usage, scaling ability of Apache and what impacts the
GIL has on performance when using multiple processors. Also ensure you
read comments as some discussion on trade offs between mod_wsgi and
mod_fastcgi or proxy like solutions.

Graham