Django, shared hosting memory limits and memory optimization
chrominance
without much concern for memory limits--coming from PHP, my knowledge
of memory issues is practically nil. Of course, Webfaction's plans all
have memory limits, and we're currently on Shared 1, which imposes a
40MB limit. We're running the standard apache2/mod_python config, with
media served via a separate apache2 process as Webfaction and the
Django site recommends. The Debug setting has been set to False. I
believe those are the standard steps recommended for reducing resource
usage.
Here's a look at the ps output:
PID RSS COMMAND
17122 19652 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
19823 32232 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
19824 30612 /home/---/webapps/django/apache2/bin/httpd -f /home/---/
webapps/django/apache2/conf/httpd.conf
4253 1808 sshd: ---@pts/2
4254 1456 -bash
5204 936 /bin/sh /home/---/scripts/mem-usage
5205 764 ps -u --- -o pid,rss,command
5206 708 awk {print $0}{sum+=$2} END {print "Total", sum}
Total 88168
Presumably the three apache2 processes are the Django processes that
actually count against the memory usage limit; they add up to about
80MB. Clearly there is a problem here. Because we don't yet have the
domain for the site, traffic has been extremely low (basically just
one person entering data into the site); I fear for the day we open up
the site to everyone, mainly because I have no clue how memory usage
will rocket when people start visiting the site on a regular basis.
I also have no conception of what is "normal" memory usage for an
application. For all I know, the complexity of the site may simply
demand 80MB of RAM, and there might be nothing I can do about it. How
I go about figuring that out, though, is a big mystery for me.
So I guess my questions are: how do I optimize my Django setup to use
less memory? Should I be looking at the Django project I built for
ways to optimize the code, and what should I look for? Is there a
general guide on optimizing Django applications, or at least some tips
and tricks? And how can I audit the project I've built to figure out
if 80MB is actually reasonable or if I should be able to crush it down
further?
(P.S. If someone has any idea what Webfaction means when it says you
should be able to run "3 small Django sites" in 40MB, I'd love to
know. What constitutes a "small site," anyways?)
Jay Parlar
>
> I've recently put up a newspaper site on Webfaction that was developed
> without much concern for memory limits--coming from PHP, my knowledge
> of memory issues is practically nil. Of course, Webfaction's plans all
> have memory limits, and we're currently on Shared 1, which imposes a
> 40MB limit. We're running the standard apache2/mod_python config, with
> media served via a separate apache2 process as Webfaction and the
> Django site recommends. The Debug setting has been set to False. I
> believe those are the standard steps recommended for reducing resource
> usage.
Did you remember to restart Apache after setting Debug to False? I
have a site running on Shared1, and the memory usage was close to what
you have, until I set Debug to False and restarted.
Also, you might want to ping the Webfaction Django forum
(http://forum.webfaction.com/viewforum.php?id=19) as they're pretty
good about responding, and one of the admins might check if something
is wrong with your config.
And I *believe* the "3 small sites" comes from the fact that on
Shared1, a max of 3 Apache processes will run, so you can run a site
on each process. I might be completely wrong on that though.
Jay P
chrominance
webapps/django directory provided by Webfaction. Perhaps there's a
more absolute restart command I don't know about, but I'm pretty sure
that's the one--whenever I use it, memory usage seems to go down.
Also, I'm not sure if this is relevant, but whenever I do restart the
apache processes, memory usage doesn't reset to the same baseline
level--for example, the site ran just fine with about 30-40MB of RAM
over the weekend, but restarting apache now only gets us down to about
65MB. And upon my last check the apache processes peaked at about
85MB. If there's a memory leak in my application, it wouldn't still
leak after I've restarted apache, would it?
I'll try the Webfaction forum too. Thanks, and keep the advice coming!
On May 29, 8:15 pm, "Jay Parlar" <par...@gmail.com> wrote:
Graham Dumpleton
process, which effectively acts as a supervisor process, and the child
processes which actually accept and handle requests. When one uses
'restart' with the traditional 'apachectl' management command, it only
kills off and restarts the child processes and does not stop the
parent process. Because the child processes are a fork of the parent
process, if for some reason there was a memory leak in the parent
process, that leak would be inherited by the child process on a
'restart'. To fully kill off the parent and the children, you would do
a 'stop' followed by a 'start' if using 'apachectl'. As to what the
Webfaction script does I can only speculate.
What you need to thus do is get 'ps' to also output the 'PPID' value
using 'l' flag so you can work out which is the parent 'httpd'
process. For example:
$ ps axlwww | egrep '(PPID|httpd)'
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT
TIME COMMAND
0 429 1 0 31 0 31364 2004 - Ss ??
0:00.11 /usr/local/apache-2.2.4/bin/httpd -k start
70 430 429 0 31 0 30524 592 - S ??
0:00.00 /usr/local/apache-2.2.4/bin/httpd -k start
501 431 429 0 31 0 31136 852 - S ??
0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start
501 432 429 0 31 0 31136 848 - S ??
0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start
70 433 429 0 31 0 45296 1172 - S ??
0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start
70 434 429 0 31 0 45296 1176 - S ??
0:00.01 /usr/local/apache-2.2.4/bin/httpd -k start
The parent process here is 429. Normally this process should only use
quite small amounts of private memory. I have never seen an instance
of where the parent process ballooned out in size, but if you have a
dodgy Apache module it feasibly might. For example, from 'top':
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE
VSIZE
434 httpd 0.0% 0:00.01 27 55 93 496K 2.73M 1.15M
44.2M
433 httpd 0.0% 0:00.01 27 55 93 496K 2.73M 1.14M
44.2M
432 httpd 0.0% 0:00.00 1 10 37 204K 2.50M 848K
30.4M
431 httpd 0.0% 0:00.00 1 10 37 208K 2.50M 852K
30.4M
430 httpd 0.0% 0:00.00 1 10 33 188K 2.02M 592K
29.8M
429 httpd 0.0% 0:00.11 1 11 38 60K 2.73M 1.96M
30.6M
Ie., 60Kbytes here only, versus 200-500Kbytes for children.
Anyway, work out whether how you are restarting Apache is killing of
the parent process. If not work out how to restart Apache so it is.
Also look at how much memory the parent is using and see if it is
small.
This at least is some sanity checks you can do.
BTW, what Apache modules are actually being loaded using LoadModule
directive.
Graham
On May 30, 11:42 am, chrominance <chromina...@gmail.com> wrote:
> I've restarted apache several times, using the restart command in the
> webapps/django directory provided by Webfaction. Perhaps there's a
> more absolute restart command I don't know about, but I'm pretty sure
> that's the one--whenever I use it, memory usage seems to go down.
>
> Also, I'm not sure if this is relevant, but whenever I do restart the
> apache processes, memory usage doesn't reset to the same baseline
> level--for example, the site ran just fine with about 30-40MB of RAM
> over the weekend, but restarting apache now only gets us down to about
> 65MB. And upon my last check the apache processes peaked at about
> 85MB. If there's a memory leak in my application, it wouldn't still
> leak after I've restarted apache, would it?
>
> I'll try the Webfaction forum too. Thanks, and keep the advice coming!
>
> On May 29, 8:15 pm, "Jay Parlar" <par...@gmail.com> wrote:
>
> > On 5/29/07, chrominance <chromina...@gmail.com> wrote:
>
> > > I've recently put up a newspaper site on Webfaction that was developed
> > > without much concern for memory limits--coming from PHP, my knowledge
> > > of memory issues is practically nil. Of course, Webfaction's plans all
> > > have memory limits, and we're currently on Shared 1, which imposes a
> > > 40MB limit. We're running the standard apache2/mod_pythonconfig, with
chrominance
17122 1 21504 /home/---/webapps/django/apache2/bin/httpd -f /
home/---/webapps/django/apache2/conf/httpd.conf
14180 14154 1808 sshd: ---@pts/2
14181 14180 1464 -bash
14892 17122 20496 /home/---/webapps/django/apache2/bin/httpd -f /
home/---/webapps/django/apache2/conf/httpd.conf
14893 17122 20496 /home/---/webapps/django/apache2/bin/httpd -f /
home/---/webapps/django/apache2/conf/httpd.conf
15175 14181 940 /bin/sh /home/---/scripts/mem-usage
15176 15175 764 ps -u --- -o pid,ppid,rss,command
15177 15175 708 awk {print $0}{sum+=$2} END {print "Total", sum/1024,
"MB"}
Total 66.58 MB
So it appears the first apache2 process is the parent. I'm going to
guess from your comments that 21MB is perhaps a bit much for the
parent apache process.
As for what's being loaded: mod_python, mod_log_config, mod_dir,
mod_mime, mod_rewrite, mod_env. Most of those look fairly important,
though I haven't done any research into it. Obviously mod_python is
needed at the very least.
Stopping and restarting apache does work as promised, as all my apache
processes are back to ~3MB each (15MB total). After a couple of hits
usage goes up to about 35MB. So I guess it could still be a memory
leak, though I've run ab a couple of times (1000 requests, 100
concurrency) and the memory footprint never seems to increase very
much.
On May 29, 10:29 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
Graham Dumpleton
> PID PPID RSS COMMAND
> 17122 1 21504 /home/---/webapps/django/apache2/bin/httpd -f /
> home/---/webapps/django/apache2/conf/httpd.conf
> 14180 14154 1808 sshd: ---@pts/2
> 14181 14180 1464 -bash
> 14892 17122 20496 /home/---/webapps/django/apache2/bin/httpd -f /
> home/---/webapps/django/apache2/conf/httpd.conf
> 14893 17122 20496 /home/---/webapps/django/apache2/bin/httpd -f /
> home/---/webapps/django/apache2/conf/httpd.conf
> 15175 14181 940 /bin/sh /home/---/scripts/mem-usage
> 15176 15175 764 ps -u --- -o pid,ppid,rss,command
> 15177 15175 708 awk {print $0}{sum+=$2} END {print "Total", sum/1024,
> "MB"}
> Total 66.58 MB
>
> So it appears the first apache2 process is the parent. I'm going to
> guess from your comments that 21MB is perhaps a bit much for the
> parent apache process.
Depends on what RSS means for that platform when running ps. This may
count private memory used plus shared memory use. If Webfaction is
counting shared memory use in your 40MB limit would suck somewhat, as
they would be double counting across all processes.
If you run 'top' it generally splits resident (private) from shared in
what it outputs. You may be able to get 'ps' to show them separately,
but need to find which field to display.
If you do 'ldd' on Apache executable you can get an idea of all the
shared libraries it links with. There might be a fare few. These
should count as being shared, what is left over is effectively private
memory used by process, although by rights the Apache modules also
should be seen as shared, but for mod_python in certain case this may
not be so.
BTW, if you run 'ldd' on the mod_python.so file from the Apache
modules directory, does it use Python as a shared library or is there
no reference to libpython2.?.so at all, meaning it is embedded with in
mod_python.so? What is the actual size of your mod_python.so file?
> As for what's being loaded: mod_python, mod_log_config, mod_dir,
> mod_mime, mod_rewrite, mod_env. Most of those look fairly important,
> though I haven't done any research into it. Obviously mod_python is
> needed at the very least.
Others shouldn't use much memory, the mod_python one is main issue
which is why I asked about whether it uses a shared library for Python
or not, as not using a shared library can on some systems cause
private memory related to mod_python to be larger than it needs to be.
> Stopping and restarting apache does work as promised, as all my apache
> processes are back to ~3MB each (15MB total). After a couple of hits
> usage goes up to about 35MB.
Base Django setup can take up 5-7MB without even your own stuff so it
can jump up quite a bit. Is that 35MB in total across all processes or
per process?
> So I guess it could still be a memory
> leak, though I've run ab a couple of times (1000 requests, 100
> concurrency) and the memory footprint never seems to increase very
> much.
What version of mod_python are you using?
Graham
chrominance
> count private memory used plus shared memory use. If Webfaction is
> counting shared memory use in your 40MB limit would suck somewhat, as
> they would be double counting across all processes.
>
> If you run 'top' it generally splits resident (private) from shared in
> what it outputs. You may be able to get 'ps' to show them separately,
> but need to find which field to display.
I tried top initially but I couldn't get it to return private memory
at all. I'm not sure if that's just a quirk of Red Hat or a quirk of
Webfaction's shared hosting setup.
> BTW, if you run 'ldd' on the mod_python.so file from the Apache
> modules directory, does it use Python as a shared library or is there
> no reference to libpython2.?.so at all, meaning it is embedded with in
> mod_python.so? What is the actual size of your mod_python.so file?
>
mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of
the shared libraries.
> Base Django setup can take up 5-7MB without even your own stuff so it
> can jump up quite a bit. Is that 35MB in total across all processes or
> per process?
>
About 4MB for the parent process, and 13MB for the two child
processes.
> What version of mod_python are you using?
3.2.8.
It's been a couple of hours since the hard restart and memory usage is
still at about 35MB. I'm now wondering if perhaps the last time I did
a full restart (if ever), I'd left Debug = True in the settings.py,
and the SQL query storage hadn't been emptied or some other junk.
Perhaps doing the full stop/start with Debug = False set might've
cleared out all the garbage. I'll watch it over the next couple of
hours with my fingers crossed...
Graham Dumpleton
> > BTW, if you run 'ldd' on the mod_python.so file from the Apache
> > modules directory, does it use Python as a shared library or is there
> > no reference to libpython2.?.so at all, meaning it is embedded with in
> > mod_python.so? What is the actual size of your mod_python.so file?
>
> mod_python.so is about 4MB. It doesn't show libpython2.x.so as one of
> the shared libraries.
Which shows that the Python installation on the box is really crappy.
One of the problems with using RedHat derived distributions.
That the mod_python.so is so large means that there is no shared
library built and provided with the Python installation, or at least
not with the version that mod_python was compiled against.
Alternatively, there is a shared library, but it is only in /usr/lib
and not along side the libpython2.?.a file in /usr/lib/python2.?/
config directory.
In other words, when mod_python was built, all it found in the /usr/
lib/python.2?/config directory was the static library archive. As a
consequence, 'libtool' when it creates the mod_python.so file has no
choice but to extract all the object files from the static Python
library and embed them into mod_python.so itself. Thus, instead of
mod_python.so being less than 0.5MB through being able to link
dynamically to the Python library it is much larger.
Worse is that the size of mod_python.so indicates that debug may have
been compiled into the Python library. For an optimised build of the
Python library it is normally only about 1MB, not over 3MB as in this
case.
Even more worse than that, is what now happens when that mod_python.so
is loaded by Apache. Now the way I understand it, normally a
dynamically loaded module should really be shared and thus memory cost
only applies once across all processes that load it, however, if the
object files in that static Python library are not truly position
independent, when mod_python.so gets loaded, the dynamic linker will
have to perform address relocations to fixup references to function
and data from the Python library objects. As a consequence, rather
than being shared memory, the mod_python.so becomes private memory to
the parent Apache process. When Apache forks off the child processes,
they in turn will each end up with their own private copy of up to 4MB
of memory that mod_python.so consumes when it could all have been
shared. So, even on a fresh restart, the Apache processes are going to
be much larger than they need to be because of the way that Python was
built in the first place.
Now whether this is going to happen like this I am not 100% sure as it
can depend on the operating system and perhaps the age of the
operating system version as well and how they handle loading of
dynamically loadable modules. I am led to believe that this can occur
on Solaris, to what degree it applies on different Linux versions I am
not as sure.
Either way, that debug seems to be built into your Python library is
not good. If Webfaction used a better build of Python with
optimisation and a shared library they could quite easily cut down on
memory use allowing your applications to have more space within their
set limit.
What is even more annoying is that various people run around
continually saying how crap mod_python is because of how much memory
it uses, when in practice a lot of it has to do with poorly built
Python versions as I show above. :-(
> > Base Django setup can take up 5-7MB without even your own stuff so it
> > can jump up quite a bit. Is that 35MB in total across all processes or
> > per process?
>
> About 4MB for the parent process, and 13MB for the two child
> processes.
>
> > What version of mod_python are you using?
>
> 3.2.8.
Which has various memory leaks. Even 3.2.10 has memory leaks. I highly
recommend upgrading to mod_python 3.3.1 if you can do it as it
eliminates a lot of memory leaks. If you also do a rebuild of Python
with optimisation and a shared library you should hopefully see less
memory being used then at startup.
Graham
chrominance
memory usage was indeed my failure to properly restart all the apache
processes; it looks like the parent process was still storing debug
info from when I had DEBUG=True in my settings.py, so that setting it
to False and soft restarting via apachectl had no effect. For the past
couple of days memory has been sitting between 30-35MB, which is a
hell of a lot better.
While I was looking up memory optimization stuff I stumbled upon a
couple of useful links I'll throw out for posterity's sake:
http://www.car-chase.net/2007/apr/30/django-deployment-lessons-learned/
http://emergent.urbanpug.com/?p=60
http://blog.webfaction.com/tips-to-keep-your-django-mod-python-memory-usage-down
A couple of people pointed the Webfaction crew to Graham's comments
and they say they're looking into the mod_python setup, so hopefully
that leads to some results down the road. They've also been testing
Litespeed/FastCGI setups, which might help with limiting memory usage
via FastCGI config.
Graham Dumpleton
> Epilogue: it looks like the major culprit behind my skyrocketing
> memory usage was indeed my failure to properly restart all the apache
> processes; it looks like the parent process was still storing debug
> info from when I had DEBUG=True in my settings.py, so that setting it
> to False and soft restarting via apachectl had no effect. For the past
> couple of days memory has been sitting between 30-35MB, which is a
> hell of a lot better.
The problem with that hypothesis is that mod_python in the Apache
parent process never actually imports any of your user code modules
for Django, that is only ever done in the child processes. Thus, there
is no way it could have remembered the setting in the first place.
This is why I couldn't understand why your Apache parent process was
so big, as it effectively doesn't do that much.
One possibility for child processes being so big is that for some
reason Python was loading the .pyc file for the settings file and it
was produced from settings.py when DEBUG=True was set but for some
reason when it was set to DEBUG=False the .pyc file wasn't regenerated
and Python was still using it for some reason.
Thus, as a precaution, it may be advisable to delete all .pyc files if
you have run the development Django server and then subsequently using
Apache on the same files. This would especially be required if you had
copied all the .py/.pyc files from another box and not preserved date/
time stamps as the copy would have set the .py/.pyc files to be the
same and thus Python wouldn't realise the .py file was actually newer.
Do note that although Python running as used by mod_python will
use .pyc files, it will not replace them if Apache runs as a user
different to the owner of the directories/files.
Graham
Graham Dumpleton
bring down memory use, is if you have access to main Apache
configuration file, try setting:
PythonOptimize On
at global scope outside of any virtual host containers.
This will have the same affect as having supplied '-O -O' option to
command line Python. Do note though that this will result in doc
strings being discarded, which is where potential for memory savings
come from, but some Python modules rely on doc strings to be present
to work. Thus, you yo will need to do sufficient testing to ensure you
aren't using any Python modules which have a dependency on doc strings
being present.
Graham
On Jun 3, 8:41 am, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
chrominance
Apache process hasn't budged from 3MB since the restart and I can't
imagine what else would have changed between the 90MB period and the
30MB period. I loaded my project from svn so there were no .pyc files
initially, I've never used the built-in dev server, and I didn't
delete the .pyc files when I restarted Apache so I doubt that's made
any difference.
On Jun 2, 6:41 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
Remi
> optimisation and a shared library they could quite easily cut down on
> memory use allowing your applications to have more space within their
> set limit.
>
> What is even more annoying is that various people run around
> continually saying how crap mod_python is because of how much memory
> it uses, when in practice a lot of it has to do with poorly built
> Python versions as I show above. :-(
>
> > > Base Django setup can take up 5-7MB without even your own stuff so it
> > > can jump up quite a bit. Is that 35MB in total across all processes or
> > > per process?
>
> > About 4MB for the parent process, and 13MB for the two child
> > processes.
>
> > > What version of mod_python are you using?
>
> > 3.2.8.
>
> Which has various memory leaks. Even 3.2.10 has memory leaks. I highly
> recommend upgrading to mod_python 3.3.1 if you can do it as it
> eliminates a lot of memory leaks. If you also do a rebuild of Python
> with optimisation and a shared library you should hopefully see less
> memory being used then at startup.
Just a quick update to mention that we've taken these comments into
account and we've optimized our Django setup at WebFaction:
- Django apps now comes with mod_python-3.3.1 by default
- mod_python is now compiled with a shared libpython.so (the size of
mod_python.so went from 3MB down to 300KB).
- you can now choose between Python-2.4 or Python-2.5 when creating a
Django app.
Remi
--
WebFaction - Hosting for an agile web
http://www.webfaction.com