Configuring Apache and mod_python for Django

[Peter Rowell]

In another thread (http://groups.google.com/group/django-users/
browse_thread/thread/962cfdf7609839eb/),

On Mar 23, 11:48 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
> For example, using prefork MPM because PHP is not thread safe in
> conjunction with Python web applications in embedded mode in a memory
> constrained VPS is a bad idea. You either ditch PHP and use worker
> MPM, or you go to mod_wsgi and use daemon mode for the Python web
> application.

He said it with authority. After perusing his site (in particular,
http://www.dscpl.com.au/wiki/ModPython/Articles), it would appear that
he speaks from intimate knowledge of the issue.

His statement bothered me for two reasons:

1. It was at odds with the Django docs (http://www.djangoproject.com/
documentation/modpython/), which specifically say "you should use
Apache's prefork MPM, as opposed to the worker MPM."

2. Because it forced me to confront my own ignorance of exactly *why*
prefork MPM is preferred (absolutely required?) over worker MPM.


Questions:

+ What is it about python and/or Django that forces(?) the use of
prefork MPM rather than worker MPM?

+ Does this have to do with python's GIL (global interpreter lock)?

+ Is there some inherent non-thread-safe aspect of either? Globals?
Database caching?

+ Or is it, as Graham suggests, that it is the combination of PHP and
Python that requires prefork?

Inquiring minds want to know.

[Graham Dumpleton]

Also see what I said towards end of:

http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango

as to why.

I might clarify later and address your specific questions when have
time.

Graham

[Graham Dumpleton]

Now for some more comments.

On Mar 28, 2:01 am, Peter Rowell <hedron...@gmail.com> wrote:
> In another thread (http://groups.google.com/group/django-users/
> browse_thread/thread/962cfdf7609839eb/),
>
> On Mar 23, 11:48 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
> wrote:
>
> > For example, using prefork MPM because PHP is not thread safe in
> > conjunction with Python web applications in embedded mode in a memory
> > constrained VPS is a bad idea. You either ditch PHP and use worker
> > MPM, or you go to mod_wsgi and use daemon mode for the Python web
> > application.
>

> He said it with authority. After perusing his site (in particular,http://www.dscpl.com.au/wiki/ModPython/Articles), it would appear that

> he speaks from intimate knowledge of the issue.
>
> His statement bothered me for two reasons:
>
> 1. It was at odds with the Django docs (http://www.djangoproject.com/
> documentation/modpython/), which specifically say "you should use
> Apache's prefork MPM, as opposed to the worker MPM."

Part of the reasoning for this was that prefork was single threaded
and Django wasn't originally written with threading in mind. There
were also issues back then with some database adapters not being
thread safe. Thus it was erring on the side of caution. These days a
lot of people use Django on Windows and with FASTCGI in multithreaded
configurations and Django documentation even covers those
multithreaded configurations.

> 2. Because it forced me to confront my own ignorance of exactly *why*
> prefork MPM is preferred (absolutely required?) over worker MPM.
>
> Questions:
>
> + What is it about python and/or Django that forces(?) the use of
> prefork MPM rather than worker MPM?

These days there isn't really anything, except maybe perhaps the
thread safety of your own code on top of Django and any special third
party modules you may use. Thus why you simply need to make sure you
test your own code on a comparable setup before production deployment.

> + Does this have to do with python's GIL (global interpreter lock)?

Not at all. Some will argue that the GIL affects multithreaded
performance, but in practice for web applications running under Apache
it doesn't have as great an effect as a non web application running in
a standalone process. See:

http://blog.dscpl.com.au/2007/09/parallel-python-discussion-and-modwsgi.html

> + Is there some inherent non-thread-safe aspect of either? Globals?
> Database caching?

There can be if access wasn't coded to be thread safe.

> + Or is it, as Graham suggests, that it is the combination of PHP and
> Python that requires prefork?

I can't remember what the original conversation was about but possibly
what I was talking about was that if you want to cut down on memory
usage, you can reduce the number of Apache child processes by using
worker MPM instead of prefork MPM. The problem with switching to
worker MPM is that you can cause problems if you are also using PHP
within the same web server. This is because not all third party
modules for PHP may not be thread safe. Older versions of PHP may also
not be thread safe from memory. These issues cause a problem for PHP
applications, not the Python application.

> Inquiring minds want to know.

Hopefully that and the document I pointed at previously explains
things a bit better. Otherwise you may need to explain your concerns a
bit better.

Graham

[Peter Rowell]

Graham --

Thanks very much for your replies.

My concerns were from having two authorities saying different things
-- it makes the unwashed masses nervous. :-)

Of course, *now* I need to do a code review, since I hadn't been
thinking "thread safe" when I wrote it.

Is anyone (Django core developers?) reading this who can verify that
Django is, in fact, thread safe and plays well under worker MPM?

TIA,
Peter

[Graham Dumpleton]

The best I have ever been able to get out of anyone associated with
Django development about thread safety is the comments by Jacob Kaplan-
Moss in the following thread:

http://groups.google.com/group/django-developers/browse_frm/thread/905f79e350525c95/dfed56f8ed65aed2

Graham

[Peter Rowell]

> The best I have ever been able to get out of anyone associated with
> Django development about thread safety is the comments by Jacob Kaplan-
> Moss in the following thread:

Sigh. Reading through that thread was singularly unsatisfying. I guess
that, for the moment, I'll have to stick with prefork.

If Django, minus any DB-related issues, is thread safe, and if the
supplied apps (auth comes to mind) are written correctly, then perhaps
Django-out-of-the-box could be declared thread safe.

Some appropriate warnings/suggestions could be put in the docs. Maybe
point out user operations that could pose a problem and suggest
possible ways to handle them.

Thanks again,
Peter

[Eric]

I currently use lighttpd and fastcgi, and I noticed that in the django
documentation that they mention using threaded instead of prefork when
running your fcgi server.

I had the same fears and I used prefork first, but after some load
testing of both, I decided to use threaded. I'm a brave soul and I
assumed that if the documentation prefered threaded, it was safe.

Generally your pretty safe with threading unless you're using module
level or global variables.

I hope that helps.

Eric.

[Eric]

Sorry, I forgot to post the url to the fastcgi docs
http://www.djangoproject.com/documentation/fastcgi/

[James Bennett]

On Sat, Mar 29, 2008 at 3:52 AM, Graham Dumpleton
<Graham.D...@gmail.com> wrote:
> The best I have ever been able to get out of anyone associated with
> Django development about thread safety is the comments by Jacob Kaplan-
> Moss in the following thread:

And that's really about the best anyone can say; there's never been
any exhaustive review of the codebase to look for threading issues,
but at the same time there are people who run Django under various
threaded solutions and it seems to be working for them (and if it
doesn't they should report bugs).

The more general problem of writing thread-safe code is, of course, a
huge topic, so I'm not sure that we could take Peter's suggestion and
document "how to write thread-safe Django apps" without straying quite
far from the "Django" bit and over into the "writing thread-safe code"
bit ;)

--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."