Setup_environ before or after forking?

[Andrew Fong]

I am running a python process separate from my actual Django web
server to do background processing. Since I need database access and I
like using Django's models, I call
django.core.management.setup_environ at the start of my background
process.

I am now trying to fork the process using the processing module
(http://pypi.python.org/pypi/processing) so I can run code on multiple
cores concurrently. Specifically, I use the Pool class to create a
bunch of worker processes that I can asynchronously assign work to.
Each of my worker processes also need DB access via Django's models.

Question: Does it make more sense to call setup_environ before or
after forking?

In particular, I don't know how Django sets up its database
connection. Do forked processes share one connection or does each
process have their own connection? Are there pros and cons to each?

Also, if it matters, I'm currently running MySQL 5.0.5 as the DB.

Thanks!

-- Andrew

[Malcolm Tredinnick]

On Tue, 2009-01-27 at 12:56 -0800, Andrew Fong wrote:
[...]

> Question: Does it make more sense to call setup_environ before or
> after forking?

I'd do it after forking.

>
> In particular, I don't know how Django sets up its database
> connection.

A connection is created the first time you access a database and then
closed when the response is sent back (or, in standalone cases, when you
either explicitly close it or when the process ends).

> Do forked processes share one connection or does each
> process have their own connection?

The most precise answer is "it depends", but the simplest answer is that
they would be shared and it wouldn't behave as you expect. Don't do
that. Forked processes share file descriptors, for example (well,
they're duplicated, not shared), so if the connection uses a file
descriptor, which it will, that is duplicated.

> Are there pros and cons to each?

Creating the connection after forking will work. Creating the connection
before forking will likely lead to lots of difficult to diagnose race
conditions and the like.

Regards,
Malcolm

[Chainz O.W.N]

If you are still in need of a Django expert, have you checked out https://datepalm.engineering to see if they can help?

They are Django experts from what I understand, and based in the US.