Standalone script with django's ORM and multiprocessing

[Spajderix]

Hi!

I've written a standalone script, which looks throught a table in db for
tasks to perform. It then starts subprocesses for each task it founds
using multiprocessing.Process class from python. Everything's fine when
there is one task, but when i try to start more subprocesses at once i get:

OperationalError: (2013, 'Lost connection to MySQL server during query')
raise errorclass, errorvalue
OperationalError: (2006, 'MySQL server has gone away')

I guess that this happens because all subprocesses share one connection,
and when one of them closes this connection, rest of subprocesses raises
an error. Do you know a way to go round this problem?

Regards
Spajderix

[Malcolm Tredinnick]

That certainly sounds believable. We call close() explicitly, too, so
that is why ongoing operations are interrupted in the middle.

> Do you know a way to go round this problem?

If you close the database connection, Django will open a new one the
next time it needs it. So I suspect you can work around this by
explicitly closing the connection immediately after you start a new
process (in the new process). Then the process will get its own
connection when you try to do something.

Regards,
Malcolm

[Spajderix]

Malcolm Tredinnick pisze:

Thank you! That solved the problem:)

[prabhu S]

Solution appears like a hack to me. Why do you close the connection in
every process? Can you not just close once in parent? Execute commits
alone in each process.

[Malcolm Tredinnick]

On Tue, 2009-08-11 at 03:12 -0700, prabhu S wrote:
[...]

> Why do you close the connection in
> every process? Can you not just close once in parent?

That would also work.

Regards,
Malcolm

[Spajderix]

prabhu S pisze:

> Solution appears like a hack to me. Why do you close the connection in
> every process? Can you not just close once in parent? Execute commits
> alone in each process.
>

I've checked that. Unfortunately, in my case, it won't work. I have a
loop looking like this:

for job in joblist:
p = Process(target=self.run_job)
p.start()

When i close connection before this loop i get errors like at the
beggining. When i close connection just before creating a new process i
get: 'lost connection while performing a query' (or something like that
:)). Finally when i close connection just right after this loop it
starts to behave weirdly. One time it works fine, but next time i launch
it i get mysql has gone away errors. I suppose this happens because
sometimes subprocesses fire up queries after parent finishes this loop,
but sometimes they're faster and still use parent's connection.

Closing connections in subprocesses might look a bit hacky. I could try
to somehow reinitialize db connection, but simply closing it, and
leaving the rest to django is a lot easier :)

Thank you for your replies. I appreciate it.

Regards
Spajderix