Python Multiprocessing With Django

[Vitaly Babiy]

I have a management command that that starts up a few process to process a request: http://dpaste.com/7925/.

But every so offen I get a exception say mysql went a way, but it didn't.
Process PoolWorker-3:
Traceback (most recent call last):
 File "/usr/lib/python2.5/site-

packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py", line 236, in _bootstrap
   self.run()
 File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/process.py", line 93, in run
   self._target(*self._args, **self._kwargs)
 File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", line 71, in worker
   put((job, i, result))
 File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/queues.py", line 353, in put
   return send(obj)
PicklingError: Can't pickle <class 'http_tracker.models.DoesNotExist'>: attribute lookup http_tracker.models.DoesNotExist failed
Traceback (most recent call last):
 File "/home/vbabiy/projects/git-projects/howsthedotcom/manage.py", line 11, in <module>
   execute_manager(settings)
 File "/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line 350, in execute_manager
   utility.execute()
 File "/usr/lib/python2.5/site-packages/django/core/management/__init__.py", line 295, in execute
   self.fetch_command(subcommand).run_from_argv(self.argv)
 File "/usr/lib/python2.5/site-packages/django/core/management/base.py", line 195, in run_from_argv
   self.execute(*args, **options.__dict__)
 File "/usr/lib/python2.5/site-packages/django/core/management/base.py", line 222, in execute
   output = self.handle(*args, **options)
 File "/home/vbabiy/projects/git-projects/howsthedotcom/tracker/management/commands/tracker.py", line 37, in handle
   res = p.map(check_site, sites)
 File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", line 148, in map
   return self.map_async(func, iterable, chunksize).get()
 File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-linux-i686.egg/multiprocessing/pool.py", line 422, in get
   raise self._value
_mysql_exceptions.OperationalError: (2013, 'Lost connection to MySQL server during query')

Any one got any idea why this is happening?

Vitaly Babiy

[Jirka Vejrazka]

> I have a management command that that starts up a few process to process a
> request: http://dpaste.com/7925/.

Hi, I was recently debugging similar issue and came to a conclusion
(which may be wrong of course :) that multiprocessing and Django DB
connections don't play well together. I ended up closing Django DB
connection first thing in the new process. It'll recreate a new
connection when it needs one, but that one will have no references to
the connection used by the parent.

So, my Process.start() calls a function which starts with:
>>> from django.db import connection
>>> connection.close()

This solved my problem.

Cheers

Jirka

[Vitaly Babiy]

Alright I will have to give this a try

Vitaly Babiy

[Vitaly Babiy]

Thanks Jirka, that fixed the problem. But I think I will file a bug report that seem to be very hacky :)

Vitaly Babiy

[Jirka Vejrazka]

> Thanks Jirka, that fixed the problem. But I think I will file a bug report
> that seem to be very hacky :)

Great, I'm glad it solved your problem. Personally, I don't think it
is a bug, as we both pushed Django well beyond its "area of
expertise". I really believe that Django is great for serving web
content and everyone (including) me trying to implement it in
different areas must be ready for problems and gotchas here and there.
But YMMV.

Cheers

Jirka

[Subha Bera]

My program trigger some regression file in a remote host for that I used paramiko module.

 But I want to trigger same kinda file to different machines parallel-ly.

I am using django so that people can use this as web app.

I am using url-view-template model. How can use I multi threading in the views function? If you come up with a solution please let me know.

It requires the( if "__name__"="main": ).How can I do that?