How to do background processing with Django
shabda
taking actions. So how can I start background process, would it be as
simple as
...
t = threading.Thread(...)
t.setDaemon(True)
t.start()
return HttpResponse(..)
Or does django/apache have limitation on how threading can be used?
Would my thread be killed, once the response is sent back?
Tane Piper
It might not be exactly what you need, but there is Django Queue
Service (http://code.google.com/p/django-queue-service/)
I've implemented it in our django application to provide a queue
service to do backend processes. Our app is a mercurial manager, and
I built a queue to handle our remote cloning requests. There is a
queue called 'repoclone' and when a user creates on, this is the
workflow:
1. User creates request to clone remote repository
2. Django checks details are valid
3. Creates a message in the 'repoclone' queue
4. Django saves the instance of the repo details in the database with
created = False
5. Cron runs every 1 min, poping the next message off the queue
6. Django reads the queue JSON and decodes, grabs the repo information
from the database
7. Does the clone request
8. Updates the database to advise the clone has been created.
Hope that helps.
--
Tane Piper
Blog - http://digitalspaghetti.me.uk
Skype: digitalspaghetti
Wii: 4734 3486 7149 1830
This email is: [ ] blogable [ x ] ask first [ ] private
Graham Dumpleton
mini HTTP server set up to respond to XML-RPC requests. The Apache/
Django processes can then make an XML-RPC request to the long running
persistent back end process which then does the work in a separate
thread, via some form of queuing mechanism and thread pooling
mechanism as needs be to keep number of active threads down, or using
other means as appropriate.
Being in a single back end process, you can then also support XML-RPC
requests to check on status of jobs being done, cancel jobs etc etc.
You also don't then have to deal with the fact that Apache is
multiprocess and subsequent requests may not come back to the same
process if something needs to query outcome of job. You also don't
have to deal with possibility of Apache killing off one its processes,
which might contain running jobs, when it is recycling its processes.
Graham
shabda
things to do. I want to package it so that the only required things
are django and apache. Adding a HTTP server or configuring django-
queuing is too painful for the average user. :(
On Mar 26, 2:28 pm, Graham Dumpleton <Graham.Dumple...@gmail.com>
wrote:
Ian Lawrence
> are django and apache. Adding a HTTP server or configuring django-
> queuing is too painful for the average user. :(
what does the end user have to do with this?
you have a server by doing
wget http://svn.cherrypy.org/trunk/cherrypy/wsgiserver/__init__.py -O
wsgiserver.py
and setting up xml-rpc is not much more than this...i did this recently at
https://help.ubuntu.com/community/UMEGuide/ApplicationDevelopment/GPSEnabledWebApplication
and it is pretty simple to do (and really the best way IMO to run
background processes)
Ian
shabda
application, and want to use this, in a shared hosting environment.
On Mar 26, 7:39 pm, "Ian Lawrence" <irlawre...@googlemail.com> wrote:
> Hi
>
> > are django and apache. Adding a HTTP server or configuring django-
> > queuing is too painful for the average user. :(
>
> what does the end user have to do with this?
> you have a server by doing
> wgethttp://svn.cherrypy.org/trunk/cherrypy/wsgiserver/__init__.py-O
> wsgiserver.py
Joseph Heck
application either using Django as a pass through or configuring up
something with a reverse proxy in Apache or such. There's no reason
that I can think of that the complexity of maintaining a queueing
system for asynchronous processing would need to be passed outward to
a user.
If you want to give the user feedback on what's happening, you might
consider maintaining a state message like "in progress" or such within
Django that can be polled via an Ajax call or the like.
-joe