Launching Background Processes from a View
139 views
Skip to first unread message
Heath
Sep 6, 2010, 1:19:08 AM9/6/10
to Django users
Hello All,
I realize this has been discussed at some length, but the posts I
found keep leading me back to the start, so I thought I'd try..
I currently have some python "worker" scripts that make great use
pyInotify, an event based "watch" manager. When files arrive in a "hot
folder", the correct actions proceed. These scripts use a forever loop
while "watching".
Historically, the processes were managed at the command line, and
killed when no longer needed.
There are many instances now, and I wish to manage these processes via
a web app. The Django workflow is:
base URL mapped to a central monitoring view. The model keeps the
instance name, pid and status.
When action is desired (creating, starting, stopping) , function
buttons in the template call jQuery click handlers which in turn call
an Ajax request back to the appropriate view function.
The view function performs the required action and returns the name,
PID and custom status, and updates the database and returns any
appropriate data back to jQuery for DOM insertion.
The worker script subclasses multiprocessing.Process so that I can get
the pid and use the control methods
The problem is, since these worker scripts just sit and wait on the
watched folders, the view never returns and the page appears to
"hang".
I've investigated subprocess.Popen, os.system(), various spawns, and
external modules such as celery, supervisor and and even god. - The
latter seem more suited for the queueing of "short-term" tasks.
What I require seems simple, just run the requested process in the
background. The terminal equivalent would be:
"nohup <process> <args> &" and return control to the view.
Any ideas on how to achieve this?
Many thanks,
-heath carlisle
I realize this has been discussed at some length, but the posts I
found keep leading me back to the start, so I thought I'd try..
I currently have some python "worker" scripts that make great use
pyInotify, an event based "watch" manager. When files arrive in a "hot
folder", the correct actions proceed. These scripts use a forever loop
while "watching".
Historically, the processes were managed at the command line, and
killed when no longer needed.
There are many instances now, and I wish to manage these processes via
a web app. The Django workflow is:
base URL mapped to a central monitoring view. The model keeps the
instance name, pid and status.
When action is desired (creating, starting, stopping) , function
buttons in the template call jQuery click handlers which in turn call
an Ajax request back to the appropriate view function.
The view function performs the required action and returns the name,
PID and custom status, and updates the database and returns any
appropriate data back to jQuery for DOM insertion.
The worker script subclasses multiprocessing.Process so that I can get
the pid and use the control methods
The problem is, since these worker scripts just sit and wait on the
watched folders, the view never returns and the page appears to
"hang".
I've investigated subprocess.Popen, os.system(), various spawns, and
external modules such as celery, supervisor and and even god. - The
latter seem more suited for the queueing of "short-term" tasks.
What I require seems simple, just run the requested process in the
background. The terminal equivalent would be:
"nohup <process> <args> &" and return control to the view.
Any ideas on how to achieve this?
Many thanks,
-heath carlisle
Aljoša Mohorović
Sep 6, 2010, 3:25:24 AM9/6/10
to django...@googlegroups.com
On Mon, Sep 6, 2010 at 7:19 AM, Heath <he...@brownbarn.com> wrote:
> What I require seems simple, just run the requested process in the
> background. The terminal equivalent would be:
>
> "nohup <process> <args> &" and return control to the view.
>
> Any ideas on how to achieve this?
> What I require seems simple, just run the requested process in the
> background. The terminal equivalent would be:
>
> "nohup <process> <args> &" and return control to the view.
>
> Any ideas on how to achieve this?
if this already works in a shell is there some reason why
"os.system('cmd')" or something similar doesn't work for you?
Aljosa Mohorovic
Heath
Sep 6, 2010, 1:20:45 PM9/6/10
to Django users
Thanks! yes,
os.system() will launch the process and return control, but then I'd
have to write a utility to get the PID and other data about the
process.
I guess I'm looking for a definitive answer that the built-in process
modules cannot launch a process in the background.
Here are my findings:
subprocess.Popen and multiprocessing offer properties and methods
ideal for my task, but won't return to the view until the process
ends.
Daemonizing these will allow the calling script (django, in this case)
to quit, but still won't return from the view.
celery/gettoq/rabbitMQ allows for the creation of a queue, but you
must specify a predefined (as far as I know) number of workers in the
config. This is great for processor-intensive tasks that may only run
for a short period of time, as the worker pool stays the same size,
and processes new requests when there is a worker available. But, in
my case, the processes are not doing much until there is folder
activity, and I need to add worker instances dynamically.
I feel like I'm just missing something.. Launching system processes
must be a common idiom for a web app, no?
I do appreciate the response,
-heath carlisle
On Sep 6, 12:25 am, Aljoša Mohorović <aljosa.mohoro...@gmail.com>
wrote:
os.system() will launch the process and return control, but then I'd
have to write a utility to get the PID and other data about the
process.
I guess I'm looking for a definitive answer that the built-in process
modules cannot launch a process in the background.
Here are my findings:
subprocess.Popen and multiprocessing offer properties and methods
ideal for my task, but won't return to the view until the process
ends.
Daemonizing these will allow the calling script (django, in this case)
to quit, but still won't return from the view.
celery/gettoq/rabbitMQ allows for the creation of a queue, but you
must specify a predefined (as far as I know) number of workers in the
config. This is great for processor-intensive tasks that may only run
for a short period of time, as the worker pool stays the same size,
and processes new requests when there is a worker available. But, in
my case, the processes are not doing much until there is folder
activity, and I need to add worker instances dynamically.
I feel like I'm just missing something.. Launching system processes
must be a common idiom for a web app, no?
I do appreciate the response,
-heath carlisle
On Sep 6, 12:25 am, Aljoša Mohorović <aljosa.mohoro...@gmail.com>
wrote:
kmpm
Sep 6, 2010, 3:05:24 PM9/6/10
to Django users
I have been using Celery some and from what I got I thought the
concurrency/workers in the config controlled how many processes that
would be at maximum started from the worker that checks the queue but
I could be wrong. Another twist on the celery thing is that you could
actually have several machines looking at the same config+queue and
spreading the load among them and if the queue grows to big you could
start up even more servers/machines dealing with the increased load
until things cool down again.
This is a blog entry I found talking about some of the advantages of
Celery. http://ericholscher.com/blog/2010/jun/23/large-problems-django-mostly-solved-delayed-execut/
/kmpm
concurrency/workers in the config controlled how many processes that
would be at maximum started from the worker that checks the queue but
I could be wrong. Another twist on the celery thing is that you could
actually have several machines looking at the same config+queue and
spreading the load among them and if the queue grows to big you could
start up even more servers/machines dealing with the increased load
until things cool down again.
This is a blog entry I found talking about some of the advantages of
Celery. http://ericholscher.com/blog/2010/jun/23/large-problems-django-mostly-solved-delayed-execut/
/kmpm
Heath
Sep 16, 2010, 10:11:23 AM9/16/10
to Django users
My eventual solution was to just rework the way my watcher scripts
execute. I now use celery and rabbitMQ, and let pyInotify send a new
job to any available worker. For my application, 20 or so workers is
fine and this can scale as needed.
I'm still looking for a definitive answer on whether the built-in
process
modules cannot launch and detach a process in the background, (ala
"nohup <process> <args> &" ) and return control to the view.
Thanks for the responses.
-h
On Sep 6, 12:05 pm, kmpm <pe...@birchroad.net> wrote:
> I have been using Celery some and from what I got I thought the
> concurrency/workers in the config controlled how many processes that
> would be at maximum started from the worker that checks the queue but
> I could be wrong. Another twist on the celery thing is that you could
> actually have several machines looking at the same config+queue and
> spreading the load among them and if the queue grows to big you could
> start up even more servers/machines dealing with the increased load
> until things cool down again.
> This is a blog entry I found talking about some of the advantages of
> Celery. http://ericholscher.com/blog/2010/jun/23/large-problems-django-mostly...
execute. I now use celery and rabbitMQ, and let pyInotify send a new
job to any available worker. For my application, 20 or so workers is
fine and this can scale as needed.
I'm still looking for a definitive answer on whether the built-in
process
modules cannot launch and detach a process in the background, (ala
"nohup <process> <args> &" ) and return control to the view.
Thanks for the responses.
-h
On Sep 6, 12:05 pm, kmpm <pe...@birchroad.net> wrote:
> I have been using Celery some and from what I got I thought the
> concurrency/workers in the config controlled how many processes that
> would be at maximum started from the worker that checks the queue but
> I could be wrong. Another twist on the celery thing is that you could
> actually have several machines looking at the same config+queue and
> spreading the load among them and if the queue grows to big you could
> start up even more servers/machines dealing with the increased load
> until things cool down again.
> This is a blog entry I found talking about some of the advantages of
>
> /kmpm
>
> On Sep 6, 7:20 pm, Heath <he...@brownbarn.com> wrote:
>
> > Thanks! yes,
>
> > os.system() will launch the process and return control, but then I'd
> > have to write a utility to get the PID and other data about the
> > process.
>
> > I guess I'm looking for a definitive answer that the built-in process
> > modules cannot launch a process in thebackground.
>
> > Here are my findings:
>
> > subprocess.Popen and multiprocessing offer properties and methods
> > ideal for my task, but won't return to the view until the process
> > ends.
> > Daemonizing these will allow the calling script (django, in this case)
> > to quit, but still won't return from the view.
>
> > celery/gettoq/rabbitMQ allows for the creation of a queue, but you
> > must specify a predefined (as far as I know) number of workers in the
> > config. This is great for processor-intensive tasks that may only run
> > for a short period of time, as the worker pool stays the same size,
> > and processes new requests when there is a worker available. But, in
> > my case, the processes are not doing much until there is folder
> > activity, and I need to add worker instances dynamically.
>
> > I feel like I'm just missing something..Launchingsystem processes
> /kmpm
>
> On Sep 6, 7:20 pm, Heath <he...@brownbarn.com> wrote:
>
> > Thanks! yes,
>
> > os.system() will launch the process and return control, but then I'd
> > have to write a utility to get the PID and other data about the
> > process.
>
> > I guess I'm looking for a definitive answer that the built-in process
> > modules cannot launch a process in thebackground.
>
> > Here are my findings:
>
> > subprocess.Popen and multiprocessing offer properties and methods
> > ideal for my task, but won't return to the view until the process
> > ends.
> > Daemonizing these will allow the calling script (django, in this case)
> > to quit, but still won't return from the view.
>
> > celery/gettoq/rabbitMQ allows for the creation of a queue, but you
> > must specify a predefined (as far as I know) number of workers in the
> > config. This is great for processor-intensive tasks that may only run
> > for a short period of time, as the worker pool stays the same size,
> > and processes new requests when there is a worker available. But, in
> > my case, the processes are not doing much until there is folder
> > activity, and I need to add worker instances dynamically.
>
Brian Bouterse
Sep 16, 2010, 10:39:29 AM9/16/10
to django...@googlegroups.com
Yes, python multiprocessing can spawn background threads. Just spawn the thread and don't join() it. You could also call things linux things like nohup through the python process module.
--
Brian Bouterse
ITng Services
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
--
Brian Bouterse
ITng Services
Paul Winkler
Sep 16, 2010, 10:54:39 AM9/16/10
to Django users
On Sep 16, 10:11 am, Heath <he...@brownbarn.com> wrote:
> My eventual solution was to just rework the way my watcher scripts
> execute. I now use celery and rabbitMQ, and let pyInotify send a new
> job to any available worker. For my application, 20 or so workers is
> fine and this can scale as needed.
>
> I'm still looking for a definitive answer on whether the built-in
> process
> modules cannot launch and detach a process in the background, (ala
> "nohup <process> <args> &" ) and return control to the view.
Sure they can. Check the docs for the subprocess module,
> My eventual solution was to just rework the way my watcher scripts
> execute. I now use celery and rabbitMQ, and let pyInotify send a new
> job to any available worker. For my application, 20 or so workers is
> fine and this can scale as needed.
>
> I'm still looking for a definitive answer on whether the built-in
> process
> modules cannot launch and detach a process in the background, (ala
> "nohup <process> <args> &" ) and return control to the view.
specifically the P_NOWAIT example.
- PW
Reply all
Reply to author
Forward
0 new messages