spawning pyhon apps...
bruce
toying with an idea.. trying to figure out a good/best way to spawn multiple
python scripts from a parent python app. i'm trying to figure out how to
determine when all child apps have completed, or to possibly determine if
any of the child processes have died/halted..
parent app
spawn child1
spawn child2
spawn child3
.
.
.
spawn childn
do i iterate through a os.waitpid(pid) for each pid of the child processes i
create?
is there another approach? code samples/tutorial...??
i've seen various approaches via google, but not just what i'm looking for..
thanks
Chris Rebert
> hi...
>
> toying with an idea.. trying to figure out a good/best way to spawn multiple
> python scripts from a parent python app. i'm trying to figure out how to
> determine when all child apps have completed, or to possibly determine if
> any of the child processes have died/halted..
>
> parent app
> spawn child1
> spawn child2
> spawn child3
> .
> .
> .
> spawn childn
>
> do i iterate through a os.waitpid(pid) for each pid of the child processes i
> create?
>
> is there another approach? code samples/tutorial...??
Use the `subprocess` module's
(http://docs.python.org/library/subprocess.html) Popen class to spawn
the processes.
Then use .wait() or .poll(), and .returncode on the Popen instances to
check their exit status.
Cheers,
Chris
--
Follow the path of the Iguana...
http://rebertia.com
Jason Scheirer
Investigate the subprocess module, you probably want Popen objects.
You can do a poll loop like
my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
%i.txt"%x) for x in xrange(10)]
while any(popenobject.statuscode is None for popenobject in
my_popenobjects):
time.sleep(0.25)
If your tasks are more like function calls and less like shell
scripts, then investigate writing your python as an importable module
and use the multiprocessing module, which will do threading/
subprocessing for you.
bruce
forgive me... but in your sample:
my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
%i.txt"%x) for x in xrange(10)]
are you spawning 'foo.py' 10 times? that can't be right!
so just what is "foo.py" used for? what am i missing...
it looks like the my_popenobjects array is iterated through to check the
statuscode. is the statuscode the value that would be returned from a child
python script via something like "return(2)"....
i've seen mention of os.waitpid(..) does this play into waiting for child
processes to complete, or determine if they've terminated??
thanks
Chris Rebert
> hi jason....
>
> forgive me... but in your sample:
> %i.txt"%x) for x in xrange(10)]
Indeed, it probably ought to be (note the 2nd pair of brackets):
my_popenobjects = [subprocess.Popen(["foo.py",
"--filename=file%i.txt"%x]) for x in xrange(10)]
> so just what is "foo.py" used for? what am i missing...
It's the name of the program you want to run. In this case, it happens
to be a Python script.
>
> it looks like the my_popenobjects array is iterated through to check the
> statuscode. is the statuscode the value that would be returned from a child
> python script via something like "return(2)"....
It's the POSIX exit code of the program (i.e. what int you return from
main() in a C program, or what you pass to sys.exit() in Python)
> i've seen mention of os.waitpid(..) does this play into waiting for child
> processes to complete, or determine if they've terminated??
Don't know specifically, but that's another, lower-level API. The
`subprocess` module is superior.
Jason Scheirer
> hi jason....
>
> forgive me... but in your sample:
> my_popenobjects = [subprocess.Popen("foo.py", "--filename=file
> %i.txt"%x) for x in xrange(10)]
> are you spawning 'foo.py' 10 times? that can't be right!
> so just what is "foo.py" used for? what am i missing...
>
> it looks like the my_popenobjects array is iterated through to check the
> statuscode. is the statuscode the value that would be returned from a child
> python script via something like "return(2)"....
>
> i've seen mention of os.waitpid(..) does this play into waiting for child
> processes to complete, or determine if they've terminated??
>
> thanks
>
> -----Original Message-----
>
> [mailto:python-list-bounces+bedouglas=earthlink....@python.org]On Behalf
> Of Jason Scheirer
> Sent: Friday, January 09, 2009 3:19 PM
> my_popenobjects):
> time.sleep(0.25)
>
> If your tasks are more like function calls and less like shell
> scripts, then investigate writing your python as an importable module
> and use the multiprocessing module, which will do threading/
> subprocessing for you.
> --http://mail.python.org/mailman/listinfo/python-list
>
>
Correction: statuscode is wrong. It's returncode.
Foo.py is the hypothetical Python script you want to run in a
subprocess. In this example, I have an external shell script named
foo.py, and I am indeed spawning 10 copies of it, each with a second
argument that varies (foo.py --filename=file0.txt, foo.py --
filename=file1.txt, ... foo.py --filename=file9.txt). You don't need
os.waitpid() with a Popen object, there is a Popen.wait() method you
can call which will accomplish the exact same thing. I'm polling the
Popen.returncode for each process' return code (which is the numeric
code a process returns when it finishes, like sys.exit(x) or return,
or gives None if it's not done yet. What this sample is doing is
opening 10 copies of the script and running them in parallel, if you
want to run it is serial then you can do a simple for loop and .wait()
on each:
for cmd in ('a', 'b', 'c'):
sp = subprocess.Popen(cmd)
sp.wait()
print "Command %r completed with status %i" % (cmd, sp.returncode)
I'm still not 100% sure what you're trying to accomplish. What is the
exact problem you are wishing to solve?
bruce
or i could also, simply iterate through a loop of the names of the "child
processes" i want to spawn, using the names in the subprocess.popen, and
then proceeding with the rest of your example..
question though... if i have a child app that's hanging.. how do i kill it.
or is this getting into the aspect of using interprocess pipes, where if the
parent isn't getting a 'live ping' via the pipe back from the child after a
certain amount of time... it could kill the child...
thoughts/comments...
thanks
Jason Scheirer
> thanks jason....
>
> or i could also, simply iterate through a loop of the names of the "child
> processes" i want to spawn, using the names in the subprocess.popen, and
> then proceeding with the rest of your example..
>
> question though... if i have a child app that's hanging.. how do i kill it.
>
> or is this getting into the aspect of using interprocess pipes, where if the
> parent isn't getting a 'live ping' via the pipe back from the child after a
> certain amount of time... it could kill the child...
>
> thoughts/comments...
>
> thanks
>
> -----Original Message-----
> From: python-list-bounces+bedouglas=earthlink....@python.org
>
> [mailto:python-list-bounces+bedouglas=earthlink....@python.org]On Behalf
> Of Jason Scheirer
Yes, so to open your processes you can loop over a list of commands
and create new subprocess.Popen(cmd) objects for each.
Everything is explained:
http://docs.python.org/library/subprocess.html#popen-objects
You can do .terminate() to kill a process that may be hanging, you can
get the .stdout and poll on its .read() to see if it's still putting
anything out to the console.
Mark Wooding
> toying with an idea.. trying to figure out a good/best way to spawn
> multiple python scripts from a parent python app. i'm trying to figure
> out how to determine when all child apps have completed, or to
> possibly determine if any of the child processes have died/halted..
You don't say what platform you're using, but you mention os.waitpid so
I'll randomly assume it's Unix-like.
> do i iterate through a os.waitpid(pid) for each pid of the child
> processes i create?
That will technically work, and give you the information you wanted,
though not necessarily in the most timely fashion. It'll block on each
process in turn, waiting for its exit status -- so it'll finish as soon
as all the children are dead, but if (say) the fifth process dies first,
you won't find out until the first four have also passed on.
If you don't have anything better for your program to do, and you're
really on Unix, you can call
kid, status = os.waitpid(0, 0)
to wait for something to happen to any of your process's children; the
kid is the process-id of the child being reported and the status is what
happened to it.
If you do have other things for your process to be doing, then your best
bet is to establish a signal handler for SIGCHLD and installing a
handler of the form
import signal as S
import os as OS
import errno as E
## Children sometimes die. It's sad.
def sigchld(sig, frame):
try:
while True:
kid, status = OS.waitpid(0, OS.WNOHANG)
if kid == 0:
break
## Handle death of KID here.
except OSError, err:
if err.errno != E.ECHILD:
raise
### ...
## Establish handler.
S.signal(S.SIGCHLD, sigchld)
should work.
If you're running on Windows then these tricks won't work. As a grim
hack, you could start a thread per child process and have each thread
wait for its own child (sending the exit status through a queue or
something). I'm afraid I don't know Windows well enough to offer
slicker solutions; maybe someone else can help.
-- [mdw]