[Python 2.4/2.5] subprocess module is sorely deficient?
Harishankar
Sorry to start off on a negative note in the list, but I feel that the Python
subprocess module is sorely deficient because it lacks a mechanism to:
1. Create non-blocking pipes which can be read in a separate thread (I am
currently writing a mencoder GUI in Tkinter and need a full fledged process
handler to control the command line and to display the progress in a
text-box)
2. Kill the subprocess in a platform independent manner (i.e. no third party
modules and no hacks).
Is there any way to use non-blocking Popen objects using subprocess? and 2 -
is there a way to kill the subprocess in a platform independent manner in a
purely Pythonic way? I thought initially that this problem is simple enough,
but over the last couple of days I've been really struggling to find any
answer. I've been through dozens of mailing list archives in to find a
solution. Unfortunately none of the solutions seem to fit my needs.
My only solution seems to be to offer the end user the mencoder command line
and make them execute it manually and be done with it but that seems a rather
weak solution.
--
Regards,
V. Harishankar
Paul Boddie
>
> Is there any way to use non-blocking Popen objects using subprocess? and 2 -
> is there a way to kill the subprocess in a platform independent manner in a
> purely Pythonic way? I thought initially that this problem is simple enough,
> but over the last couple of days I've been really struggling to find any
> answer. I've been through dozens of mailing list archives in to find a
> solution. Unfortunately none of the solutions seem to fit my needs.
If you want some hints about using subprocesses with non-blocking I/O,
you might find some in my jailtools and pprocess projects:
http://www.python.org/pypi/jailtools
http://www.python.org/pypi/pprocess
Although these projects involve things which are not exactly cross-
platform, the communications mechanisms should be portable, perhaps
with a bit of effort (since I don't recall whether the poll library
function is available on Windows, so you might have to use the select
function instead). It can be awkward sustaining non-blocking
communications with processes if they use buffered I/O, and the only
way I could make Python-based subprocesses work in jailtools was to
invoke them with the unbuffered option (-u).
> My only solution seems to be to offer the end user the mencoder command line
> and make them execute it manually and be done with it but that seems a rather
> weak solution.
The subprocess module may be an improvement over the popen2 module and
various os module functions, but it's still rather arcane.
Paul
Harishankar
> On 22 Apr, 12:52, Harishankar <v.harishan...@gmail.com> wrote:
> > Is there any way to use non-blocking Popen objects using subprocess? and
> > 2 - is there a way to kill the subprocess in a platform independent
> > manner in a purely Pythonic way? I thought initially that this problem is
> > simple enough, but over the last couple of days I've been really
> > struggling to find any answer. I've been through dozens of mailing list
> > archives in to find a solution. Unfortunately none of the solutions seem
> > to fit my needs.
>
> If you want some hints about using subprocesses with non-blocking I/O,
> you might find some in my jailtools and pprocess projects:
>
> http://www.python.org/pypi/jailtools
> http://www.python.org/pypi/pprocess
>
should be built-in to Python in the future.
> Although these projects involve things which are not exactly cross-
> platform, the communications mechanisms should be portable, perhaps
> with a bit of effort (since I don't recall whether the poll library
> function is available on Windows, so you might have to use the select
> function instead). It can be awkward sustaining non-blocking
> communications with processes if they use buffered I/O, and the only
> way I could make Python-based subprocesses work in jailtools was to
> invoke them with the unbuffered option (-u).
>
> > My only solution seems to be to offer the end user the mencoder command
> > line and make them execute it manually and be done with it but that seems
> > a rather weak solution.
>
> The subprocess module may be an improvement over the popen2 module and
> various os module functions, but it's still rather arcane.
Yes. I am quite sure there must be an elegant solution to the subprocess
handling/management. Problem is I've been at this for three days and I'm
getting quite bleary eyed trying to pore through a lot of documentation ;-)
Nicola Musatti
> Hi,
>
> Sorry to start off on a negative note in the list, but I feel that the Python
> subprocess module is sorely deficient because it lacks a mechanism to:
>
> 1. Create non-blocking pipes which can be read in a separate thread (I am
> currently writing a mencoder GUI in Tkinter and need a full fledged process
> handler to control the command line and to display the progress in a
> text-box)
I suggest you check out this: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
Cheers,
Nicola Musatti
Nick Craig-Wood
> Sorry to start off on a negative note in the list, but I feel that the Python
> subprocess module is sorely deficient because it lacks a mechanism to:
>
> 1. Create non-blocking pipes which can be read in a separate thread (I am
> currently writing a mencoder GUI in Tkinter and need a full fledged process
> handler to control the command line and to display the progress in a
> text-box)
>
> 2. Kill the subprocess in a platform independent manner (i.e. no third party
> modules and no hacks).
You are correct on both of those points. Subprocess isn't for
interacting with subprocesses - this should be written in large
letters in the help!
> Is there any way to use non-blocking Popen objects using
> subprocess?
There is a recipe in the cookbook
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
Which I've used and it works.
you can also (if on unix) use
http://www.noah.org/wiki/Pexpect
I think the best solution would be to port Pexpect to windows which
wouldn't be that difficult according to my reading of the code. If
only I had more free time!
> and 2 - is there a way to kill the subprocess in a platform
> independent manner in a purely Pythonic way? I thought initially
> that this problem is simple enough, but over the last couple of
> days I've been really struggling to find any answer. I've been
> through dozens of mailing list archives in to find a
> solution. Unfortunately none of the solutions seem to fit my needs.
No...
This is the best I came up with to add to the subprocess recipe above
import os
from subprocess import *
from subprocess import mswindows
from time import sleep
if mswindows:
import win32api
else:
import signal
class PopenNB(Popen):
# - see cookbook recipe for rest of stuff
# ...
def kill(self, killpg=False):
"""
Kill the running process
"""
pid = self.pid
if mswindows:
# Kill the process using win32api and pid - ignore errors
try:
PROCESS_TERMINATE = 1
handle = win32api.OpenProcess(PROCESS_TERMINATE, False, pid)
win32api.TerminateProcess(handle, -1)
win32api.CloseHandle(handle)
except pywintypes.error:
pass
else:
# Kill the process by sending the pid / process group a
signal
if killpg:
try:
pgid = os.getpgid(pid)
except OSError:
killpg = False
try:
if killpg:
os.killpg(pgid, signal.SIGTERM)
else:
os.kill(pid, signal.SIGTERM)
except OSError:
return
sleep(1.0)
try:
if killpg:
os.killpg(pgid, signal.SIGKILL)
else:
os.kill(pid, signal.SIGKILL)
except OSError:
return
--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick
Harishankar
> There is a recipe in the cookbook
>
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
>
> Which I've used and it works.
similar without having to use the module win32api, but looks like that's not
the case.
>
> you can also (if on unix) use
>
> http://www.noah.org/wiki/Pexpect
>
I'm on Linux. Debian. Pexpect would do the job fine too. The only thing is
it's a third party module and so would reduce the portability of my
application. But never mind. I guess I have to make a compromise one way or
the other.
Thanks for this bit of code. It should probably be adaptible to my needs.
By the way, the win32api seems to be a nonstandard module (i.e. not present in
the main distribution).
Harishankar
> I suggest you check out this:
> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
>
> Cheers,
> Nicola Musatti
> http://mail.python.org/mailman/listinfo/python-list
An interesting solution. Thanks a lot for the link.
Tim Golden
> On Tuesday 22 Apr 2008 18:00:02 Nick Craig-Wood wrote:
>>
>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
>>
>> Which I've used and it works.
Well if you want to, you can reproduce the same effect by using ctypes
which *is* in the standard library. But why reinvent the wheel?
> By the way, the win32api seems to be a nonstandard module (i.e. not present in
> the main distribution).
Correct. It's part of the pywin32 extensions, one of many useful packages available
to the discerning Python programmer who doesn't feel in some way bound to
whatever comes bundled with the standard library.
TJG
Harishankar
> Well if you want to, you can reproduce the same effect by using ctypes
> which *is* in the standard library. But why reinvent the wheel?
The reason is once again, rightly or wrongly I feel that using non-standard
extensions could make it:
1. Difficult to distribute the application as I am not able to package the
third-party extension with distutils.
2. Difficult to predict its behaviour with future versions of Python.
> Correct. It's part of the pywin32 extensions, one of many useful packages
> available to the discerning Python programmer who doesn't feel in some way
> bound to whatever comes bundled with the standard library.
>
> TJG
I wouldn't feel "bound" if I restricted the program to myself. But if I want
to distribute it (as I intend to) I have to think of others as well.
A.T.Hofkamp
> Hi,
>
> Sorry to start off on a negative note in the list, but I feel that the Python
> subprocess module is sorely deficient because it lacks a mechanism to:
>
> 1. Create non-blocking pipes which can be read in a separate thread (I am
I don't know about threading, but you can read/write streams in a non-blocking
way in Python (under Linux) with the os.read/os.write functions (these map
directly to read(2)/write(2) primitives). You should first check that something
can be read/written. Use eg select.select() for this. Several GUI toolkits also
allow monitoring of file handles.
Whether this also works at other OSes, I don't know.
Alternatively, you can create your own pipes, make them non-blocking, and give
the file handles to subprocess.
> 2. Kill the subprocess in a platform independent manner (i.e. no third party
> modules and no hacks).
The concept of sub-process is platform dependent already.
Usually however, closing the child input stream is sufficient for most child
programs to decide that they can quit.
> Is there any way to use non-blocking Popen objects using subprocess? and 2 -
> is there a way to kill the subprocess in a platform independent manner in a
> purely Pythonic way? I thought initially that this problem is simple enough,
> but over the last couple of days I've been really struggling to find any
> answer. I've been through dozens of mailing list archives in to find a
> solution. Unfortunately none of the solutions seem to fit my needs.
Interfacing with the rest of the world implies you are going to need services
provided by the OS at one time or another. Python is rather transparent here
and gives you quick access to the OS services.
While this is bad for uniformity, it is good to let people make their own
choices in optimally using the OS services. Python helps here by giving
light-weight access to the underlying OS, making explicit what part is OS
dependent.
(and if still in doubt, why do you think were all the solutions you found 'not
fitting'?).
> My only solution seems to be to offer the end user the mencoder command line
> and make them execute it manually and be done with it but that seems a rather
> weak solution.
This is always a good fallback.
Sincerely,
Albert
Mark Wooding
> Harishankar <v.hari...@gmail.com> wrote:
>> 1. Create non-blocking pipes which can be read in a separate thread
>
> You are correct on both of those points.
I must be missing something. What's wrong with spawning the subprocess
with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/
whatever, making your end nonblocking with fcntl.fcntl and then using
os.read/os.write in the obvious ways?
-- [mdw]
sturlamolden
> Sorry to start off on a negative note in the list, but I feel that the Python
> subprocess module is sorely deficient because it lacks a mechanism to
Have you looked at the processing module in cheese shop?
Nick Craig-Wood
Nothing apart from the fact it doesn't work on windows. The buffering
will cause you grief too. If you want to do this properly under unix
use pexpect not subprocess.
http://www.noah.org/wiki/Pexpect
Proper non blocking IO is an absolute nightmare under Windows in my
experience! It really isn't the Windows way so you are fighting the
system the whole time.
Christian Heimes
> 2. Kill the subprocess in a platform independent manner (i.e. no third party
> modules and no hacks).
I've added the feature to the Popen class a few days ago. The new
methods are kill(), terminate() and send_signal(sig). On Windows all
methods just fall back to _subprocess.TerminateProcess. On POSIX OS
os.kill() is used.
The code also works on Python 2.4 and 2.5 but I can't add new features
to maintainence branches.
Christian
Christian Heimes
> Nothing apart from the fact it doesn't work on windows. The buffering
> will cause you grief too. If you want to do this properly under unix
> use pexpect not subprocess.
>
> http://www.noah.org/wiki/Pexpect
>
> Proper non blocking IO is an absolute nightmare under Windows in my
> experience! It really isn't the Windows way so you are fighting the
> system the whole time.
Nick is correct. The subproces tries to work around the issues with
threads. But it's no more than an ugly workaround fir Windows' short
comings on async file IO. It's a shame Windows implements the select()
syscall in wsock32 and limits its usage to sockets.
By the way I'm willing to dedicate some time to help enhancing the
subprocess. Everybody is invited to submit patches and I'll review and
check them into the trunk and py3k ASAP. Any help is appreciated:
enhancements for async IO, doc updates, more examples ...
Christian
Python core developer
"Martin v. Löwis"
> modules and no hacks).
What's wrong with the .terminate method of the Popen object?
Regards,
Martin
Christian Heimes
>> 2. Kill the subprocess in a platform independent manner (i.e. no third party
>> modules and no hacks).
>
> What's wrong with the .terminate method of the Popen object?
It's brand new ;)
Christian
Christian Heimes
>> 2. Kill the subprocess in a platform independent manner (i.e. no third party
>> modules and no hacks).
>
> What's wrong with the .terminate method of the Popen object?
It's brand new ;)
Christian
Mike Hansen
> wouldn't be that difficult according to my reading of the code. If
> only I had more free time!
Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to
interface with all sorts of other systems. We recently received
funding from Microsoft to do a native port of Sage (and all of its
components to Windows. Part of this will most likely be a port of
pexpect to Windows.
--Mike
Heikki Toivonen
> Sorry to start off on a negative note in the list, but I feel that the Python
> subprocess module is sorely deficient because it lacks a mechanism to:
At OSAF we used a slightly modified killableprocess module with a
wrapper to deal with complexities of various redirections in
cross-platform way. I actually blogged about this a week ago so rather
than rehash the issues I'll point you to the article which contains
links to all the pieces we used:
http://www.heikkitoivonen.net/blog/2008/04/16/pythons-ossystem-considered-harmful/
--
Heikki Toivonen
Harishankar
> Nick Craig-Wood schrieb:
> > Nothing apart from the fact it doesn't work on windows. The buffering
> > will cause you grief too. If you want to do this properly under unix
> > use pexpect not subprocess.
> >
> > http://www.noah.org/wiki/Pexpect
> >
> > Proper non blocking IO is an absolute nightmare under Windows in my
> > experience! It really isn't the Windows way so you are fighting the
> > system the whole time.
>
> threads. But it's no more than an ugly workaround fir Windows' short
> comings on async file IO. It's a shame Windows implements the select()
> syscall in wsock32 and limits its usage to sockets.
>
> By the way I'm willing to dedicate some time to help enhancing the
> subprocess. Everybody is invited to submit patches and I'll review and
> check them into the trunk and py3k ASAP. Any help is appreciated:
> enhancements for async IO, doc updates, more examples ...
>
> Christian
> Python core developer
Thanks a lot to everybody who's been following this discussion. Very
interesting indeed.
I'm currently thinking of working around this problem by actually opening a
new terminal window and running the command from there, thus allowing the
user full control over the process.
Is there any platform independent way to launch a terminal window from a
desktop (Windows, Linux, etc.)?
Harishankar
> At OSAF we used a slightly modified killableprocess module with a
> wrapper to deal with complexities of various redirections in
> cross-platform way. I actually blogged about this a week ago so rather
> than rehash the issues I'll point you to the article which contains
> links to all the pieces we used:
>
> http://www.heikkitoivonen.net/blog/2008/04/16/pythons-ossystem-considered-h
>armful/
your website in my searches. I just wanted to be absolutely sure that it
worked because you had mentioned that it has some drawbacks.
Christian Heimes
> Is there any platform independent way to launch a terminal window from a
> desktop (Windows, Linux, etc.)?
No, there isn't. It usually not possible to create a graphical terminal
window on a remote server.
Christian
Nick Craig-Wood
Hooray!
Harishankar
Ah, well, since my application is a desktop tool and it requires a GUI I'm
doing something like this:
However, I have to then force the user to use xterm (which is a popular/common
X Terminal)
if (sys.platform.startswith ('win'):
# launch the windows cmd.exe with the command
...
else:
# warn the user that xterm is required and then launch xterm
...
Harishankar
> I don't know about all Linux distros, but my Ubuntu machine (8.04 Beta),
> has the 'TERM' (xterm) and 'COLORTERM' (gnome-terminal) keys in os.environ.
> You might be able to use that to ensure that the terminal is installed, but
> you should probably look at a couple of other popular distros first to make
> sure that the key is there.
This is set on Debian too. Thanks. I should be able to use this environment
variable on most Linux distributions, I suspect.
Paul Boddie
> On Wednesday 23 Apr 2008 15:11:21 Ben Kaplan wrote:
> > I don't know about all Linux distros, but my Ubuntu machine (8.04 Beta),
> > has the 'TERM' (xterm) and 'COLORTERM' (gnome-terminal) keys in os.environ.
> > You might be able to use that to ensure that the terminal is installed, but
> > you should probably look at a couple of other popular distros first to make
> > sure that the key is there.
>
> This is set on Debian too. Thanks. I should be able to use this environment
> variable on most Linux distributions, I suspect.
Here on RHEL 4, COLORTERM has an empty value. I suppose that the
freedesktop.org standards should cover this kind of thing, and there
are some scripts out there which attempt to provide cross-desktop
support on Free Software desktops:
http://portland.freedesktop.org/wiki/XdgUtils
Similarly, the desktop module should provide support for various
desktop features, but opening command line windows or terminals isn't
yet possible:
http://www.python.org/pypi/desktop
It'd be interesting to know if we could find out (or make up) reliable
ways of opening the user's preferred terminal application.
Paul
Mark Wooding
> On Wednesday 23 Apr 2008 15:11:21 Ben Kaplan wrote:
>> I don't know about all Linux distros, but my Ubuntu machine (8.04 Beta),
>> has the 'TERM' (xterm) and 'COLORTERM' (gnome-terminal) keys in os.environ.
> variable on most Linux distributions, I suspect.
No! The TERM variable is not the name of a terminal emulation program!
It's a terminal name intended to be looked up in the termcap and/or
terminfo databases. For example, I use pterm as my terminal, but it
(correctly) sets TERM=xterm.
I'd suggest trying to run (in order, until one actually works):
$X_TERMINAL_EMULATOR, sensible-x-terminal-emulator, x-terminal-emulator,
xterm.
-- [mdw]