Sorry to start off on a negative note in the list, but I feel that the Python subprocess module is sorely deficient because it lacks a mechanism to:
1. Create non-blocking pipes which can be read in a separate thread (I am currently writing a mencoder GUI in Tkinter and need a full fledged process handler to control the command line and to display the progress in a text-box)
2. Kill the subprocess in a platform independent manner (i.e. no third party modules and no hacks).
Is there any way to use non-blocking Popen objects using subprocess? and 2 - is there a way to kill the subprocess in a platform independent manner in a purely Pythonic way? I thought initially that this problem is simple enough, but over the last couple of days I've been really struggling to find any answer. I've been through dozens of mailing list archives in to find a solution. Unfortunately none of the solutions seem to fit my needs.
My only solution seems to be to offer the end user the mencoder command line and make them execute it manually and be done with it but that seems a rather weak solution.
On 22 Apr, 12:52, Harishankar <v.harishan...@gmail.com> wrote:
> Is there any way to use non-blocking Popen objects using subprocess? and 2 - > is there a way to kill the subprocess in a platform independent manner in a > purely Pythonic way? I thought initially that this problem is simple enough, > but over the last couple of days I've been really struggling to find any > answer. I've been through dozens of mailing list archives in to find a > solution. Unfortunately none of the solutions seem to fit my needs.
If you want some hints about using subprocesses with non-blocking I/O, you might find some in my jailtools and pprocess projects:
Although these projects involve things which are not exactly cross- platform, the communications mechanisms should be portable, perhaps with a bit of effort (since I don't recall whether the poll library function is available on Windows, so you might have to use the select function instead). It can be awkward sustaining non-blocking communications with processes if they use buffered I/O, and the only way I could make Python-based subprocesses work in jailtools was to invoke them with the unbuffered option (-u).
> My only solution seems to be to offer the end user the mencoder command line > and make them execute it manually and be done with it but that seems a rather > weak solution.
The subprocess module may be an improvement over the popen2 module and various os module functions, but it's still rather arcane.
On Tuesday 22 Apr 2008 17:06:26 Paul Boddie wrote:
> On 22 Apr, 12:52, Harishankar <v.harishan...@gmail.com> wrote: > > Is there any way to use non-blocking Popen objects using subprocess? and > > 2 - is there a way to kill the subprocess in a platform independent > > manner in a purely Pythonic way? I thought initially that this problem is > > simple enough, but over the last couple of days I've been really > > struggling to find any answer. I've been through dozens of mailing list > > archives in to find a solution. Unfortunately none of the solutions seem > > to fit my needs.
> If you want some hints about using subprocesses with non-blocking I/O, > you might find some in my jailtools and pprocess projects:
Thank you. I will take a look at those. Actually I feel a mechanism like this should be built-in to Python in the future.
> Although these projects involve things which are not exactly cross- > platform, the communications mechanisms should be portable, perhaps > with a bit of effort (since I don't recall whether the poll library > function is available on Windows, so you might have to use the select > function instead). It can be awkward sustaining non-blocking > communications with processes if they use buffered I/O, and the only > way I could make Python-based subprocesses work in jailtools was to > invoke them with the unbuffered option (-u).
> > My only solution seems to be to offer the end user the mencoder command > > line and make them execute it manually and be done with it but that seems > > a rather weak solution.
> The subprocess module may be an improvement over the popen2 module and > various os module functions, but it's still rather arcane.
Yes. I am quite sure there must be an elegant solution to the subprocess handling/management. Problem is I've been at this for three days and I'm getting quite bleary eyed trying to pore through a lot of documentation ;-)
On Apr 22, 12:52 pm, Harishankar <v.harishan...@gmail.com> wrote:
> Hi,
> Sorry to start off on a negative note in the list, but I feel that the Python > subprocess module is sorely deficient because it lacks a mechanism to:
> 1. Create non-blocking pipes which can be read in a separate thread (I am > currently writing a mencoder GUI in Tkinter and need a full fledged process > handler to control the command line and to display the progress in a > text-box)
Harishankar <v.harishan...@gmail.com> wrote: > Sorry to start off on a negative note in the list, but I feel that the Python > subprocess module is sorely deficient because it lacks a mechanism to:
> 1. Create non-blocking pipes which can be read in a separate thread (I am > currently writing a mencoder GUI in Tkinter and need a full fledged process > handler to control the command line and to display the progress in a > text-box)
> 2. Kill the subprocess in a platform independent manner (i.e. no third party > modules and no hacks).
You are correct on both of those points. Subprocess isn't for interacting with subprocesses - this should be written in large letters in the help!
> Is there any way to use non-blocking Popen objects using > subprocess?
I think the best solution would be to port Pexpect to windows which wouldn't be that difficult according to my reading of the code. If only I had more free time!
> and 2 - is there a way to kill the subprocess in a platform > independent manner in a purely Pythonic way? I thought initially > that this problem is simple enough, but over the last couple of > days I've been really struggling to find any answer. I've been > through dozens of mailing list archives in to find a > solution. Unfortunately none of the solutions seem to fit my needs.
No...
This is the best I came up with to add to the subprocess recipe above
import os from subprocess import * from subprocess import mswindows from time import sleep
if mswindows: import win32api else: import signal
class PopenNB(Popen): # - see cookbook recipe for rest of stuff # ... def kill(self, killpg=False): """ Kill the running process """ pid = self.pid if mswindows: # Kill the process using win32api and pid - ignore errors try: PROCESS_TERMINATE = 1 handle = win32api.OpenProcess(PROCESS_TERMINATE, False, pid) win32api.TerminateProcess(handle, -1) win32api.CloseHandle(handle) except pywintypes.error: pass else: # Kill the process by sending the pid / process group a signal if killpg: try: pgid = os.getpgid(pid) except OSError: killpg = False try: if killpg: os.killpg(pgid, signal.SIGTERM) else: os.kill(pid, signal.SIGTERM) except OSError: return sleep(1.0) try: if killpg: os.killpg(pgid, signal.SIGKILL) else: os.kill(pid, signal.SIGKILL) except OSError: return
Thanks. I found that recipe too. I was hoping I could cook up something similar without having to use the module win32api, but looks like that's not the case.
I'm on Linux. Debian. Pexpect would do the job fine too. The only thing is it's a third party module and so would reduce the portability of my application. But never mind. I guess I have to make a compromise one way or the other.
>> Which I've used and it works. > Thanks. I found that recipe too. I was hoping I could cook up something > similar without having to use the module win32api...
Well if you want to, you can reproduce the same effect by using ctypes which *is* in the standard library. But why reinvent the wheel?
> By the way, the win32api seems to be a nonstandard module (i.e. not present in > the main distribution).
Correct. It's part of the pywin32 extensions, one of many useful packages available to the discerning Python programmer who doesn't feel in some way bound to whatever comes bundled with the standard library.
> Well if you want to, you can reproduce the same effect by using ctypes > which *is* in the standard library. But why reinvent the wheel?
The reason is once again, rightly or wrongly I feel that using non-standard extensions could make it:
1. Difficult to distribute the application as I am not able to package the third-party extension with distutils. 2. Difficult to predict its behaviour with future versions of Python.
> Correct. It's part of the pywin32 extensions, one of many useful packages > available to the discerning Python programmer who doesn't feel in some way > bound to whatever comes bundled with the standard library.
> TJG
I wouldn't feel "bound" if I restricted the program to myself. But if I want to distribute it (as I intend to) I have to think of others as well.
On 2008-04-22, Harishankar <v.harishan...@gmail.com> wrote:
> Hi,
> Sorry to start off on a negative note in the list, but I feel that the Python > subprocess module is sorely deficient because it lacks a mechanism to:
> 1. Create non-blocking pipes which can be read in a separate thread (I am
I don't know about threading, but you can read/write streams in a non-blocking way in Python (under Linux) with the os.read/os.write functions (these map directly to read(2)/write(2) primitives). You should first check that something can be read/written. Use eg select.select() for this. Several GUI toolkits also allow monitoring of file handles.
Whether this also works at other OSes, I don't know.
Alternatively, you can create your own pipes, make them non-blocking, and give the file handles to subprocess.
> 2. Kill the subprocess in a platform independent manner (i.e. no third party > modules and no hacks).
The concept of sub-process is platform dependent already.
Usually however, closing the child input stream is sufficient for most child programs to decide that they can quit.
> Is there any way to use non-blocking Popen objects using subprocess? and 2 - > is there a way to kill the subprocess in a platform independent manner in a > purely Pythonic way? I thought initially that this problem is simple enough, > but over the last couple of days I've been really struggling to find any > answer. I've been through dozens of mailing list archives in to find a > solution. Unfortunately none of the solutions seem to fit my needs.
Interfacing with the rest of the world implies you are going to need services provided by the OS at one time or another. Python is rather transparent here and gives you quick access to the OS services.
While this is bad for uniformity, it is good to let people make their own choices in optimally using the OS services. Python helps here by giving light-weight access to the underlying OS, making explicit what part is OS dependent.
(and if still in doubt, why do you think were all the solutions you found 'not fitting'?).
> My only solution seems to be to offer the end user the mencoder command line > and make them execute it manually and be done with it but that seems a rather > weak solution.
Nick Craig-Wood <n...@craig-wood.com> wrote: > Harishankar <v.harishan...@gmail.com> wrote: >> 1. Create non-blocking pipes which can be read in a separate thread >> [...]
> You are correct on both of those points.
I must be missing something. What's wrong with spawning the subprocess with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/ whatever, making your end nonblocking with fcntl.fcntl and then using os.read/os.write in the obvious ways?
On Apr 22, 12:52 pm, Harishankar <v.harishan...@gmail.com> wrote:
> Sorry to start off on a negative note in the list, but I feel that the Python > subprocess module is sorely deficient because it lacks a mechanism to
Have you looked at the processing module in cheese shop?
Mark Wooding <m...@distorted.org.uk> wrote: > Nick Craig-Wood <n...@craig-wood.com> wrote: > > Harishankar <v.harishan...@gmail.com> wrote: > >> 1. Create non-blocking pipes which can be read in a separate thread > >> [...]
> > You are correct on both of those points.
> I must be missing something. What's wrong with spawning the subprocess > with subprocess.Popen as usual, passing subprocess.PIPE as stdin/stdout/ > whatever, making your end nonblocking with fcntl.fcntl and then using > os.read/os.write in the obvious ways?
Nothing apart from the fact it doesn't work on windows. The buffering will cause you grief too. If you want to do this properly under unix use pexpect not subprocess.
Proper non blocking IO is an absolute nightmare under Windows in my experience! It really isn't the Windows way so you are fighting the system the whole time.
> 2. Kill the subprocess in a platform independent manner (i.e. no third party > modules and no hacks).
I've added the feature to the Popen class a few days ago. The new methods are kill(), terminate() and send_signal(sig). On Windows all methods just fall back to _subprocess.TerminateProcess. On POSIX OS os.kill() is used.
The code also works on Python 2.4 and 2.5 but I can't add new features to maintainence branches.
> Nothing apart from the fact it doesn't work on windows. The buffering > will cause you grief too. If you want to do this properly under unix > use pexpect not subprocess.
> Proper non blocking IO is an absolute nightmare under Windows in my > experience! It really isn't the Windows way so you are fighting the > system the whole time.
Nick is correct. The subproces tries to work around the issues with threads. But it's no more than an ugly workaround fir Windows' short comings on async file IO. It's a shame Windows implements the select() syscall in wsock32 and limits its usage to sockets.
By the way I'm willing to dedicate some time to help enhancing the subprocess. Everybody is invited to submit patches and I'll review and check them into the trunk and py3k ASAP. Any help is appreciated: enhancements for async IO, doc updates, more examples ...
> I think the best solution would be to port Pexpect to windows which > wouldn't be that difficult according to my reading of the code. If > only I had more free time!
Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to interface with all sorts of other systems. We recently received funding from Microsoft to do a native port of Sage (and all of its components to Windows. Part of this will most likely be a port of pexpect to Windows.
Harishankar wrote: > Sorry to start off on a negative note in the list, but I feel that the Python > subprocess module is sorely deficient because it lacks a mechanism to:
At OSAF we used a slightly modified killableprocess module with a wrapper to deal with complexities of various redirections in cross-platform way. I actually blogged about this a week ago so rather than rehash the issues I'll point you to the article which contains links to all the pieces we used:
> Nick Craig-Wood schrieb: > > Nothing apart from the fact it doesn't work on windows. The buffering > > will cause you grief too. If you want to do this properly under unix > > use pexpect not subprocess.
> > Proper non blocking IO is an absolute nightmare under Windows in my > > experience! It really isn't the Windows way so you are fighting the > > system the whole time.
> Nick is correct. The subproces tries to work around the issues with > threads. But it's no more than an ugly workaround fir Windows' short > comings on async file IO. It's a shame Windows implements the select() > syscall in wsock32 and limits its usage to sockets.
> By the way I'm willing to dedicate some time to help enhancing the > subprocess. Everybody is invited to submit patches and I'll review and > check them into the trunk and py3k ASAP. Any help is appreciated: > enhancements for async IO, doc updates, more examples ...
> Christian > Python core developer
Thanks a lot to everybody who's been following this discussion. Very interesting indeed.
I'm currently thinking of working around this problem by actually opening a new terminal window and running the command from there, thus allowing the user full control over the process.
Is there any platform independent way to launch a terminal window from a desktop (Windows, Linux, etc.)?
On Wednesday 23 Apr 2008 08:27:01 Heikki Toivonen wrote:
> At OSAF we used a slightly modified killableprocess module with a > wrapper to deal with complexities of various redirections in > cross-platform way. I actually blogged about this a week ago so rather > than rehash the issues I'll point you to the article which contains > links to all the pieces we used:
killableprocess.py looks like a good solution indeed. I actually came across your website in my searches. I just wanted to be absolutely sure that it worked because you had mentioned that it has some drawbacks.
Mike Hansen <mhan...@gmail.com> wrote: > > I think the best solution would be to port Pexpect to windows which > > wouldn't be that difficult according to my reading of the code. If > > only I had more free time!
> Sage ( http://www.sagemath.org ) uses pexpect fairly extensively to > interface with all sorts of other systems. We recently received > funding from Microsoft to do a native port of Sage (and all of its > components to Windows. Part of this will most likely be a port of > pexpect to Windows.
On Wednesday 23 Apr 2008 14:46:20 Christian Heimes wrote:
> Harishankar schrieb: > > Is there any platform independent way to launch a terminal window from a > > desktop (Windows, Linux, etc.)?
> No, there isn't. It usually not possible to create a graphical terminal > window on a remote server.
> Christian
Ah, well, since my application is a desktop tool and it requires a GUI I'm doing something like this:
However, I have to then force the user to use xterm (which is a popular/common X Terminal)
if (sys.platform.startswith ('win'): # launch the windows cmd.exe with the command ... else: # warn the user that xterm is required and then launch xterm ...
