Direct interaction with subprocess - the curse of blocking I/O
Pascal Chambon
I've had real issues with subprocesses recently : from a python script,
on windows, I wanted to "give control" to a command line utility, i.e
forward user in put to it and display its output on console. It seems
simple, but I ran into walls :
- subprocess.communicate() only deals with a forecast input, not
step-by-step user interaction
- pexpect module is unix-only, and for automation, not interactive input
- when wanting to do all the job manually (transfering data between the
standard streams of the python program and the binary subprocess, I met
the issue : select() works only on windows, and python's I/O are
blocking, so I can't just, for example, get data from the subprocess'
stdout and expect the function to return if no input is present - the
requesting thread might instead block forever.
Browsing the web, I found some hints :
- use the advanced win32 api to create non-blocking I/O : rather
complicated, non portable and far from the python normal files
- use threads that block on the different streams and eat/feed them
without ever stopping : rather portable, but gives problems on shutdown
(How to terminate these threads without danger ? On some OSes, a process
never dies as long as any thread - even "daemonic" - lives, I've seen
people complaining about it).
So well, I'd like to know, do you people know any solution to this
simple problem - making a user interact directly with a subprocess ? Or
would this really require a library handling each case separately (win32
api, select().....) ?
Thanks a lot for your interest and advice,
regards,
Pascal
Nobody
> I've had real issues with subprocesses recently : from a python script,
> on windows, I wanted to "give control" to a command line utility, i.e
> forward user in put to it and display its output on console.
Are you talking about a popen(..., 'w') situation? I.e. where Python
feeds data to the child's stdin but the child's stdout doesn't go through
Python?
Or a slave process, where both stdin and stdout/stderr are piped to/from
Python?
The latter is inherently tricky (which is why C's popen() lets you connect
to stdin or stdout but not both). You have to use either multiple threads,
select/poll, or non-blocking I/O.
If the child's output is to the console, it should presumably be the
former, i.e. piping stdin but allowing the child to inherit stdout, in
which case, where's the problem? Or are you piping its stdout via Python
for the hell of it?
Scott David Daniels
> I've had real issues with subprocesses recently : from a python script,
> on windows, I wanted to "give control" to a command line utility, i.e
> Browsing the web, I found some hints :
> - use the advanced win32 api to create non-blocking I/O : rather
>
> So well, I'd like to know, do you people know any solution to this
> simple problem - making a user interact directly with a subprocess ? Or
> would this really require a library handling each case separately (win32
> api, select().....) ?
I would guess the architectural differences are so great that an attempt
to "do something simple" is going to involve architecture-specific code.
and I personally wouldn't have it any other way. Simulating a shell
with hooks on its I/O should be so complicated that a "script kiddie"
has trouble writing a Trojan.
--Scott David Daniels
Scott....@Acm.Org
Lawrence D'Oliveiro
Chambon wrote:
> I met the issue : select() works only on windows ...
No it doesn't. It works only on sockets on Windows, on Unix/Linux it works
with all file descriptors <http://docs.python.org/library/select.html>.
ryles
> and I personally wouldn't have it any other way. Simulating a shell
> with hooks on its I/O should be so complicated that a "script kiddie"
> has trouble writing a Trojan.
+1 QOTW
Lawrence D'Oliveiro
c1a6-45b7-978...@f33g2000vbm.googlegroups.com>, ryles wrote:
Trouble is, script kiddies, by definition, don't need to write anything.
They just need to download something somebody else has already written.
--
Lawrence "I counter your +1 with my -1" D'Oliveiro
spillz
> Hello everyone
>
> I've had real issues with subprocesses recently : from a python script,
> on windows, I wanted to "give control" to a command line utility, i.e
> forward user in put to it and display its output on console. It seems
> simple, but I ran into walls :
If you are willing to have a wxPython dependency, wx.Execute handles
non-blocking i/o with processes on all supported platforms
more discussion of python's blocking i/o issues here:
http://groups.google.com/group/comp.lang.python/browse_thread/thread/a037349e18f99630/60886b8beb55cfbc?q=#60886b8beb55cfbc
yam850
> On Jun 29, 3:15 pm, Pascal Chambon <chambon.pas...@gmail.com> wrote:
> > I've had real issues with subprocesses recently : from a python script,
> > on windows, I wanted to "give control" to a command line utility, i.e
> > forward user in put to it and display its output on console. It seems
> > simple, but I ran into walls :
> If you are willing to have a wxPython dependency, wx.Execute handles
I made a python method/function for non blocking read from a file
object.
I use it in one of my python programs.
When looking at the code bear in mind that I am not an expert and I am
happy to see comments.
#------------------------------------------------------------------
def non_blocking_readline(f_read=sys.stdin, timeout_select=0.0):
"""to readline non blocking from the file object 'f_read'
for 'timeout_select' see module 'select'"""
import select
text_lines = '' # empty string
while True: # as long as there are bytes
to read
try: # try select
rlist, wlist, xlist = select.select([f_read], [], [],
timeout_select)
except: # select ERROR
print >>sys.stderr, ("non_blocking_read select ERROR")
break
if DEBUG: print("rlist=%s, wlist=%s, xlist=%s" % (repr(rlist),
repr(wlist), repr(xlist)))
if len(rlist) > 0:
text_read = f_read.readline() # get a line
if DEBUG: print("after read/readline text_read:'%s', len=
%s" % (text_read, repr(len(text_read))))
if len(text_read) > 0: # there were some bytes
text_lines = "".join([text_lines, text_read])
if DEBUG: print("text_lines:'%s'" % (text_lines))
else:
break # there was no byte in a
line
else:
break # there was no byte in the
f_read
if text_lines == '':
return None
else:
return text_lines
--
Kurt
Scott David Daniels
> I made a python method/function for non blocking read from a file
OK, here's a fairly careful set of comments with a few caveats:
Does this work on windows? The top comment should say where you
know it works. Does this code correctly read a single line?
perhaps you need to check for a final '\n' whenever you actually
get characters, and break out there. The rest below simply is
style-changing suggestions; take what you like and leave the rest.
>
> def non_blocking_readline(f_read=sys.stdin, timeout_select=0.0):
> """to readline non blocking from the file object 'f_read'
> for 'timeout_select' see module 'select'"""
> import select
Typically put imports at the top of the source
XXX text_lines = '' # empty string
Either no comment here or say _why_ it is empty.
(A) text_lines = [] # for result accumulation
if collecting pieces (see below)
(B) text_lines = '' # for result accumulation
if collecting a single string
> while True: # as long as there are bytes to read
> try: # try select
> rlist, wlist, xlist = select.select([f_read], [], [],
> timeout_select)
XXX except: # select ERROR
XXX print >>sys.stderr, ("non_blocking_read select ERROR")
I'd not hide the details of the exception like that. Don't do empty
excepts. Either don't catch it at all here, (I prefer that) or do
something like the following to capture the error:
except Exception, why:
print >>sys.stderr, "non_blocking_read select: %s" % why
> break
XXX if DEBUG: print("rlist=%s, wlist=%s, xlist=%s" % (repr(rlist),
XXX repr(wlist), repr(xlist)))
Don't be scared of vertical space; if you must, define a function DPRINT
to ignore args; use %r to get repr
So, either:
if DEBUG:
print("rlist=%r, wlist=%r, xlist=%r" % (
rlist, wlist, xlist))
or:
elsewhere:
def DPRINT(format_str, *args, **kwargs):
'''Conditionally print formatted output based on DEBUG'''
if DEBUG:
print(format_str % (args or kwargs))
and then here:
DPRINT("rlist=%r, wlist=%r, xlist=%r", rlist, wlist, xlist)
XXX if len(rlist) > 0:
Idiomatic Python: use the fact that empty containers evaluate false:
if rlist:
> text_read = f_read.readline() # get a line
XXX if DEBUG: print("after read/readline text_read:'%s', len=
> %s" % (text_read, repr(len(text_read))))
Similar comment to above:
if DEBUG:
print("after read/readline text_read:%r, len=%s" % (
text_read, len(text_read))
or:
DPRINT("after read/readline text_read:%r, len=%s",
text_read, len(text_read))
XXX if len(text_read) > 0: # there were some bytes
if text_read: # there were some bytes
XXX text_lines = "".join([text_lines, text_read])
Note the join is a good idea only if you have several elements.
For a single concatenation, "=" is just fine. The (A) case combines
at the end, the (B) case if you expect multiple concatenates are rare.
(A) text_lines.append(text_read)
(B) text_lines += text_read
XXX if DEBUG: print("text_lines:'%s'" % (text_lines))
Similar to above
if DEBUG:
print("text_lines:%r" % text_lines)
or:
DPRINT("text_lines:%r", text_lines)
XXX else:
XXX break # there was no byte in a line
XXX else:
XXX break # there was no byte in the f_read
To make the flow of control above more clear (one case continues, others
get out), I'd also replace the above with:
continue # In one case we keep going
break # No new chars found, let's get out.
XXX if text_lines == '':
XXX return None
XXX else:
XXX return text_lines
Simplify using 'or's semantics:
(A) return ''.join(text_lines) or None
(B) return text_lines or None
So, with everything applied:
import select
def DPRINT(format_str, *args, **kwargs):
'''Conditionally print formatted output based on DEBUG'''
if DEBUG:
print(format_str % (args or kwargs))
def non_blocking_readline(f_read=sys.stdin, timeout_select=0.0):
"""to readline non blocking from the file object 'f_read'
for 'timeout_select' see module 'select'
"""
text_lines = '' # for result accumulation
while True: # as long as there are bytes to read
rlist, wlist, xlist = select.select([f_read], [], [],
timeout_select)
DPRINT("rlist=%r, wlist=%r, xlist=%r",
rlist, wlist, xlist)
if rlist:
text_read = f_read.readline() # get a line
DPRINT("after read/readline text_read:%r, len=%s",
text_read, len(text_read))
if text_read: # there were some bytes
text_lines += text_read
DPRINT("text_lines:%r", text_lines)
continue # Got some chars, keep going.
break # Nothing new found, let's get out.
return text_lines or None
--Scott David Daniels
Scott....@Acm.Org
yam850
> yam850 wrote:
> > object.... I am happy to see comments.
> OK, here's a fairly careful set of comments with a few caveats:
> --Scott David Daniels
> Scott.Dani...@Acm.Org
Wow, thats a GREAT answer!!!
Thanks!!!
I will learn a lot while "digesting" this mail.
G
--
yam850