Working around buffering issues when writing to pipes
sven _
I'm trying to find a solution to <URL:http://bugs.python.org/issue1241>. In
short: unless specifically told not to, normal C stdio will use full output
buffering when connected to a pipe. It will use default (typically
unbuffered) output when connected to a tty/pty.
This is why subprocess.Popen() won't work with the following program when
stdout and stderr are pipes:
#include <stdio.h>
#include <unistd.h>
int main()
{
int i;
for(i = 0; i < 5; i++)
{
printf("stdout ding %d\n", i);
fprintf(stderr, "stderr ding %d\n", i);
sleep(1);
}
return 0;
}
Then
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
is read using polling, but the pipes don't return any data until the
program is done. As expected, specifying a bufsize of 0 or 1 in the Popen()
call has no effect. See end of mail for example Python scripts.
Unfortunately, changing the child program to flush stdout/err or specifying
setvbuf(stdout,0,_IONBF,0); is an undesired workaround in this case.
I went with pexpect and it works well, except that I must handle stdout and
stderr separately. There seems to be no way to do this with pexpect, or am
I mistaken?
Are there alternative ways of solving this? Perhaps some way of
I'm on Linux, and portability to other platforms is not a requirement.
s
#Test script using subprocess:
import subprocess
cmd = '/tmp/slow' # The C program shown above
print 'Starting...'
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
while True:
lineout = proc.stdout.readline()
lineerr = proc.stderr.readline()
exitcode = proc.poll()
if (not lineout and not lineerr) and exitcode is not None: break
if not lineout == '': print lineout.strip()
if not lineerr == '': print 'ERR: ' + lineerr.strip()
print 'Done.'
#Test script using pexpect, merging stdout and stderr:
import sys
import pexpect
cmd = '/home/sveniu/dev/logfetch/bin/slow'
print 'Starting...'
child = pexpect.spawn(cmd, timeout=100, maxread=1, logfile=sys.stdout)
try:
while True: child.expect('\n')
except pexpect.EOF: pass
print 'Done.'
Mark Wooding
> In short: unless specifically told not to, normal C stdio will use
> full output buffering when connected to a pipe. It will use default
> (typically unbuffered) output when connected to a tty/pty.
Wrong. Standard output to a terminal is typically line-buffered.
(Standard error is never buffered by default, per the ISO C standard.)
> This is why subprocess.Popen() won't work with the following program
> when stdout and stderr are pipes:
Yes, obviously.
> I went with pexpect and it works well, except that I must handle
> stdout and stderr separately. There seems to be no way to do this with
> pexpect, or am I mistaken?
You could do it by changing the command you pass to pexpect.spawn so
that it redirects its stderr to (say) a pipe. Doing this is messy,
though -- you'd probably need to set the pipe's write-end fd in an
environment variable or something.
It's probably better to use os.pipe and pty.fork.
As mentioned above, the inferior process's output will still be
line-buffered unless it does something special to change this.
-- [mdw]