line-by-line output from a subprocess

157 views
Skip to first unread message

Chermside, Michael

unread,
May 23, 2005, 1:06:15 PM5/23/05
to pytho...@python.org
I am using the subprocess module to invoke a command-line utility and
process the output.

However, I would like to process the output line-by-line as it is
generated rather than
running the subprocess to completion and THEN processing the results.
So, for instance,
I'd like to write code like this:

all_files = []
import subprocess
subp = subprocess.Popen('dir', stdout=subprocess.PIPE)
for line in subp.stdout:
sys.stdout.print(line)
all_files.append(line)

...and have it print the lines to stdout one-by-one until the entire
list is
collected. (Pretend that 'dir' is VERY slow.)

The communicate() method in subprocess blocks until the subprocess has
exited... I'm sure there is some good reason for this behavior. But how
does
one achieve what I am trying to do?

(PS: this is on Windows... although I'd rather use an OS-agnostic
solution if
one exists.)

-- Michael Chermside


*****************************************************************************
This email may contain confidential or privileged information. If you believe
you have received the message in error, please notify the sender and delete
the message without copying or disclosing it.
*****************************************************************************

Simon Percivall

unread,
May 23, 2005, 4:22:04 PM5/23/05
to
Okay, so the reason what you're trying to do doesn't work is that the
readahead buffer used by the file iterator is 8192 bytes, which clearly
might be too much. It also might be because the output from the
application you're running is buffered, so you might have to do
something about that as well.

Anyway, if the output from the child application is unbuffered, writing
a generator like this would work:

def iterread(fobj):
stdout = fobj.stdout.read(1) # or what you like
data = ""
while stdout:
data += stdout
while "\n" in data:
line, data = data.split("\n", 1)
yield line
stdout = fobj.stdout.read(1)
if data:
yield data,

Jp Calderone

unread,
May 23, 2005, 4:58:00 PM5/23/05
to pytho...@python.org

Or, doing the same thing, but with less code:

def iterread(fobj):
return iter(fobj.readline, '')

Haven't tried this on subprocess's pipes, but I assume they behave much the same way other file objects do (at least in this regard).

Jp

Simon Percivall

unread,
May 23, 2005, 9:02:28 PM5/23/05
to
Jp Calderone wrote:
> Or, doing the same thing, but with less code:

Hmm ... What have I been smoking?

Reply all
Reply to author
Forward
0 new messages