I am using the subprocess module to invoke a command-line utility and process the output.
However, I would like to process the output line-by-line as it is generated rather than running the subprocess to completion and THEN processing the results. So, for instance, I'd like to write code like this:
all_files = [] import subprocess subp = subprocess.Popen('dir', stdout=subprocess.PIPE) for line in subp.stdout: sys.stdout.print(line) all_files.append(line)
...and have it print the lines to stdout one-by-one until the entire list is collected. (Pretend that 'dir' is VERY slow.)
The communicate() method in subprocess blocks until the subprocess has exited... I'm sure there is some good reason for this behavior. But how does one achieve what I am trying to do?
(PS: this is on Windows... although I'd rather use an OS-agnostic solution if one exists.)
-- Michael Chermside
*************************************************************************** ** This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. *************************************************************************** **
Okay, so the reason what you're trying to do doesn't work is that the readahead buffer used by the file iterator is 8192 bytes, which clearly might be too much. It also might be because the output from the application you're running is buffered, so you might have to do something about that as well.
Anyway, if the output from the child application is unbuffered, writing a generator like this would work:
def iterread(fobj): stdout = fobj.stdout.read(1) # or what you like data = "" while stdout: data += stdout while "\n" in data: line, data = data.split("\n", 1) yield line stdout = fobj.stdout.read(1) if data: yield data,
>Okay, so the reason what you're trying to do doesn't work is that the >readahead buffer used by the file iterator is 8192 bytes, which clearly >might be too much. It also might be because the output from the >application you're running is buffered, so you might have to do >something about that as well.
>Anyway, if the output from the child application is unbuffered, writing >a generator like this would work:
>def iterread(fobj): > stdout = fobj.stdout.read(1) # or what you like > data = "" > while stdout: > data += stdout > while "\n" in data: > line, data = data.split("\n", 1) > yield line > stdout = fobj.stdout.read(1) > if data: > yield data,
|
