read a process output with subprocess.Popen

[Ashok Prabhu]

Hi,

I m trying a read the output of a process which is running
continuously with subprocess.Popen. However the readline() method
hangs for the process to finish. Please let me know if the following
code can be made to work with subprocess.Popen with threads or queues.
I tried a lot of methods but to no avail. It would be great if someone
can make it work.

import subprocess

p1 = subprocess.Popen('tail -f /var/log/
messages',stdout=subprocess.PIPE,shell=True)
p2 = subprocess.Popen('grep
something',stdin=p1.stdout,stdout=subprocess.PIPE,shell=True)

while 1:
line = p2.stdout.readline()
print line

Thanks,
~Ashok.

[Nobody]

On Thu, 04 Feb 2010 04:28:20 -0800, Ashok Prabhu wrote:

> I m trying a read the output of a process which is running
> continuously with subprocess.Popen. However the readline() method
> hangs for the process to finish. Please let me know if the following
> code can be made to work with subprocess.Popen with threads or queues.
> I tried a lot of methods but to no avail. It would be great if someone
> can make it work.

This is an issue with grep, not Python per se.

By default, stdout (in the C library) is line-buffered if it refers to a
TTY, and block-buffered otherwise (e.g. if it refers to a pipe). grep
doesn't change the default buffering, so when it's stdout is a pipe, its
output is written in 4K blocks.

If you only need this to work with GNU grep, you can use the
--line-buffered switch to force stdout to be flushed after each line.

If it needs to be portable, you can implement the grep part in Python
(i.e. read p1.stdout and ignore any lines which don't contain a given
string or which don't match a given regex).