python Popen stderr=PIPE 6000+ lines causes hang

[speedyrazor]

I am running a command line application via Popen in OSX which produces many
lines of command line readout, passing this via stderr to my wx.TextCtrl
window. I am using threads and classes and everything has been working fine,
until I run something which produces thousands of lines, at which point my
application runs good until it hangs at line 6498 and i'm not sure why, any
ideas please?

self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)
while True:
line = self.process1.stderr.readline().decode('utf-8')
wx.CallAfter(self.frame.running_log1.AppendText, line)
self.process1.stderr.flush()
if "Some text" in line:
break



--
View this message in context: http://wxpython-users.1045709.n5.nabble.com/python-Popen-stderr-PIPE-6000-lines-causes-hang-tp5721433.html
Sent from the wxPython-users mailing list archive at Nabble.com.

[Steve Barnes]

Sounds like a buffer overrun.

[Nathan McCorkle]

I'd call flush() before calling readline.

[Nathan McCorkle]

also, try adding to Popen the bufsize arg:

https://docs.python.org/2/library/subprocess.html

"

bufsize, if given, has the same meaning as the corresponding argument to the built-in open() function: 0 means unbuffered, 1 means line buffered, any other positive value means use a buffer of (approximately) that size. A negative bufsize means to use the system default, which usually means fully buffered. The default value for bufsize is 0 (unbuffered).

Note

 

If you experience performance issues, it is recommended that you try to enable buffering by setting bufsize to either -1 or a large enough positive value (such as 4096).

"

On second thought, I'm not sure you calling flush on that will have any meaning, as this end of the PIPE is the receiving end, while I believe the process generating the output controls flushing.

So maybe try commenting out the flush, and play with bufsize.

On Saturday, June 28, 2014 12:32:52 AM UTC-7, kevjwells wrote:

[Anders J. Munch]

speedyrazor wrote:
>
> self.process1 = Popen(shlex.split(command), shell=False, stderr=PIPE)

The process might be stuck on a blocking write to stdout. You may
need to create a stdout pipe as well, and read from that in a separate
thread.

regards, Anders