python script hangs when run from subprocess

[Larry....@gmail.com]

I have a python script and when I run it directly from the command line it runs to completion. But I need to run it from another script. I do that like this:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
rv = p.wait()
out_buf = p.stdout.read()

When I do this, wait never returns. If I trace the underlying script it's always in the same write to stderr that never seems to complete:

write(2, "/KA22/05Feb12/Images/12063LBO003"..., 24457

I run many other scripts and commands in the same manner, and they all complete, it's just this one. Anyone have any ideas why this is happening, and how I can further debug or fix this?

TIA!
-larry

[Peter Otten]

The script writes to an OS buffer, and when that buffer is full it blocks
forever. p.wait() in turn then waits forever for the script to terminate...

As a fix try

out_buf = subprocess.Popen(...).communicate()[0]


This uses threads or select (depending on the OS) to avoid the problem --
and is prominently mentionend in the documentation:

http://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait

[Nobody]

On Sat, 07 Sep 2013 03:55:02 -0700, Larry....@gmail.com wrote:

> I have a python script and when I run it directly from the command line
> it runs to completion. But I need to run it from another script. I do
> that like this:
>
> p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
> rv = p.wait()
> out_buf = p.stdout.read()
>
> When I do this, wait never returns.

The last two statements are the wrong way around. If you're reading a
process' output via a pipe, you shouldn't wait() for it until it has
closed its end of the pipe.

As it stands, you have a potential deadlock. If the subprocess tries to
write more data than will fit into the pipe, it will block until the
parent reads from the pipe. But the parent won't read from the pipe until
after the subprocess has terminated, which won't happen because the
subprocess is blocked waiting for the parent to read from the pipe ...

[Larry....@gmail.com]

Thanks. I hadn't seen this. I'll check it out.

[Larry....@gmail.com]

Thanks. I reversed the order of the wait and read calls, and it no longer hangs.