Watching output and put back in background

[andrea crotti]

In one process I need to run many quite weird shell commands that take a
long time, possibly in parallel.

Now the problem is that I need to catpure the initial standard error of
the command and then I can go on with the second one, but how can I do
that?

It should be something like

Popen(long_command..)

- first line output
- second line
- interesting line

.. more things ..

But if I just do

out, err = proc.communicate()

it will hang there until the end, right?

Maybe a solution would be to redirect the stderr to file and watch that
instead..

Or otherwise I could use a thread for each shell command, but I would
like to avoid head-aches with possible race-conditions..

Ideas?

[Nobody]

On Thu, 18 Oct 2012 14:05:58 +0100, andrea crotti wrote:

> Maybe a solution would be to redirect the stderr to file and watch that
> instead..
>
> Or otherwise I could use a thread for each shell command, but I would like
> to avoid head-aches with possible race-conditions..

If you're running multiple concurrent commands, and you have redirected
their output streams to pipes, something needs to keep reading those pipes
if you don't want the commands to hang.

Rather than having a separate thread for each process, you could have a
single thread which manages all "background" processes using select(),
poll() or non-blocking I/O, but that's easier to do on Unix than on
Windows (Popen.communicate() uses a pair of threads on Windows).

Redirecting output to files then reading them upon completion is the
simplest solution, but you can't easily monitor progress that way (there's
no easy way to get notification when more output is written).