Is there an easy way to capture stdout/stderr for a shell command and pipe it to the log pane?

[Jacob Peck]

Hi all,

I have an outline with some @button nodes that fire off external shell commands using g.execute_shell_commands.  I tend to run Leo without a console window open, so anything those @button-triggered shell commands print to the console is lost to me.  Is there an easy way/straightforward way to redirect to the stdout/stderr for those commands only to the log pane?  I'm struggling to figure it out by manipulating sys.stdout and sys.stderr, though I admit this is far from my area of specialty.

An example of a button:

```
@language python
@tabwidth -2

''' Executes the currently selected sketch with the 'vsk run' command.
'''

import os.path

fn = c.getNodeFileName(p)
if len(fn) == 0:
  # non-external file selected
  g.es('Not a valid sketch node.', color='red')

else:
  sketch_path = os.path.dirname(fn)
  g.execute_shell_commands(f'&vsk run {sketch_path}')
```

That command is long-running (i.e. it stays open in another pane, thus the '&' to unblock the Leo GUI).  Sometimes it indicates errors on the console output, which just plain doesn't exist when I'm running Leo without a console window open.

Ideally any solution would be cross-platform, since I do this on both Windows and Linux... but really I'd be happy with platform-specific solutions too.  I'd really rather not have to have a console window open to see these errors if at all possible.

Thanks in advance for any ideas and suggestions,

Jake

[Thomas Passin]

I don't know the answer but I know a thing or two that might be useful.  By starting the command with "&", as you have, you avoid the proc.communicate() call, which would block Leo until it returns.  However, if you try to capture output with a pipe, the pipe can block if it fills up, and that happens with a surprisingly small number of bytes.  Then you block because the pipe is waiting to be cleared, and you are probably waiting to get output from the pipe.

So that won't work.  

The way I managed this with the background process manager (BPM) - it's what launches pylint, mypy, etc - is to use a separate thread to launch the process, and call proc.communicate() in that thread.  The thread blocks waiting for the process to complete, but that doesn't matter since it doesn't block Leo itself.  This leaves the question of how to get the results.  The thread puts the results into a variable in the BPM, and the BPM registers a callback at idle time.  The callback checks to see if the variable is not None. If the BPM sees that there is a result in that variable, it cancels the callback and makes use of the result.  Then it writes the messages to the log pane.  You have to use a lock to make sure that the external process's results are put atomically into the variable. In your case I might have it write the results to a different tab to avoid getting mixed up with regular log output. I'm going by memory here and might have a detail or two inaccurate.

So study the BPM and see if you could do something similar with your code.  The BPM makes sure that it only ever runs one process at a time, BTW. That avoids the possibility of getting intermingled results or other confusions.

Another possibility is to have your external process redirect output to a known file.  You could check periodically to see if that file has gotten new data and read the new data if so.  The problem there might be to know when the external process is finished writing data.  You could also hget a race condition between the writing process and the reading process, so you would have to figure out how to handle that situation.  I don't happen to know how that works and it might be different between Windows and Linux.

Another possibility would be to launch your external program from a batch/shell file and have the batch/shell file redirect the output to a known file.  When the programs finishes, the batch file could set an environmental variable.  You could do the idle-time callback trick to check for that variable, or you could have the checks done periodically by a separate thread.  That thread could even read the results, format them, and write them to a new tab or pass them on to another batch file.

I would say the last possibility sounds the most promising to me.

[Thomas Passin]

I see that I slightly lost track of the fact that you want to look at ongoing error messages.  As I said, the big problem that I am aware of is that the message pipe fills up and then blocks quickly.  That suggests that a thread or other process should read it frequently to allow new data to be inserted.  I do remember that there is some discussion on line - not necessarily Python-specific - to be found.