Documentation: the BackgroundManager class

[Edward K. Ream]

The BackgroundManager class is something I have vaguely dreamed about for  decades. It handles all details of running separate processes in the background without blocking Leo.

Previous efforts were feeble.  They merely provided a thin (useless!) wrapper around subprocess.Popen.

The BackgroundManager is much more capable.  It manages a queue of background processes, and runs them one after another in the background.  It registers a handler with the IdleTimeManager that checks to see if the presently running background process has completed.  If so, it writes its output to the log and starts another background process in the queue.

Because only one background process from the queue is ever running, the background processes can to write their output to Leo's log without the output becoming intermixed.

You can kill all processes by calling BM.kill().  That's what the 'kill-pylint' command does.  You can add processes to the queue at any time.  For example, you can rerun the 'pylint' command while a background process is running.

The BackgroundManager is completely safe: all of its code runs in the main process.

Improvements

Ah, the joys of documentation.  As I write this, I realize that it would be better to use a singleton BackgroundManager, say g.app.backgroundManager.  This would allow multiple kinds of background processes to coexist peacefully without intermixed output.

The following relatively minor changes are needed:

1. BM.start_process already can handle process corresponding to different commands passed to subprocess.Popen.  However, the queue must be upgraded so it knows the type of each queued process. That is, a 'kind' argument should be added to BM.start_process.

2. Add a corresponding 'kind' argument to BM.kill.  This would kill the queued processes of the given type, or all queued processes if the 'kind' is None or 'all'.

Another possible improvement: optionally run shlex on the command argument passed to BM.start_process. 

At present, the 'pylint' command creates a string the hard, hard way.  That probably won't change, but it might be possible to simplify command creation using shlex.  The jury is still out on this one.

Summary

For the first time, there is an easy way of running processes safely in the background.

Relatively easy changes will allow background processes of different kinds to be run sequentially.

This removes another source of Emacs envy ;-)

Edward

[Edward K. Ream]

On Friday, October 28, 2016 at 5:38:13 AM UTC-5, Edward K. Ream wrote:

The BackgroundManager class...handles all details of running separate processes in the background without blocking Leo.

At present, if you run pylint on several files and then switch to another outline tab, the output will switch to the new log pane!  Try it.

Instead, output should go to the log pane from which the commands were originated.  It will be easy to do this. The new queue will remember the outline from which the command was invoked.

EKR

[Edward K. Ream]

On Friday, October 28, 2016 at 5:38:13 AM UTC-5, Edward K. Ream wrote:

 

...it would be better to use a singleton BackgroundManager, say g.app.backgroundManager.  This would allow multiple kinds of background processes to coexist peacefully without intermixed output.

Rev d52a103 completes all the improvements discussed today:

- Renamed BackgroundManager to BackgroundProcessManager.
- Created singleton BPM instance, g.app.backgroundProcessManager.
- The BPM sends output to the originating log pane if possible.
- The 'pylint' and 'pylint-kill' commands now use the BPM.

Note that arbitrarily many other instances of the BPM class could reasonably coexist with the processes managed by the "singleton" g.app.backgroundProcessManager, provided that the other instances ran processes that did not produce any output.  So, arbitrarily many processes could run in the background, provided that the only one process at a time produces output.

Edward

[Edward K. Ream]

On Friday, October 28, 2016 at 7:12:45 AM UTC-5, Edward K. Ream wrote:

> Rev d52a103 completes all the improvements discussed today:

Rev b1bd2b4 ensures that message always go to the originating pane.  This was a bit tricky. I finally saw that the BPM must use a new put_log method rather than g.es, g.trace or g.es_print.

Everything looks good for now.  I am proud of this code.  It does an important job simply.

EKR

[Edward K. Ream]

On Friday, October 28, 2016 at 7:12:45 AM UTC-5, Edward K. Ream wrote:

Arbitrarily many other instances of the BPM class could reasonably coexist with the processes managed by the "singleton" g.app.backgroundProcessManager, provided that the other instances ran processes that did not produce any output.

Oops.  This is misleading. To run processes without output, just call subprocess.Popen as many times as you like. There is no reason to instantiate more BPM instances.

EKR

[Edward K. Ream]

On Friday, October 28, 2016 at 7:12:45 AM UTC-5, Edward K. Ream wrote:

While writing documentation for the scripting miscellany, I realized that BPM.start_process should have a shell keyword argument, with default=True.  Recent revs do this.

EKR