Status report: using pyzo features in Leo

[Edward K. Ream]

This time around everything things much easier.  There is no need for the "let's pretend it's easy principle", because everything is easy :-)  No more messing with shims or other hacks.  The pyzo_support plugins will likely be retired soon.

The pyzo branch will contain work on #1149: pyzo file browser and #1158: pyzo shells.  It contains a complete copy of all of pyzo's sources in leo/external/pyzo and sub-folders.  I can modify anything at will.  Here are the principles:

Changes to pyzo

Initial work on the pyzo file browser used copies of pyzo code.  That gets old in a hurry.

Instead, we want to import from leo/external/pyzo.  To do this effectively, the key principle is:

    importing pyzo must not execute pyzo.start

It's easy.  pyzo.__main__ never calls pyzo.start. And we don't want to import pyzo/yotonloader.py unless we actually want to start the pyzo shell. And that's about it, aside from other minor tweaks.

Running the code externally

As a quick check, I want to run the code externally, which will actually fire up all of pyzo.  This script does so:

python -c "from leo.external.pyzo import start; start()"

Notice that start must now be called explicitly.  Again, that's all there is to it.

Other details

1. There is no need to worry about menus or menu shims because they won't ever get created.

2. There is no need to worry about pyzo's configuration.  For now, the code will have read-only access.  When not running externally I'll just hack the code so settings never get written.

Summary

All the old problems seem to have disappeared.  I expect that completing will take a matter of days.

With the new infrastructure in place, it may even be worth considering using pyzo's text editor as Leo's body pane.  That would make integration with pyzo's shell easier, and has other advantages.  Otoh, there will be substantial work involved.  All such work will be done in the leo/external/pyzo directory tree.

Edward

[Edward K. Ream]

On Monday, August 5, 2019 at 1:27:12 PM UTC-5, Edward K. Ream wrote:

Summary

All the old problems seem to have disappeared.  I expect that completing will take a matter of days.

Oops, I meant so say, "I expect that completing #1149 will take a matter of days."

Edward

[Edward K. Ream]

On Monday, August 5, 2019 at 1:27:12 PM UTC-5, Edward K. Ream wrote:

This time around everything things much easier.  There is no need for the "let's pretend it's easy principle", because everything is easy :-)

Problems with imports are appearing.  They are similar, but not the same, as other import "projects".

No more messing with shims or other hacks.

Still true.

The pyzo_support plugins will likely be retired soon.

Still true.

Edward

[Edward K. Ream]

On Monday, August 5, 2019 at 1:29:10 PM UTC-5, Edward K. Ream wrote:

Oops, I meant so say, "I expect that completing #1149 will take a matter of days."

Maybe :-)

Edward

[Edward K. Ream]

​​On Mon, Aug 5, 2019 at 1:27 PM Edward K. Ream <edre...@gmail.com> wrote:

> There is no need for the "let's pretend it's easy principle", because everything is easy :-)

Leo's pyzo_file_browser plugin now puts pyzo's standard file browser​ in a Leo dock! This is a big milestone.

You can show/hide this dock by alt-clicking in the title area of any dock.

Warning

​At present the dock widget will crash if try to open any file, because the plugin only inits the required parts of pyzo. If you try to open a file you will be executing pyzo's open-file logic, which, naturally enough, assumes a full init.  I have disabled pyzo's exception handling, so Leo crashes and burns.  Don't play with this plugin unless you can live with this.

Adapting the file browser to Leo

The plugin need only override pyzoFileBrowser.PyzoFileBrowser.  I expect considerable work, but localized and fairly straightforward.

Summary

1. Leo can use almost all of pyzo's code from leo/external/pyzo unchanged.  This is the holy grail.  See the post script for details.

2. When enabled, the plugin creates pyzo's standard file browser in a Leo dock, without starting pyzo. This is, by far, the hardest part of the project.

3. It should be straightforward to integrate the file browser fully into Leo.  Until then, don't enable the plugin unless you are prepared to have Leo quit unexpectedly.

Edward

P.S. Here are the code level details:

A code breakthrough

In pyzo's original code, these lines appear in pyzo/__main__.py

thisDir = os.path.abspath(os.path.dirname(__file__))
sys.path.insert(0, os.path.split(thisDir)[0])

This ensures that all of pyzo's relative imports will work correctly.  The following lines do the same for the plugin:

pyzo_dir = g.os_path_finalize_join(g.app.loadDir, '..', 'external')
sys.path.insert(0, pyzo_dir)

This ensures that all the relative imports in leo/external/plugins work as expected.  This is a huge breakthrough, because it means that Leo can use 99.9% of the code in leo/external/pyzo without changes!

Imports

Even after this breakthrough, several hours of work were needed to init only the necessary parts of pyzo.  Here are the imports in the pyzo_file_browser plugin:

# 0. Crucial.
pyzo_dir = g.os_path_finalize_join(g.app.loadDir, '..', 'external')
assert g.os_path_exists(pyzo_dir), repr(pyzo_dir)
sys.path.insert(0, pyzo_dir)
# 1. Must be first.
import pyzo
assert pyzo
# 2. Must be next.
import pyzo.core.main as main
main.loadIcons()
main.loadFonts()
from pyzo.core import menu
assert menu
# 3. All other imports:
from pyzo.tools.pyzoFileBrowser import PyzoFileBrowser
assert PyzoFileBrowser
# 4. Instantiate tool manager
from pyzo.tools import ToolManager
    # tools/__init__.py defines ToolManager.
pyzo.toolManager = ToolManager()
    # From mainWindow._populate.​

These imports roughly correspond to the several months work I originally spent on pyzo.

There are some strange aspects of the code.  Various imports are required (in the correct order) even though they aren't immediately used.  This is due to side effect of the imports.  The big advance in the present code is that I don't plan to change pyzo's code in any way!

Finally, here is the code that actually creates the browser:

def onCreate(tag, keys):
    """Create a pyzo file browser in c's outline."""
    c = keys.get('c')
    dw = c and c.frame and c.frame.top
    if not dw:
        return
    # Use Leo's main window, not pyzo's main window.
    pyzo.main = dw
    # Load the file browser.
    pyzo.toolManager.loadTool('pyzofilebrowser')

That's all.

Edward 

[Edward K. Ream]

On Wednesday, August 7, 2019 at 10:05:48 AM Edward K. Ream wrote:

> Leo's pyzo_file_browser plugin now puts pyzo's standard file browser​ in a Leo dock!

Recent revs prevent crashes in Leo's pyzo code, both in general and specifically when using the file browser.

Double-clicking a file in the file browser now does nothing.  I'm investigating how to open files in a Leonine manner.

Edward

[Edward K. Ream]

On Monday, August 5, 2019 at 1:27:12 PM UTC-5, Edward K. Ream wrote:

This time around everything things much easier.

This post describes recent progress in the pyzo branch. As usual, the summary contains the gist.

The pyzo_file_browser plugin now can embed all of pyzo's docks (except the Shell and Workspace docks) into Leo.  The pyzo docks are, to various degrees, partially or fully functional.

Enabling docks

LeoSettings.leo now contains the @data pyzo_tool_ids node.  It's contents are:

# Valid ids:

# pyzofilebrowser
# pyzohistoryviewer
# pyzointeractivehelp
# pyzologger
# pyzowebbrowser
# pyzosourcestructure
    # Partially functional.

# Not ready yet.
# pyzoworkspace

Copy this node to myLeoSettings.leo and uncomment the id of the desired pyzo dock.

Only Leo's copy of pyzo is executing

In the pyzo branch, Leo's external/pyzo directory tree contains a copy of pyzo's sources.

Yesterday's big breakthrough is that just a few changes to pyzo's code suffice to make imports "just work".

I've made a few other minor changes to the pyzo sources.  They are all marked with ekr:change in the pyzo sources.  Eventually these changes might be made via monkey-patching, but there is no need to do so at present.

Please read the next two sections carefully.  The present union of Leo and pyzo is confusing.

Startup and configuration

During startup, the plugin recreates most, but not all, of pyzo's startup logic, using the copy of pyzo's source code in Leo.  What happens depends on pyzo's configuration file, config.ssdf.

- If you don't have the "real" pyzo installed, the plugin's startup logic will (I haven't checked) create a standard config.ssdf file for you.

- If you do have the "real" pyzo installed, the plugin's startup logic will use the "real" config.ssdf.

At present, the plugin will update config.ssdf when Leo exits. This will affect the real pyzo, if present.

Pyzo editors are hidden

During startup, Leo will create a hidden pyzo editor for all files mentioned in config.ssdf.  Opening a file (in Leo) using the Pyzo File Browser dock will also create a hidden pyzo editor. All these hidden editors are likely pretty much fully functional, as far as the rest Leo's embedded pyzo code is concerned.  Some examples:

- The Source Structure dock will show the structures of the last file opened.

- On exit, the embedded pyzo code will update config.ssdf. If you then open the "real" pyzo, pyzo will open the files you have opened in Leo.

To do: configuration

It will be easy to suppress the updates of config.ssdf from within Leo.  That makes sense for many reasons.

The embedded version of pyzo will likely continue to read config.ssdf for stuff that Leo's users don't know or care about. Eventually, we'll want to use Leo's own configuration system for all pyzo stuff that Leo's users do care about.

Leo's non-pyzo code already remembers the placement of all docks. That's a feature of Qt's main window code.

To do: other

Progress has been so fast that I'm a bit dizzy.  Here are some preliminary thoughts:

1. Making the Shell and Workspace docks visible seems like a reasonable first step. This will likely complete most startup-related issues. I don't expect serious problems.

2. I'll start making pyzo's configuration more Leonine.  Leo probably should never write config.ssdf.

3. The big question is how to unify/reconcile pyzo's hidden editors with Leo's body pane?  Do we adapt Leo's body classes to pyzo, or adapt pyzo's editor classes to Leo?  It might go either way.  Imo, pyzo's editors are much better than Leo's, but adapting pyzo's editors so they look (and act) like Leo's body pane might be quite a trick.  Either way, I have a general idea of what is required.  I'll leave the details for later.

Summary

The pyzo_file_browser plugin is misnamed.  It will be renamed pyzo_in_leo.py eventually.

Enable pyzo docks using @data pyzo_tool_ids.

The plugin interacts with the real pyzo via config.ssdf.  Don't have the real pyzo open while using this plugin.

The plugin creates hidden pyzo editors during startup.  Opening a file using the Pyzo File Browser dock will also create a hidden pyzo editor.  These hidden editors interact with other pyzo docks.

The big question: how to reconcile pyzo's editor tabs with Leo's body pane?  The entire pyzo_in_leo project depends on this step.  Some relatively clean way will likely emerge. Imo, this project is worth almost any amount of work.

If you feel adventurous, please try out the plugin in the pyzo branch. All questions and comments welcome. 

Edward

[Chris George]

Pyzo docks are floaters.

Pyzo docks do not nest.

Using the X in the titlebar to close a Pyzo dock is a one way trip. The title is removed from the pop-up menu.

Clicking the diamond in a titlebar removes the titlebar of the Pyzo dock widget below it. It also turns off the ability to access the pop-up from the titlebar of the widget whose diamond was clicked.

I will keep playing with it throughout my day.

HTH,

Chris

[Edward K. Ream]

On Thu, Aug 8, 2019 at 11:41 AM Chris George <techn...@gmail.com> wrote:

Pyzo docks are floaters.

Pyzo docks do not nest.

Using the X in the titlebar to close a Pyzo dock is a one way trip. The title is removed from the pop-up menu.

Clicking the diamond in a titlebar removes the titlebar of the Pyzo dock widget below it. It also turns off the ability to access the pop-up from the titlebar of the widget whose diamond was clicked.

I will keep playing with it throughout my day.

Good work.  Thanks for the testing.

Edward

[Edward K. Ream]

On Thu, Aug 8, 2019 at 9:41 AM Edward K. Ream <edre...@gmail.com> wrote:

> Progress has been so fast that I'm a bit dizzy.

And now my subconscious is screaming at me.  This isn't the critic. It would be unwise to tell it to buzz off.

The visual image is that with all the new docks Leo won't be Leo.  The response would seem to be, "all the docks are optional, so what's the problem?"  Some replies:

1. The new pyzo code will bloat Leo and increase the difficulty of maintaining Leo.  Do I really want to saddle my successors with that?  Integrating pyzo with Leo will take real work, and somebody is going to understand that work in detail.

2. A day or two ago I started out just wanting to see if the Pyzo File Browser (pfb) could be integrated into Leo.  Clearly, this should not have to drag huge amounts of pyzo code into Leo.  It turned out to be easy to load all the docks with very little work, so that's where I went.  Perhaps instead I should copy the minimal code needed into the (properly named) pyzo_file_browser plugin.  That should sidestep the big question.

3. There are aspects of pyzo's editors that I admire.  Perhaps I should start another prototype project that would integrate the good parts of pyzo's editors directly into Leo, without actually using much (or any) of pyzo's code.

Summary

I'll probably go ahead and get the remaining docks working, that is visually.  I might as well finish this phase of the project. 

The big question (how to integrate pyzo and Leo) can not be ignored.  Using huge amounts of pyzo code promises difficulties for future devs.  Instead, using just a bit of pyzo code in a few places may make more sense.

[Edward K. Ream]

On Friday, August 9, 2019 at 2:41:39 AM UTC-5, Edward K. Ream wrote:

And now my subconscious is screaming at me...The big question (how to integrate pyzo and Leo) can not be ignored.

After a bit more sleep I see that this project is still experimental, no matter how much apparent progress there has been.

Big questions imply lots of study

The next steps will to focus on a series of smaller questions and projects.

Wrapper functions

Let's compare this project with #990: embed neovim into Leo.  In each case, we want to make the new body pane scriptable.

But that's easy!  Just implement the wrapper api in each editor: @file leoFrame.py-->API classes-->class WrapperAPI, which means that each editor has a "widget" and "wrapper" ivar.  In pyzo's case, self.widget = self for instances of QtWidgets.QWidget.

Repaint

I like the effect of pyzo's repaint code.  Leo doesn't use QWidget.repaint this way, and I think it should. #1278 suggests using a table to control repaint.  This would free Leo from the details of pyzo's code.

Signals

Pyzo uses user-defined signals effectively.  I'll want to study this in detail, and possibly adapt this approach to Leo's legacy code.

Key handling

#1269 suggests using per-pane tables, created when Leo starts up, instead of the present baroque code.  Specific keys might trigger specific repaints, possibly as the result of specific signals.

Show whole files in the body pane

This is how pyzo works. Kent has requested this once or twice.  I have just created #1279 for this.

At present, Leo opens a wrapper outline containing the whole file in an @edit node.  The pyzo way is more intuitive.

Summary

The "pyzo complex" of issues and questions promise to improve Leo.  I'll approach the task at hand incrementally, focusing on small bits of the puzzle.

All the issues presented here are experimental.  I can't make any firm predictions about when, or if, they will happen.

I'll wrap up the present phase of the pyzo_file_browser plugin, and rename it to pyzo_in_leo.py.  After that I'll turn my attention to studying some of the smaller projects mentioned here.

Edward

[Edward K. Ream]

On Friday, August 9, 2019 at 7:08:56 AM UTC-5, Edward K. Ream wrote:

> I'll rename [pyzo_file_browser.py] to pyzo_in_leo.py.

Done at 82e2bd8 in the pyzo branch.  I'm thinking of taking a break from the pyzo projects for awhile while I fix bugs :-)

Edward

[Matt Wilkie]

Show whole files in the body pane

This is how pyzo works. Kent has requested this once or twice.  I have just created #1279 for this.

+1 from me as well.

[Edward K. Ream]

On Monday, August 5, 2019 at 1:27:12 PM UTC-5, Edward K. Ream wrote:

> This time around everything things much easier.

Status

Recent revs in the pyzo branch put the code that used to be in pyzo_file_browser.py into the pyzo_in_leo.py plugin.

The pyzo_file_browser.py now contains a copy of all the classes and functions needed to put pyzo's file browser in Leo.  Except for icons, it is as functional as the code in pyzo_in_leo.py. That is, the code must still be connected to Leo. That will be straightforward because I can change the source code at will. It will also be straightforward to add icons.

I have now fully explored both ends of the usage spectrum: pyzo_in_leo.py imports everything, while pyzo_file_browser.py copies everything.  Both ends of the spectrum are revealing.  Neither, by itself, is a total answer.

Remarks

Imo, the acceleration of progress is a reflection of my increasing expertise with pyzo's code base.  Like most expertise, it's a bit difficult to describe.  Suffice it to say that experience has shown me what to do, what to avoid, and why.

Previously I said that copying classes pyzo_file_browser.py "got old in a hurry".  That was a novice talking :-)  This time around I could see the value in doing so.  Let me explain...

1. Imports:  In some sense imports are a necessary evil.  They allow code to be broken into files, and that's pretty much it.  Yeah, these files define python modules, but in this case that's not important.

Please, don't bother reminding me of the virtues of python modules. The plugin is an experimental prototype, and there are virtues to having everything in one file: all the pyzo-centric imports disappear. This eliminates a lot of blah, blah, blah. That leads to insight.

2. Startup: In the "real" pyzo, startup and imports are intimately linked.  The plugin shows that this need not be so. In fact, the onCreate function now just creates a new dock widget using Leo's existing code, instantiates the PyzoFileBrowser class (defined in the plugin), and calls dock.setWidget to complete the process. That's all.

3. Configuration: The plugin defines a FileBrowserConfig class. I don't think of it as a shim, it's just a necessary part of the code. The config class makes everything explicit, and extending this class will be straightforward. That's big progress.

4. Relationships: This is the big one. The plugin makes absolutely clear which code is (and is not!) needed by the file browser alone.  This is a crucial insight, worth far more than the the few hours it took to get it.  It leads to...

Refactoring pyzo code

Earlier in this post I said, "Both ends of the usage spectrum are revealing.  Neither, by itself, is a total answer."  Now, a way of getting the benefits of each is starting to appear. Here's how:

1. Remove side effects from pyzo imports:

Most (all?) pyzo modules start by importing the pyzo module itself.  This has numerous side effects besides defining classes and functions.  In other words, "import pyzo" does stuff, not all of which are compatible with Leo.  I'll be investigating replacing these side effects with explicit do-something calls.  This would allow this plugin to gain access to the pyzo code without copying!

2. Refactor various utilities into new modules in leo/external/pyzo.

This is a companion to removing side effects, but it has other virtues.  In particular, accessing pyzo's icons could be made simpler.  Similarly for other utilities, such as the translate function and its helpers.

Important: I am now giving myself expanded permission to change leo/external/pyzo. Imo, there is no other way to reconcile pyzo code with Leo.  Yes, one could imagine monkey-patching pyzo's files, but the side effects of pyzo imports would make that difficult or even impossible.  Instead, I'll clearly mark all changed code.

A grand strategy

This plugin suggests a relatively straightforward way towards including all and only the desired pyzo docks into Leo.

1. I'll refactor the code in leo/external/pyzo as discussed above. 

2. There will be a separate Leo plugin for each pyzo dock. Each plugin will import pyzo code from leo/external/pyzo as needed, without worrying about side effects.  The startup code (in the top-level onCreate function of each plugin) will then "make things happen" using the new do-something calls.  And that's just about it.

Summary

The present pyzo_file_browser plugin contains copies of all code needed to embed pyzo's file browser in Leo.  It works except for icons, which happen to be vital for clarity.  It also must be connected to Leo, which will also be straightforward.

This plugin has revealed a grand strategy, namely a refactoring of leo/external/pyzo into side-effect-free imports and do-something calls.  Imo, this refactoring should be relatively straightforward.  This grand strategy should provide the benefits of both use-by-copy and use-by-import.

If this grand strategy succeeds, it will allow each pyzo dock to be defined in a separate Leo plugin. Each such plugin will contain nothing but imports, calls to do-something functions, and a few lines of code in each plugin's onCreate top-level function.

As always, nothing is assured.  All aspects of this project remain experimental.  However, I now have my "marching orders".

Edward

[Edward K. Ream]

On Sat, Aug 10, 2019 at 8:33 AM Edward K. Ream <edre...@gmail.com> wrote:

> If this grand strategy succeeds, it will allow each pyzo dock to be defined in a separate Leo plugin. Each such plugin will contain nothing but imports, calls to do-something functions, and a few lines of code in each plugin's onCreate top-level function.

Eventually, I expect everything to end up in the pyzo_in_leo plugin.  However, during development, having a separate plugin for each pyzo dock probably will be the simplest way.  Imo, incremental development is a must.

Edward

[Edward K. Ream]

On Saturday, August 10, 2019 at 11:12:45 AM UTC-5, Edward K. Ream wrote:

> During development, having a separate plugin for each pyzo dock...will be the simplest way.  Imo, incremental development is a must.

Rev 75d917 in the pyzo branch adds icons to the pyzo_file_browser plugin, completely vindicating an incremental approach:

1. All new work started in the plugin. This made it dead easy to see what is, and isn't, needed.  In particular, the new PyzoIcons class is a drastic simplification of the Dict class in zon.py. Startup code is completely straightforward.

2. After completing the plugin, I copied the new icons code from the plugin to leo/external/pyzo/pyzo_icons.py.  All future pyzo/leo hybrid docks will be able to use pyzo_icons.py during startup without ever importing pyzo.  This is an important pattern!

No code was moved out of the leo/external/pyzo directory tree.  pyzo_icons.py contains copies of the plugin code. Eventually, perhaps, we could eliminate the duplication, but I'm not even going to think about doing that now.

The next step will be to fully connect the file browser dock to Leo.  I'll follow much the same plan.

Edward

[Edward K. Ream]

On Saturday, August 10, 2019 at 3:08:54 PM UTC-5, Edward K. Ream wrote:

> Rev 75d917 in the pyzo branch adds icons to the pyzo_file_browser plugin, completely vindicating an incremental approach:

Last night's rev 77004da makes more of the dock functional.

The plugin now contains an interim copy of pyzo's config code in zon.py.  This replaces the FileBrowserConfig that I wrote previously. Not sure why FileBrowserConfig doesn't work...

Debugging and responsibility

Recall that the "big question" is how to meld pyzo's code with Leo.  Recent work makes the following clear:

- Gaining access to (copies of) Pyzo's code will be relatively easy.

- Many changes to pyzo's code will be needed.  All such changes must be debugged, and become the responsibility of Leo's devs, present and future.

Incremental packaging

Moving pyzo's config code into the plugin was just barely feasible, even for a prototype.  Several renaming hacks were needed. Those hacks were justified in the moment because I had full control of all names in one place.

The new leo/external/pyzo/pyzo_config.py contains copies of the config-related code in the plugin.  However, there is now duplication between pyzo_config.py and the previous pyzo_icons_.py.  Doh! I now see that putting them in the pyzo folder was a mistake.  There would be no way to access them without importing pyzo (executing pyzo.__init__)!

Instead, I plan to put all needed support files in a new leo/plugins/pyzo_support directory.  Functions from various places in pyzo's startup code will go into pyzo_support/pyzo_functions.py.  Modules from pyzo's directory tree will go into subdirectories of pyzo_support.

Example

The top level of the plugin is now:

<< pyzo_file_browser imports >>
iconprovider = QtWidgets.QFileIconProvider()
@others

# Compute standard places.
pyzoDir, appDataDir = getResourceDirs()
# Load all icons.
pyzo_icons = loadIcons()
# From pyzo.start
pyzo_config = loadConfig()

This does the most (all?) of the needed startup code.  The getResourceDirs, loadIcons and loadConfig functions are minor mods of the "real" pyzo startup functions.  They will migrate from the plugin to pyzo_support/pyzo_functions.py.

The config class code in the plugin will migrate to pyzo_support/util/zon.py, without the naming hacks.

Important: The plugin uses pyzo_config where the original code uses pyzo.config.  Eventually, it would be good to monkey-patch pyzo.config, but this would require that importing pyzo/__init__.py have no side effects.  I've discussed this previously, but now it's becoming more important.  This means that significant changes will eventually have to be made to pyzo's startup logic.

Configuration, again

Pyzo's config code must be significantly changed.  I reverted to code in zon.py because I had no clue about what the legacy code was doing. But the legacy code will not suffice. Just as a start, some pyzo options will become Leo options.

The big change in attitude is that there is no such thing as a config shim. There is just configuration code, which will change as needed.

Summary

The big question is how to meld Leo and pyzo.  The answer is becoming clearer.  The pyzo_file_browser plugin has, somewhat unexpectedly, become an excellent test bed.  Incremental changes to this plugin (and soon, support files) replace the notion of shims.

The upcoming leo/plugins/pyzo_support directory tree will contain a significant amount of pyzo code, some of it modified. This directory will allow access to pyzo's code, without any side effects caused by importing pyzo itself. This pattern is an essential step in all pyzo-related projects.

Many mods to pyzo's code will be required.  All those mods must be debugged. All become the responsibility of Leo's devs.

Work on the pyzo_file_browser plugin has been rapid, and fun. Clearly, all the puzzles that arise can be solved fairly elegantly and rapidly. The resulting techniques will be the basis of integrating other parts of pyzo into Leo.  Generalizing these techniques is always in my mind.

Onward!

Edward

P.S. I am committed that the file browser plugin will become part of Leo.  I have no commitments that the other parts of pyzo become part of Leo, despite them being nominally scheduled for Leo 6.1.  Having said that, I may get carried away by the puzzles involved, and do my best to move as many as I can into Leo as soon as I can.

EKR

[Edward K. Ream]

On Sunday, August 11, 2019 at 7:11:31 AM UTC-5, Edward K. Ream wrote:

Incremental packaging

...I plan to put all needed support files in a new leo/plugins/pyzo_support directory.  Functions from...pyzo's startup code will go into pyzo_support/pyzo_functions.py.  Modules...will go into subdirectories of pyzo_support.

Done at rev c38b4dd6. This is a significant reorg.

All pyzo-related plugins and/or code can now easily share a single copy of modified pyzo code.

This is a work in progress.  Details will change as needed without notice.

Edward

[Edward K. Ream]

On Sunday, August 11, 2019 at 7:11:31 AM UTC-5, Edward K. Ream wrote:

> I plan to put all needed support files in a new leo/plugins/pyzo_support directory.

Did that yesterday.  It worked.

> Generalizing [the code] is always in my mind.

When I awoke I saw that yesterday's code must indeed be generalized.  Now.

Yesterday's code changed pyzo.config to pyzo_config everywhere in pyzo_support.  That's hardly a general solution.

leo/plugins/pyzo

The next step will be to create a second, complete, copy of pyzo in the plugins directory.  Why do that, you ask?

leo/external/pyzo is the reference copy. It contains useful traces.  It won't ever change significantly, and will probably be removed once all pyzo-related projects are complete.

leo/plugins/pyzo will be the experimental copy. The goal will be to find the minimal changes needed to adapt pyzo to Leo, not the other way around.  To make imports work without changes, the "root" folder must be called "pyzo", not pyzo_support.

De-fanging "import pyzo"

Almost all of pyzo's modules start with "import pyzo".  This import must be made benign. It must init all global data without instantiating pyzo's editors, shells, tools or servers, etc.

I plan to add a start_leo_in_pyzo function somewhere, probably in pyzo.__init__.py itself.  This will do all and only what Leo needs.

This is a highly experimental plan. The experimental copy will: reduce risk, allow easy reversions of code, and allow me to compare reference traces with experimental traces.

Summary

leo/plugins/pyzo will allow me to experiment safely with significant changes to pyzo's code, especially pyzo's startup code.

Side effects of "import pyzo" must be eliminated.

Edward

[Edward K. Ream]

On Monday, August 12, 2019 at 5:55:08 AM UTC-5, Edward K. Ream wrote:

 

> leo/plugins/pyzo will allow me to experiment safely with significant changes to pyzo's code, especially pyzo's startup code.

Done.

> Side effects of "import pyzo" must be eliminated.

 

Done.  

Today marks a huge milestone. The pyzo_file_browser plugin has collapsed in complexity!

- The plugin gets all pyzo's code using imports from leo/plugins/pyzo!

- The new start_pyzo_in_leo() function completely de-fang's "import pyzo"!

Here are the plugin's imports:

# Must patch sys.path here.
import sys
plugins_dir = g.os_path_finalize_join(g.app.loadDir, '..', 'plugins')
sys.path.insert(0, plugins_dir)

# Start pyzo, de-fanged.
import pyzo
pyzo.start_pyzo_in_leo()

# Import the file browser.
from pyzo.tools.pyzoFileBrowser import PyzoFileBrowser

That's all!

Summary

At long last there is an easy path for using any part of pyzo's code, modified as we like, without monkey-patches.

The generalized pyzo_in_leo plugin will use exactly this pattern.

Edward

[Edward K. Ream]

On Mon, Aug 12, 2019 at 9:24 AM Edward K. Ream <edre...@gmail.com> wrote:

> At long last there is an easy path for using any part of pyzo's code, modified as we like, without monkey-patches.

In retrospect, it's now clear that:

1. The new scheme combines the best features of use-by-copy and use-by-import:

- use-by-copy: Leo's devs can (must!) change the code in leo/plugins/pyzo so as to adapt pyzo to Leo.

- use-by-import: pyzoic plugins can simply import using the imports shown earlier.

2. The new scheme is the only way that has any realistic chance of working.

In short, today's work fully, elegantly, and permanently answers the big question.  I live for moments like this.

Edward

[Edward K. Ream]

On Monday, August 12, 2019 at 9:59:05 AM UTC-5, Edward K. Ream wrote:

> The new scheme is the only way that has any realistic chance of working.

Implications

1. We can now see clearly we must import all of pyzo's code into Leo. Exactly where doesn't matter. We can't adapt pyzo to Leo in any other way. This is a significant commitment for future devs.  Imo, it's probably worth doing, but we should confront the costs, particularly...

2. Comments of the form #EKR:change-(why)mark all significant changes. In future, if we want to take advantage of updates to pyzo, we would have to retro-fit those EKR changes back into the updated copy of pyzo's sources. This would likely take only a few hours.  The alternatives would be much worse.  Experience shows that we shouldn't even think about monkey-patches.

Edward

[Brian Theado]

Edward,

On Mon, Aug 12, 2019 at 11:13 AM Edward K. Ream <edre...@gmail.com> wrote:

On Monday, August 12, 2019 at 9:59:05 AM UTC-5, Edward K. Ream wrote:

[...] 

1. We can now see clearly we must import all of pyzo's code into Leo. Exactly where doesn't matter. We can't adapt pyzo to Leo in any other way. This is a significant commitment for future devs.  Imo, it's probably worth doing, but we should confront the costs, particularly...

2. Comments of the form #EKR:change-(why)mark all significant changes. In future, if we want to take advantage of updates to pyzo, we would have to retro-fit those EKR changes back into the updated copy of pyzo's sources. This would likely take only a few hours.  The alternatives would be much worse.  Experience shows that we shouldn't even think about monkey-patches.

 

3 observations

1. IMO, at a bare minimum, you should always first check in the pyzo code without any of your changes and only then add your changes. The EKR comments are probably very helpful to you, but that should done in addition to, not instead of the git tracking of your changes. Let git be the authoritative bookkeeper of changes...it is pretty good at that. Again, this is just my opinion.
2. Git has a very useful subcommand called 'subtree' which can be used to track another repository as a subdirectory of your own repository. Details below.
3. This is premature at this point, but can some/most/all the changes you make be generalized such that they are not leo specific? IOW, all they do is help make pyzo more embeddable into *any* python project, not just Leo? If the answer is yes, then you could open a pull request in upstream pyzo. If the changes get accepted, then that's less work on the Leo side going forward.

You can read about the subtree module at https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree.

The first time you bring the source in, the command would be something like:

git subtree add --prefix leo/plugins/pyzo <pyzo git url> <pyzo branch> --squash

Then later when you want to pull more recent code from pyzo you would run a command like this:

git subtree pull --prefix leo/plugins/pyzo <pyzo git url> <pyzo branch> --squash

git takes care of all the bookkeeping of mapping the file path prefix (leo/plugins/pyzo) and you will have the opportunity to resolve conflicts (if any) just like a normal git merge.

I've used this git feature lightly before and overall found it to be more help than pain.

Brian

[Edward K. Ream]

On Mon, Aug 12, 2019 at 4:22 PM Brian Theado <brian....@gmail.com> wrote:

1. IMO, at a bare minimum, you should always first check in the pyzo code without any of your changes and only then add your changes.

> IMO, at a bare minimum, you should always first check in the pyzo code without any of your changes and only then add your changes.

I agree. My personal copy of the pyzo code is in pyzo.leo, in c:\apps\pyzo.  It's been under git control from day 1. 

As you say, it would have been better to start with a clean copy in both leo/external/pyzo and leo/plugins/pyzo. When the dust clears it might be best to start completely afresh.  This would make clear what has happened in the git log, and will ensure that all required changes are properly marked.  But this is for later.

> Git has a very useful subcommand called 'subtree' which can be used to track another repository as a subdirectory of your own repository.

Thanks for this.

> can...the changes you make be generalized such that they are not leo specific?

Big sigh. This is a question for another day. The strategy that has just become clear probably already does that.  Having said that, I'm not going to clutter my head with another constraint.

Edward

[Edward K. Ream]

On Monday, August 12, 2019 at 9:24:43 AM UTC-5, Edward K. Ream wrote:

> Today marks a huge milestone. The pyzo_file_browser plugin has collapsed in complexity!

And today marks another huge milestone: the pyzo_in_leo plugin now loads all of pyzo's docks.  The code is ridiculously simple:

Imports

# Must patch sys.path here.
import sys
plugins_dir = g.os_path_finalize_join(g.app.loadDir, '..', 'plugins')
sys.path.insert(0, plugins_dir)
#
# Start pyzo, de-fanged.
import pyzo

onCreate

def onCreate(tag, keys):
    c = keys.get('c')
    if not c and c.frame:
        return
    pyzo.start_pyzo_in_leo(c, pyzo)

start_pyzo_in_leo

This contains imports and inits from MainWindow.__init__ and MainWindow._populate. It must get a c arg in order to access Leo's main window, c.frame.top.

start_pyzo_in_leo loads all of pyzo's tool docks as follows:

table = (
        'PyzoFileBrowser',
        'PyzoHistoryViewer',
        'PyzoInteractiveHelp',
        'PyzoLogger',
        'PyzoSourceStructure',
        'PyzoWebBrowser',
        'PyzoWorkspace',
    )
    for tool_id in table:
        pyzo.toolManager.loadTool(tool_id)
            # Put a floatable dock on the right.

And that's all. Leo handles these docks just as with Leo's other docks, except pyzo's docks are floatable. Leo remembers any changes to the location and visibility of these pyzo docks.

Summary

This marks the successful conclusion of the first phase of the pyzo in Leo project.

start_pyzo_in_leo completely inits pyzo and will contain any necessary tweaks.

Leo remembers the location and visibility of these pyzo docks. All pyzo docks are floatable.

The real work of melding pyzo and Leo lies ahead. Just for example, pyzo editors, if they exist, are hidden.

Oh yeah, I just remembered that the code doesn't create pyzo's shell area.  In pyzo, it's not a dock, but onCreate could make it a dock.

Edward

[Edward K. Ream]

On Wednesday, August 14, 2019 at 4:51:15 AM UTC-5, Edward K. Ream wrote:

> the pyzo_in_leo plugin now loads all of pyzo's docks.

I have retired the pyzo_file_browser plugin.  The pyzo_in_leo plugin now does everything that the pyzo_file_browser plugin did. As a result, there is no longer any need to share the start_pyzo_in_leo and load_all_docks functions. They are now defined in the pyzo_in_leo plugin.

Imports and monkey-patching

We seem to be going around and around with regard to monkey-patching the code, but at every iteration the code (and the ideas) become simpler.

I now see that importing pyzo (pyzo.__init__.py) has always been "de-fanged".  The rules:

1. Never call pyzo.start.

2. Never import pyzo.__main__.py (it calls pyzo.start)

Instead of using pyzo.start, Leo calls start_pyzo_in_leo.  This function contains init code from pyzo.start(), MainWindow.__init__ and MainWindow._populate.

Monkey-patching code might still be problematic, because start_pyzo_in_leo instantiates many pyzo objects.  For now (forever?) I'll patch the actual pyzo sources in place.

Summary

It's easy to de-fang pyzo imports.  Don't call pyzo.start, and don't import pyzo.__main__.py.

The start_pyzo_in_leo and load_all_docks functions have moved from pyzo.__init__.py to the pyzo_in_leo plugin.

Edward