Proposed change to setup.py

[tfer]

I've been making a mistake in setting up Leo and using it for years!

I've always run Leo from my local git clone of it, (usually the develop branch), never realizing I should have been using setup.py to make sure the python it is running under is fully ready to support it.  I just used launchleo.py to start it, adding any packages that it complained about as "missing" and carrying on.

I was using the very "fat" anaconda as my system python and not using any virtual environments.  Now that their conda utility program has been made to work under powershell, (it only worked "cmd", before), --note use "<path-to-conda>/conda init" to integrate it into powershell, I realiize this wasn't the way to go.

I got rid of Anaconda, (not easy, uninstall and getting rid of directories...), then just went with Miniconda.  I now can create conda env(s), leobase, leobase1, leobase2, and so on.  This is were I finally tumbled on what setup.py was for!  

My first naive approach was to run the script directly with the leobase enviroment active, but it does not run completely this way, (it does a bunch of stuff, then exits with a failed import).  I found (and Matt pointed out to me), that I should be using pip, specifically: "pip install --editable ." --while in the git directory of leo.

Anyway, I expanded the docstring of setup.py and put in a guard to prevent anyone from using it naively like I did.   I've attached below.

[Edward K. Ream]

On Thu, Mar 26, 2020 at 1:00 AM 'tfer' via leo-editor <leo-e...@googlegroups.com> wrote:

I've been making a mistake in setting up Leo and using it for years!

Yeah, setup.py has confused a lot of people.

Anyway, I expanded the docstring of setup.py and put in a guard to prevent anyone from using it naively like I did.   I've attached below.

Looks good to me. Matt, any comments?

Edward

[Thomas Passin]

I've always installed Leo using pip.  I still don't understand what "--editable" accomplishes, so I've never used it.

What I do, when I want to run from a git clone, is to set PYTHONPATH to the top directory of the clone.  Then Python will use, say, the Leo clone but it find all its other packages from the regular installed site-packages directory.  If I'm working on code for Leo, I keep the working code in a separate development directory, and bind a hot key to a little script that copies that file or files to the git clone.

This way, if the clone gets trashed somehow, I haven't lost any work.  I can just copy it to a new clone.

This approach is also compatible with using a virtual environment.

I recall reading that launchLeo acts different somehow than leo.core.runLeo, but I don't remember the differences. For what it's worth, I normally start Leo with python -m leo.core.runLeo.

[Matt Wilkie]

Anyway, I expanded the docstring of setup.py and put in a guard to prevent anyone from using it naively like I did.   I've attached below.

Looks good to me. Matt, any comments?

Thank you tfer. In all this time it never occured to me to add a message to guide people to usng pip instead! I'll use your code with some modification to do that. (We have at least one project downstream that uses Leo's setup.py directly so we should be mindful not to block that.)

I created #1547 for this.

-matt

[Matt Wilkie]

I've always installed Leo using pip.  I still don't understand what "--editable" accomplishes, so I've never used it.

Pip install puts the package and all the files it says it needs under PYTHONHOME/Lib/site-packages and creates a launch wrapper in PYTHONHOME/Scripts.

Pip install --editable doesn't copy anything, puts link files in Lib pointing to the code location, and creates the launch wrapper to code location. 

At present editable is the most reliable because it's guaranteed all files are present. The first way leaves some files out. Some are unavoidable without deep restructuring of Leo's code repo [#603, #573] and others I'm slowly tracking down and fixing as the bug reports come in (and I figure out how to remedy). [#1512 and probably #1547]

I recall reading that launchLeo acts different somehow than leo.core.runLeo, but I don't remember the differences. For what it's worth, I normally start Leo with python -m leo.core.runLeo

These two methods should be identical. I'm interested in more info if this isn't the case.

 -matt

[Thomas Passin]

On Saturday, March 28, 2020 at 4:45:01 PM UTC-4, Matt Wilkie wrote:

I've always installed Leo using pip.  I still don't understand what "--editable" accomplishes, so I've never used it.

Pip install puts the package and all the files it says it needs under PYTHONHOME/Lib/site-packages and creates a launch wrapper in PYTHONHOME/Scripts.

Pip install --editable doesn't copy anything, puts link files in Lib pointing to the code location, and creates the launch wrapper to code location.

So if I had a clone at d:\leo-clone, I'd use the following?

python -m pip install --editable d:\leo-clone

I  have been setting PYTHONPATH to d:\leo-clone.  This has always seemed to work well, as long as the dependencies have been installed in my main Python install (because of a previous Leo install, for example).  This way, I can always switch back to using the previous install just by launching from another terminal without setting PYTHONPATH.

[Matt Wilkie]

So if I had a clone at d:\leo-clone, I'd use the following?

python -m pip install --editable d:\leo-clone

 

Yes.

I  have been setting PYTHONPATH to d:\leo-clone.  This has always seemed to work well, as long as the dependencies have been installed in my main Python install (because of a previous Leo install, for example).  This way, I can always switch back to using the previous install just by launching from another terminal without setting PYTHONPATH.

If your diferent versions are in the same SCM repo (eg. hg, git) then resettng to or checking out your desired version can be done without messing with PYTHONPATH. This is the primary benefit of editable install, it always uses what's in that directory tree right now, allowing you to test code changes immediately.

pip install -e d:\leo-clone
pushd d:\leo-clone
git checkout v6.1
python -m leo.core.runLeo
# hack, hack. do stuff in Leo 6.1. exit.
git checkout v6.2
python -m leo.core.runLeo
# hack, hack. do stuff in Leo 6.2. exit.
...etc

If you want to run different versions of Leo side by side then don't pip install Leo at all, editable or not, and set PYTHONPATH the way you have been. I think it's cleaner to not to change PYTHONPATH though and to use `python d:\leo-clone-2\launchLeo.py` for starting. (It might mess with some imports or make them unpredictable as they search in Leo's tree instead of the real python tree.)

Also for sde-byside it's best not to use the same HOME, as files in the .leo folder might get messed up.

-matt

[Thomas Passin]

On Sunday, March 29, 2020 at 12:34:56 AM UTC-4, Matt Wilkie wrote:

So if I had a clone at d:\leo-clone, I'd use the following?

python -m pip install --editable d:\leo-clone

 

Yes.

I  have been setting PYTHONPATH to d:\leo-clone.  This has always seemed to work well, as long as the dependencies have been installed in my main Python install (because of a previous Leo install, for example).  This way, I can always switch back to using the previous install just by launching from another terminal without setting PYTHONPATH.

If your diferent versions are in the same SCM repo (eg. hg, git) then resettng to or checking out your desired version can be done without messing with PYTHONPATH. This is the primary benefit of editable install, it always uses what's in that directory tree right now, allowing you to test code changes immediately.

pip install -e d:\leo-clone
pushd d:\leo-clone
git checkout v6.1
python -m leo.core.runLeo
# hack, hack. do stuff in Leo 6.1. exit.
git checkout v6.2
python -m leo.core.runLeo
# hack, hack. do stuff in Leo 6.2. exit.
...etc

You get the same result if you point PYTHONPATH to the clone location. 

 

If you want to run different versions of Leo side by side then don't pip install Leo at all, editable or not, and set PYTHONPATH the way you have been. I think it's cleaner to not to change PYTHONPATH though and to use `python d:\leo-clone-2\launchLeo.py` for starting. (It might mess with some imports or make them unpredictable as they search in Leo's tree instead of the real python tree.)

Yes, that's a problem because Leo plugins aren't found the same way as ordinary imports.  I've been thinking of asking for a change so that the plugins are found using the PYTHONPATH instead of the current method.  Then you could keep your development code entirely out of the repo until you got it to the point it was worth adding it in.  I'd like that.

 

Also for sde-byside it's best not to use the same HOME, as files in the .leo folder might get messed up.

That's a problem that I sometimes have, not only with side-by-side.  I don't like to use side-by-side anyway except for, let's say, checking that something still works in a different version of Leo.  Even virtual environments don't solve this, do they?

[Matt Wilkie]

 

Then you could keep your development code entirely out of the repo until you got it to the point it was worth adding it in.  I'd like that.

A fork would accomplish this and then be easier to add to the main repo later. I think so anyway, but there  might be things I'm not considering that could make it more work than is desireable.

 

 

Also for sde-byside it's best not to use the same HOME, as files in the .leo folder might get messed up.

That's a problem that I sometimes have, not only with side-by-side.  I don't like to use side-by-side anyway except for, let's say, checking that something still works in a different version of Leo.  Even virtual environments don't solve this, do they?

Correct, virtuals don't help with this unless you also virtualize HOME (which is certainly possible by the way. There's a mechanism in conda and probably others to run scripts automatically when the env is activated.)

-matt

[Thomas Passin]

On Sunday, March 29, 2020 at 2:01:53 PM UTC-4, Matt Wilkie wrote:

 

Then you could keep your development code entirely out of the repo until you got it to the point it was worth adding it in.  I'd like that.

A fork would accomplish this and then be easier to add to the main repo later. I think so anyway, but there  might be things I'm not considering that could make it more work than is desireable.

With the confusions I've had using github forks, I think I'd like to work in my own Mercurial repo, where I would only need my own code and not the entire Leo distro, and copy the results to my Github fork of Leo.  For that to work, the most convenient way would be for Leo to load plugins from the PYTHONPATH instead of its own location.

One of the github things that I haven't mastered is when I start a branch for a project - say based off devel - and then that upstream branch changes before I'm ready to merge or do a pull request.  Of course, this happens all the time.  If I don't update my fork to upstream, then the merge or pull request could involve all those other changed files that I wasn't working on.  If I do update to upstream, then the update would step all over my work, and I would have to copy my work back into the fork.

Do you know how to make this all work?

So until I learn how to avoid this problem, I need a place to develop my stuff that's outside of the fork.

[Matt Wilkie]

With the confusions I've had using github forks, I think I'd like to work in my own Mercurial repo, where I would only need my own code and not the entire Leo distro, and copy the results to my Github fork of Leo.  For that to work, the most convenient way would be for Leo to load plugins from the PYTHONPATH instead of its own location.

One of the github things that I haven't mastered is when I start a branch for a project - say based off devel - and then that upstream branch changes before I'm ready to merge or do a pull request.  Of course, this happens all the time.  If I don't update my fork to upstream, then the merge or pull request could involve all those other changed files that I wasn't working on.  If I do update to upstream, then the update would step all over my work, and I would have to copy my work back into the fork.

Do you know how to make this all work?

One option is to enable the hg-git plugin, so you can continue to use Mercurial and push/pull from a git repo. I used this happily for a couple of years until a change in network policy at work blocked me from being to push outside the network.

As for not being clobbered by upstream changes or getting too far behind, I use 2 main strategies:

a) keep your work in it's own folder within the repo, a place no one else is likely to touch. This is not possible if you need to touch core though. However this problem isn't solved by the off-campus route you're using now either.

b) merge early, merge often: as in daily. This way any given upstream change can (usually) be inspected and accepted/changed in small easy to understand pieces. Use an interactice tool like WinMerge or Kmeld to do the merging. (WinMerge has been easiest for me to understand, the keyboard shortcuts make sense to me.)

Really though if what you're doing now is successful keep doing it! VR3 is awesome. Better you spend your precious attention on building cool things like that than figuring out git arcana. ;-)

-matt

[Thomas Passin]

On Sunday, March 29, 2020 at 10:29:36 PM UTC-4, Matt Wilkie wrote:

b) merge early, merge often: as in daily. This way any given upstream change can (usually) be inspected and accepted/changed in small easy to understand pieces. Use an interactice tool like WinMerge or Kmeld to do the merging. (WinMerge has been easiest for me to understand, the keyboard shortcuts make sense to me.)

Thanks for your advice, Matt.  I really don't want to spend time doing daily merges if I can avoid it.  And I'll look at Winmerge.  I think my ideal would be to create a branch in my fork that would only contain the files I'm working on.  But Git (or at least Github desktop) doesn't seem to work that way.  Otherwise I suppose that frequent updates from upstream would be the best way to go.

 

Really though if what you're doing now is successful keep doing it! VR3 is awesome. Better you spend your precious attention on building cool things like that than figuring out git arcana. ;-)

Well, thanks!  Though I feel that I'm just resurrecting Peter Mills's work and giving it a few tweaks.

 

-matt