Idea for a pip powered buildout

[Leonardo Rochael Almeida]

Hi,

On 30 May 2017 at 11:55, Leonardo Rochael Almeida <leoro...@gmail.com> wrote:

[...]

I believe there is a way to decouple the buildout workflow from setuptools, use pip, and keep the most of the buildout benefits intact while making things familiar to pip users.

This would make it easier to sell the buildout idea without bumping into the setuptools drawbacks.

And with some development in other packages (namely pip), I think we can achieve all of the current buildout benefits.

I'll start a separate thread to talk about my idea.

 So, here's my idea. It's an expansion of a concept I mentioned here:

https://github.com/buildout/buildout/issues/319#issuecomment-270942506

The idea is:

 - Buildout bootstraps itself into a venv/virtualenv in `parts/buildout` using pip (or perhaps get-pip.py), and then create a symlink from `parts/buildout/bin/buildout` into `bin/buildout` (on windows we could do a hard-link, which is the only kind of link link to a file that can be created without admin priviledges). Alternatively we could create a script that does `execfile('parts/buildout/activate_this.py') before calling the main buildout entry point`.

 - Every `zc.recipe.egg` part (and recipes that inherit from `zc.recipe.egg`) would each install a `venv/virtualenv` in `parts/<partname>`, and use pip to install all required packages. Then, it would create the requested scripts and interpreter in `bin/<script_name>` by calling `execfile('parts/<partname>/activate_this.py')` and then setting up the rest of the script according to `scripts`, `interpreter`, `entry-points`, `extra-paths`, `initialization` and `arguments` before calling the console entry point.

All other settings from `zc.recipe.egg` or interactions with global settings like `versions` have analogs either in the `pip` command line or by passing a `requirements.txt` file. In particular, the `-c` parameter to pip allows us to specify version restrictions to packages that were not requested directly for installation to pip, but are mere dependencies of the requested packages.

Packages in the `develop` setting would be installed by passing `-e <directory name>` to pip.

Advantages of this way of operating buildout:

 - We get buildout completely out of the business of understanding how the Python packaging system works at any given time. If in the future an installation frontend different from pip rises to prominence, we could just switch to that.

- If any breakages happen while installing a specific package, we can isolate this failure as a failure to install a package with pip (we could even display the exact pip command used to arrive at the problematic state and have the user take his issue up to pip and/or the packager of the problematic package).

 - It's easy to explain to other developers exactly what buildout is doing. Selling the advantages of buildout then becomes easier.

 - We get automatic isolation from system packages, between the packages needed to run buildout and its extensions (that will all be installed in `parts/buildout/.../site-packages`) and the packages needed by each `zc.recipe.egg` based part, and between the parts themselves. We can decide on a per part basis (including `parts/buildout` itself) whether we want isolation from the system packages or not. As a bonus, we don't have to ask people to run their buildout in a virtualevnv.

 - As I mentioned elsewhere, we get standard tooling integration (IDE autocomplete, advanced linting that requires knowing which packages are available) without having to do contortions. We just point the tooling to the respective virtalenv inside `parts`, or the respective `python` executable.

 - We could re-gain the ability of managing parts with different Python versions.

Disadvantages of this way of operating buildout:

- Duplication of code between dists of the same version in different virtualenvs inside `parts`. As of now, all eggs (or wheels with `buildout.wheel`) that are common between parts are shared in the `eggs` directory, saving disk space, with each needed `egg` being added to the `sys.path` at the start of each generated script.

- Jim Fulton thinks adding dists to PYTHONPATH as a superior way to operate, and would prefer to focus on that.

For the first of those disadvantages, we could try to push `pip` to be able to install from unpacked wheels by symlinking (or hardlinking on Windows) the files instead of copying them. Alternatively, we could go through the virtualenv and replace the installed files with sym(hard)links from the unpacked wheel cache buildout would maintain.

For the second, I have no current workaround ;-)

Jim, what do you think of the plan above? If you would still prefer to operate on adding dists to `sys.path`, can you elaborate on what advantages it has on the plan above?

Cheers,

Leo