Yet another idea for a pip-powered buildout

[Jim Fulton]

Motivation

I *still* think buildout's approach to installing python packages is superior in many ways, however, reality...  I think to pursue the egg/ovo route would require a rewrite on top of pip components.  I'm not sure anyone wants to spend the time to do that.  Even if we did, we'd still have a mindshare challenge. Very few people know what buildout is.

The other day I learned about pipenv, https://github.com/kennethreitz/pipenv, and it sounded interesting mainly to emphasize the degree to which the community has coalesced around pip and virtualenv (for better or worse).

Also packaging is hard, especially considering security.

Idea

Don't install packages, not even for recipes.

That's pretty much it. :)

• Users manage virtualenvs (or whatever) however they want.  It's out of our hands.
• They install whatever recipes they need.
• Buildout is used solely to execute recipes using config data.
• zc.recipe.egg and kin become irrelevant.

zc.buildout could grow a new mode to enforce this.

Or perhaps there's an entirely new tool that just runs recipes.  (Of course, someone write recipes that managed venvs, but the new tool would have nothing to do with that. I like this idea because a) clean break, b) new name, and c) focusses on the real value proposition of buildout, which was automating system setup.

I realize this isn't a new idea.

Jim

--

Jim Fulton
http://jimfulton.info

[Reinout van Rees]

Op 13/06/2017 om 20:32 schreef Jim Fulton:

> Don't install packages, not even for recipes.
>
> That's pretty much it. :)
>

> * Users manage virtualenvs (or whatever) however they want. It's out
> of our hands.
> * They install whatever recipes they need.
> * Buildout is used solely to execute recipes using config data.
> * zc.recipe.egg and kin become irrelevant.

>
> zc.buildout could grow a new mode to enforce this.
>
> Or perhaps there's an entirely new tool that just runs recipes. (Of
> course, someone write recipes that managed venvs, but the new tool would
> have nothing to do with that. I like this idea because a) clean break,
> b) new name, and c) focusses on the real value proposition of buildout,
> which was automating system setup.

That would be quite a change in direction :-) If not necessarily a bad
one. Keeping everything running (and especially keeping everything
running with constant setuptools-induced breakage) takes a lot of work.

The good thing: buildout still works! After all those years. A fun fact:
when two colleagues of mine left last year they got a t-shirt with
"bin/buildout" on it as it is so central to our day-to-day python work :-)

But I'm also slowly tempted to do a more elaborate experiment with pip,
just to get rid of intermittent setuptools problems. Buildout itself is
rarely the problem nowadays.



A word of caution about "system setup": many will look at tools like for
instance "ansible" for creating nginx config files and other tasks that
I myself now do with buildout recipes.

And useful recipes/extensions like mr.developer are so tied up with
buildout's package installing behaviour... We'll have a hard time
getting them to work, I fear.





Reinout

--
Reinout van Rees http://reinout.vanrees.org/ rei...@vanrees.org
"Military engineers build missiles. Civil engineers build targets."

[Jim Fulton]

On Tue, Jun 13, 2017 at 2:53 PM, Reinout van Rees <rei...@vanrees.org> wrote:

Op 13/06/2017 om 20:32 schreef Jim Fulton:

Don't install packages, not even for recipes.

That's pretty much it. :)

  * Users manage virtualenvs (or whatever) however they want.  It's out
    of our hands.
  * They install whatever recipes they need.
  * Buildout is used solely to execute recipes using config data.
  * zc.recipe.egg and kin become irrelevant.

zc.buildout could grow a new mode to enforce this.

Or perhaps there's an entirely new tool that just runs recipes.  (Of course, someone write recipes that managed venvs, but the new tool would have nothing to do with that. I like this idea because a) clean break, b) new name, and c) focusses on the real value proposition of buildout, which was automating system setup.

That would be quite a change in direction :-) If not necessarily a bad one. Keeping everything running (and especially keeping everything running with constant setuptools-induced breakage) takes a lot of work.

Which is why, if we decided we wanted to maintain current functionality, I think we'd need to consider a rewrite.  (For some reason, I woke up in terror the other morning thinking of the security challenges around package installation.) I'd be up for trying this, but definitely not by myself.

 

The good thing: buildout still works! After all those years.

That's great.  OTOH, when setting up appveyor tests for some package a few weeks ago, I ended up installing everything, including recipes with pip, because I couldn't figure out how to make buildout work in that environment.  It's also tough getting a new person going with buildout.

 

A fun fact: when two colleagues of mine left last year they got a t-shirt  with "bin/buildout" on it as it is so central to our day-to-day python work :-)

:)

It's pretty central to mine too, and likely would even if we stopped installing packages.

 

But I'm also slowly tempted to do a more elaborate experiment with pip, just to get rid of intermittent setuptools problems. Buildout itself is rarely the problem nowadays.



A word of caution about "system setup": many will look at tools like for instance "ansible" for creating nginx config files and other tasks that I myself now do with buildout recipes.

Well:

• I don't think anyone uses ansible in development mode.
• I hate centralized configuration. I want each package to be self contained, with all relevant logic in one place.

And useful recipes/extensions like mr.developer are so tied up with buildout's package installing behaviour... We'll have a hard time getting them to work, I fear.

FTR, I'm not interested in breaking buildout. OTOH, if the support remains ad hoc (no blame intended) I might walk away from it some day.

Jim