Any interest in sprinting at PyCon?

[Jim Fulton]

IMO, buildout is worth preserving.

I think what's mainly needed is:

• documentation that doesn't suck (mea culpa)
• Not doctests (but maybe with examples that are tested using manuel)
• Goal driven: focus on what user is trying to accomplish, not on what the implementation does.
  
  
• pip(ish) support
  
  IMO, this should be done with an excellent recipe for building virtualenvs with it's features and usage driven by documentation that doesn't suck :).
  
  It should honor [versions], index, etc, as zc.recipe.eggs does now.
  
  Also, IMO, this shouldn't have anything to do with how buildout manages it's own recipes.

I think it would be fun and useful to get together to address these needs.

My focus would be on documentation but a major emphasis would be on a development story that's sane and sellable.

I guess the major contributors to buildout these days are in Europe, so I'd con$ider :) EuroPython as an alternative to PyCon.

Jim

--

Jim Fulton
http://jimfulton.info

[Dylan Jay]

Won't be at either but I suspect if backtracking could be made to work so it buildout could be made to complete in the face of complex version definitions. I think that would be a huge win. Its so hard to explain why a buildout has failed and how to go about finding conflicting conditions and pinning versions to a begginers.

--
You received this message because you are subscribed to the Google Groups "Buildout Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to buildout-develop...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim Fulton]

On Wed, Jan 18, 2017 at 10:40 PM, Dylan Jay <dj...@pretaweb.com> wrote:

Won't be at either

Bummer.

but I suspect if backtracking could be made to work so it buildout could be made to complete in the face of complex version definitions. I think that would be a huge win. Its so hard to explain why a buildout has failed and how to go about finding conflicting conditions and pinning versions to a begginers.

That's an interesting use case.  My sense is that the change to leverage pip/packaging is that this would cease to be buildout's problem. :) But of course it would still be the user's problem and the challenge will will remain in the UI to help the user figure out WTF.

Jim

 

On Wed, 18 Jan 2017, 9:49 PM Jim Fulton <j...@jimfulton.info> wrote:

IMO, buildout is worth preserving.

I think what's mainly needed is:

• documentation that doesn't suck (mea culpa)
• Not doctests (but maybe with examples that are tested using manuel)
• Goal driven: focus on what user is trying to accomplish, not on what the implementation does.
  
  
• pip(ish) support
  
  IMO, this should be done with an excellent recipe for building virtualenvs with it's features and usage driven by documentation that doesn't suck :).
  
  It should honor [versions], index, etc, as zc.recipe.eggs does now.
  
  Also, IMO, this shouldn't have anything to do with how buildout manages it's own recipes.

I think it would be fun and useful to get together to address these needs.

My focus would be on documentation but a major emphasis would be on a development story that's sane and sellable.

I guess the major contributors to buildout these days are in Europe, so I'd con$ider :) EuroPython as an alternative to PyCon.

Jim

--

Jim Fulton
http://jimfulton.info

--
You received this message because you are subscribed to the Google Groups "Buildout Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Buildout Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to buildout-development+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Theuni]

Hi there,

I won't make it to PyCon US, but I'm still pondering EuroPython.

Spending some time on buildout sounds like a good idea. We're still relying on buildout in a good number of projects and I still prefer to use it over pip's requirements.txt as I'm missing exactly the pinning. Something that could be helpful is to allow multiple different pin sets if buildout manages different virtualenvs. (Which in turn could also enable running virtualenvs w/ different Python versions than the one that buildout uses). We're already managing around this with our own tooling in batou and simply run many different buildouts, but maybe this also makes sense to internalize.

I can imagine sprinting on buildout. Something that I've been working on with my utilities was improved self-bootstrapping. It's not perfect but it's been working more reliably than bootstrap.py did. As I'm using virtualenv/pip for that and as buildout supposedly will grow more abilities in that direction, maybe self-bootstrapping becomes feasable again ... This also probalby touches that one situation where running buildout within a virtualenv requires you to manage the initial install and setuptools versions "correctly" as depending on the system's virtualenv version you may endup with pre-installed setuptools versions that causes buildouts internal bootstrapping to enter an infinite loop trying to install setuptools.

On another note, we're seeing more abilities to manage central Python installations on our systems (specifically NixOS) where we would be happy if buildout could leverage the system environment again.

And lastly, one thing that has always been bugging me is that buildout can't detect broken binary packages that need to be rebuild if the underlying system changes. 

I marked the dates for EP in my calendar, but I'm just coming back from holidays. I might opt to join for the second half plus sprints. I'm not happy to do 7 day conferences any longer, but I could imagine joining for the sprints and maybe a day of conference or so.

Cheers,

Christian

[Jim Fulton]

On Fri, Jan 20, 2017 at 10:46 AM, 'Theuni' via Buildout Development <buildout-d...@googlegroups.com> wrote:

Hi there,

I won't make it to PyCon US, but I'm still pondering EuroPython.

Spending some time on buildout sounds like a good idea. We're still relying on buildout in a good number of projects and I still prefer to use it over pip's requirements.txt as I'm missing exactly the pinning. Something that could be helpful is to allow multiple different pin sets if buildout manages different virtualenvs. (Which in turn could also enable running virtualenvs w/ different Python versions than the one that buildout uses). We're already managing around this with our own tooling in batou and simply run many different buildouts, but maybe this also makes sense to internalize.

Good points.  I guess having a recipe that manages virtualenvs could also make it easier to support multiple Python versions, especially if that's calling out to pip anyway.

 

I can imagine sprinting on buildout. Something that I've been working on with my utilities was improved self-bootstrapping. It's not perfect but it's been working more reliably than bootstrap.py did. As I'm using virtualenv/pip for that and as buildout supposedly will grow more abilities in that direction, maybe self-bootstrapping becomes feasable again ... This also probalby touches that one situation where running buildout within a virtualenv requires you to manage the initial install and setuptools versions "correctly" as depending on the system's virtualenv version you may endup with pre-installed setuptools versions that causes buildouts internal bootstrapping to enter an infinite loop trying to install setuptools.

On another note, we're seeing more abilities to manage central Python installations on our systems (specifically NixOS) where we would be happy if buildout could leverage the system environment again.

Well, it can and always could.  It all depends on the isolation you need/want.  Also, I think it was more critical early on when setuptools was evolving rapidly. I've often gotten by using the system's Python over the last few years.

I'll note in passing, that containers make me much less averse to using a system Python because containers let me control the system. :)

 

And lastly, one thing that has always been bugging me is that buildout can't detect broken binary packages that need to be rebuild if the underlying system changes. 

That's an interesting point, and another reason to use containers.

Jim

[Theuni]

Hi,

On Friday, 20 January 2017 16:54:52 UTC+1, Jim Fulton wrote:

 

I can imagine sprinting on buildout. Something that I've been working on with my utilities was improved self-bootstrapping. It's not perfect but it's been working more reliably than bootstrap.py did. As I'm using virtualenv/pip for that and as buildout supposedly will grow more abilities in that direction, maybe self-bootstrapping becomes feasable again ... This also probalby touches that one situation where running buildout within a virtualenv requires you to manage the initial install and setuptools versions "correctly" as depending on the system's virtualenv version you may endup with pre-installed setuptools versions that causes buildouts internal bootstrapping to enter an infinite loop trying to install setuptools.

On another note, we're seeing more abilities to manage central Python installations on our systems (specifically NixOS) where we would be happy if buildout could leverage the system environment again.

Well, it can and always could.  It all depends on the isolation you need/want.  Also, I think it was more critical early on when setuptools was evolving rapidly. I've often gotten by using the system's Python over the last few years.

Oh, I think I might have just gotten so used to it that I never questioned it again at some point. Thanks for the reminder! 

I'll note in passing, that containers make me much less averse to using a system Python because containers let me control the system. :)

And lastly, one thing that has always been bugging me is that buildout can't detect broken binary packages that need to be rebuild if the underlying system changes. 

That's an interesting point, and another reason to use containers.

Exactly. Or rather: we're not even using containers but the Nix package manager. This allows us to create a multitude of finely controlled Python environments that have solved the issue of binary re-linking properly. They're independent of whether to run in a sub-shell, user environment, system-wide, in containers or VMs, or ...  

 

Christian

[Maurits van Rees]

You mean you miss pointing to for example a zope-toolkit.cfg with dozens of pins even when out of all those pins you are only going to install zope.interface?

That is possible with pip:

pip install --constraint zope-toolkit.cfg zope.interface

Maurits van Rees

http://maurits.vanrees.org

P.S. I have joined the list yesterday.

[Leonardo Rochael Almeida]

Hi Theuni,

On 20 January 2017 at 13:46, 'Theuni' via Buildout Development <buildout-d...@googlegroups.com> wrote:

[...]

I can imagine sprinting on buildout. Something that I've been working on with my utilities was improved self-bootstrapping. It's not perfect but it's been working more reliably than bootstrap.py did. As I'm using virtualenv/pip for that and as buildout supposedly will grow more abilities in that direction, maybe self-bootstrapping becomes feasable again ... This also probalby touches that one situation where running buildout within a virtualenv requires you to manage the initial install and setuptools versions "correctly" as depending on the system's virtualenv version you may endup with pre-installed setuptools versions that causes buildouts internal bootstrapping to enter an infinite loop trying to install setuptools.

I've diagnosed that issue thoroughly, but didn't get the round tuits to make a PR:

https://github.com/buildout/buildout/issues/312

Fixing it should be a nice sprint warm-up exercise...

[Dylan Jay]

On Thu, 19 Jan 2017, 8:11 PM Jim Fulton <j...@jimfulton.info> wrote:

On Wed, Jan 18, 2017 at 10:40 PM, Dylan Jay <dj...@pretaweb.com> wrote:

Won't be at either

Bummer.

but I suspect if backtracking could be made to work so it buildout could be made to complete in the face of complex version definitions. I think that would be a huge win. Its so hard to explain why a buildout has failed and how to go about finding conflicting conditions and pinning versions to a begginers.

That's an interesting use case.  My sense is that the change to leverage pip/packaging is that this would cease to be buildout's problem. :) But of course it would still be the user's problem and the challenge will will remain in the UI to help the user figure out WTF.

If you have two recipes with the same dependency and the 2nd one has a more strict constraint then I think its buildouts problem since it is feasible (though inefficient) to make backtracking work by automatically uninstalling the 2nd recipe and reinstalling the 1st with the new constraint in mind. I'm not sure how pip would prevent that. 

I became very aware how limiting buildout not taking care of this situatuation is recently when I needed a lot of js developers to install the python server side to test with. It was very simple up until this constraint issue. It was easy for me with years of buildout experience to say "just pin X" but took lots of explaining. 

No one likes getting errors on first use. 

Jim

 

On Wed, 18 Jan 2017, 9:49 PM Jim Fulton <j...@jimfulton.info> wrote:

IMO, buildout is worth preserving.

I think what's mainly needed is:

• documentation that doesn't suck (mea culpa)
• Not doctests (but maybe with examples that are tested using manuel)
• Goal driven: focus on what user is trying to accomplish, not on what the implementation does.
  
  
• pip(ish) support
  
  IMO, this should be done with an excellent recipe for building virtualenvs with it's features and usage driven by documentation that doesn't suck :).
  
  It should honor [versions], index, etc, as zc.recipe.eggs does now.
  
  Also, IMO, this shouldn't have anything to do with how buildout manages it's own recipes.

I think it would be fun and useful to get together to address these needs.

My focus would be on documentation but a major emphasis would be on a development story that's sane and sellable.

I guess the major contributors to buildout these days are in Europe, so I'd con$ider :) EuroPython as an alternative to PyCon.

Jim

--

Jim Fulton
http://jimfulton.info

--
You received this message because you are subscribed to the Google Groups "Buildout Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to buildout-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Buildout Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to buildout-develop...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jim Fulton
http://jimfulton.info

--

You received this message because you are subscribed to the Google Groups "Buildout Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to buildout-develop...@googlegroups.com.

[Jim Fulton]

On Fri, Jan 20, 2017 at 7:32 PM, Dylan Jay <dj...@pretaweb.com> wrote:

On Thu, 19 Jan 2017, 8:11 PM Jim Fulton <j...@jimfulton.info> wrote:

On Wed, Jan 18, 2017 at 10:40 PM, Dylan Jay <dj...@pretaweb.com> wrote:

Won't be at either

Bummer.

but I suspect if backtracking could be made to work so it buildout could be made to complete in the face of complex version definitions. I think that would be a huge win. Its so hard to explain why a buildout has failed and how to go about finding conflicting conditions and pinning versions to a begginers.

That's an interesting use case.  My sense is that the change to leverage pip/packaging is that this would cease to be buildout's problem. :) But of course it would still be the user's problem and the challenge will will remain in the UI to help the user figure out WTF.

If you have two recipes with the same dependency and the 2nd one has a more strict constraint then I think its buildouts problem since it is feasible (though inefficient) to make backtracking work by automatically uninstalling the 2nd recipe and reinstalling the 1st with the new constraint in mind. I'm not sure how pip would prevent that. 

Pip users have the same problem.  I know this because it's been raised on the distutils list. You don't need two buildout parts. Two dependencies will do. 

My point wasn't that buildout users don't have this problem, but that the solution should happen at the pip (or setuptools) level.

When we started buildout, it wasn't a goal to solve packaging problems. The goal was to automate creation of software configurations.  I only ended up writing as much packaging code as I did because setuptools behavior didn't meet my requirements for repeatability.

I don't think that packaging is where buildout should be mainly providing value.

Jim

[Dylan Jay]

Even if its solved at a packaging level and at a pip level there will still be conflicts in buildout between parts as long as they share a single working env.

What's more I suspect its for more common between buildout parts than within a single package.  I suspect that a single package has less dependencies than a whole buildout and the package maintainer can put some pins in setup tools to help prevent it. 

But at the integration level is where the problems happen in my experience because you have a less experienced person mixing packages versions together that might be novel

 

Jim

--

Jim Fulton
http://jimfulton.info

--

[Theuni]

Hi,

unfortunately, with the shifted dates of EuroPython I won't be able to make it. :(

Just so you guys know and don't get get disappointed expecting me there ... *really sad panda*

Christian

On Wednesday, 18 January 2017 14:49:07 UTC+1, Jim Fulton wrote:

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 01/18/2017 08:48 AM, Jim Fulton wrote:
> IMO, buildout is worth preserving.
>
> I think what's mainly needed is:
>
>

> - documentation that doesn't suck (mea culpa) - Not doctests (but
> maybe with examples that are tested using manuel) - Goal driven: focus

> on what user is trying to accomplish, not on what the implementation
> does.
>

> - pip(ish) support

>
> IMO, this should be done with an excellent recipe for building
> virtualenvs with it's features and usage driven by documentation that
> doesn't suck :).
>
> It should honor [versions], index, etc, as zc.recipe.eggs does now.
>
> Also, IMO, this shouldn't have anything to do with how buildout
> manages it's own recipes.
>
> I think it would be fun and useful to get together to address these
> needs.
>
> My focus would be on documentation but a major emphasis would be on a
> development story that's sane and sellable.
>
> I guess the major contributors to buildout these days are in Europe,
> so I'd con$ider :) EuroPython as an alternative to PyCon.

I will be at the Portlan PyCon sprints for the first couple of days,
likely sprinting on Zope / PylonsProject stuff.


Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBAgAGBQJYyKcIAAoJEPKpaDSJE9HYv6MP/0jtqDrvUY9stki8Y7SFWe3c
+vvfF0ny0FzKGDlRDTdRjU8xh0WYdFS8E5wym0lRO8dKZzz281X4N7LT8+HvVt1h
wm67o0XWe5e5eQTlhJOxh6UAe1873+Np3lug2MQkCfSBGyJU974lQ4ct7OHp/27P
H3HYrVPIZNWOrLZ2POdnZJ5U7Two+Od42UwE5Nucyo48KZESVvI1zZ8lZo9wTZg7
nX6VC/zFSc/arLukIFssViOJZFBdPHxuSJoxde9BN0ou772JP2NOORcwEaa49Do4
dlSOO8I9yVQc73QS0oL7UKOnhcmstoUI4OgDRvZALfR72Wf+Nb+gdKWQ/Sls+IuU
Tpi21Je5eAzK/xbXh5sGVfmgMu5usiOX9nyS7pV6VAPtP8C9DWxCN0aeKBhp3auC
JDagjF5CirAh2WVrTdzWQA5a0sb8jJuwHOikM+LydE8UM+XmF7YJYo+xCDXHgE+K
tObearInG1Ji8ZFjsLftjqG/B7u/grYFGaBTKE8kMeOFdt5izT4QEdhKbendmnjv
dNNoME0422ZrTOnoxneUJVMq792mTmMEb5eCd9mFCTsup0NjY6DQsGTwcawc38va
L3Th0p+3fknhYfP/ojr1Jw10/XbvU6vFTGdl5iZG/TsVOkEG6MrlxaXF2kBEsUKF
qrRrHhlkrLUm8GoNQhyj
=1SE3
-----END PGP SIGNATURE-----