[Python-Dev] Status of packaging in 3.3

[Éric Araujo]

Hi all,

We need to make a decision about the packaging module in Python 3.3.
Please read this message and breathe deeply before replying :)

[Sorry this ends up being so long; Tarek, Georg, Guido, I hope you
have the time to read it.]

Let me first summarize the history of packaging in the standard
library. (Please correct if I misremember something; this email
expresses my opinion and I did not talk with Tarek or Alexis before
writing it.) Two years ago the distutils2 (hereafter d2) project was
started outside of the stdlib, to allow for fast-paced changes, releases
and testing before merging it back. Changes in distutils were reverted
to go back to misfeature-for-misfeature compatibility (but not
bug-for-bug: bug fixes are allowed, unless we know for sure everyone is
depending on them or working around them).

Tarek’s first hope was to have something that could be included in
2.7 and 3.2, but these deadlines came too fast. At one point near the
start of 2011 (didn’t find the email) there was a discussion with Martin
about adding support for the stable ABI or parallel builds to distutils,
in which Tarek and I opposed adding this new feature to distutils as per
the freeze policy, and Martin declared he was not willing to work
outside of the standard library. We (d2 developers and python-dev) then
quickly agreed that distutils2 would be merged back after the release of
3.2, which was done. There was no PEP requested for this addition,
maybe because this was not a fully new module but an improvement of an
existing one with real-world-tested features, or maybe just because
nobody thought about the process. In retrospect, a PEP would have
really helped define the scope of the changes and the roadmap for packaging.

Real life caused contributors to come and go, and the primary
maintainer (Tarek at first, me since last December) to be at times very
busy (like me these last three months), with the result that packaging
is in my opinion just not ready. Many big and small things need more
work: the three packaging PEPs implemented in d2 have small flaws or
missing pieces (I’m not repeating the list here to avoid spawning
subthreads) that need to be addressed, we’ve started to get feedback
from users and developers only recently (pip and buildout devs since
last PyCon for example) the public Python API of d2 is far from great,
the implementation is of very unequal quality, important features have
patches that are not fully ready (—and I do acknowledge that I am the
blocker for reviews on many of them—), the compiler system has not been
revised, tests are not all clear and robust, some of the bdist commands
need to be removed, a new bdist format needs to be designed, etc.

With beta coming, a way to deal with that unfortunate situation needs
to be found. We could (a) grant an exception to packaging to allow
changes after beta1; (b) keep packaging as it is now under a provisional
status, with due warnings that many things are expected to change; (c)
remove the unstable parts and deliver a subset that works (proposed by
Tarek to the Pyramid author on distutils-sig); (d) not release packaging
as part of Python 3.3 (I think that was also suggested on distutils-sig
last month).

I don’t think (a) would give us enough time; we really want a few
months (and releases) to hash out the API (most notably with the pip and
buildout developers) and clean the bdist situation. Likewise (c) would
require developer (my) time that is currently in short supply. (b) also
requires time and would make development harder, not to mention probable
user pain. This leaves (d), after long reflection, as my preferred
choice, even though I disliked the idea at first (and I fully expect
Tarek to feel the same way).

I’d like to stress that this is not as bad as it appears at first.
We (I) will have to craft reassuring wording to explain why 3.3b1 does
not include packaging any more, but I believe that it would be worse for
our users (code-wise and PR-wise) to deliver a half-finished version in
3.3 rather than to retract it and wait for 3.4. And if we keep in mind
that many people are still using a 2.x version, releasing in 3.3 or 3.4
makes no change for them: the standalone releases on PyPI will keep coming.

Developer-wise, this would *not* mean that the considerable effort
that went into porting and merging, and the really appreciated patches
from developers such as Vinay, would have been in vain: even if
packaging is removed from the main repo (or just from the release
systems), there would be a clone to continue development, or the module
would be added back right after the 3.3 release, or we would develop in
the d2 repo and merge it back when it’s ready—this is really an
implementation detail for the decision; my point is that the work will
not be lost.

Thanks for reading; please express your opinion (especially Tarek as
d2 project lead, Georg as RM and Guido as BDFL).
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Nick Coghlan]

Reverting and writing a full packaging PEP for 3.4 sounds like a wise course of action to me.

Regards,
Nick.
--
Sent from my phone, thus the relative brevity :)

[Ethan Furman]

Éric Araujo wrote:
> This leaves (d), after long reflection, as my preferred
> choice, even though I disliked the idea at first (and I fully expect
> Tarek to feel the same way).
>

> Thanks for reading; please express your opinion (especially Tarek as
> d2 project lead, Georg as RM and Guido as BDFL).

I would go with (d) -- it's still available on PyPI, and having a
half-done product in the final release would not be good.

~Ethan~ (as ordinary user ;)

[Paul Moore]

On 19 June 2012 22:46, Éric Araujo <mer...@netwok.org> wrote:
[...]

>  This leaves (d), after long reflection, as my preferred choice, even though
> I disliked the idea at first (and I fully expect Tarek to feel the same
> way).

I agree with Nick. It's regrettable, but this is probably the wisest
course of action. Remove packaging from 3.3, create a PEP clearly
defining what packaging should be, and aim to implement for 3.4.

It seems to me that there's a lot of interest in the packaging module,
but it's fragmented and people have very different goals and
expectations. Developing a PEP will likely be a big task in itself,
but I'd hope that a well-crafted PEP will provide something the
various people with an interest could get behind and work together on,
which might help ease the developer time issue. (Assuming, of course,
that championing the PEP doesn't burn Éric out completely...)

Paul.

[Chris McDonough]

On 06/19/2012 05:46 PM, Éric Araujo wrote:
> Hi all,
>
> We need to make a decision about the packaging module in Python 3.3.
> Please read this message and breathe deeply before replying :)

...

> With beta coming, a way to deal with that unfortunate situation needs to
> be found. We could (a) grant an exception to packaging to allow changes
> after beta1; (b) keep packaging as it is now under a provisional status,
> with due warnings that many things are expected to change; (c) remove
> the unstable parts and deliver a subset that works (proposed by Tarek to
> the Pyramid author on distutils-sig); (d) not release packaging as part
> of Python 3.3 (I think that was also suggested on distutils-sig last
> month).

I think it'd be very wise to choose (d) here. We've lived so long
without a credible packaging story that waiting one (or even two) more
major release cycles isn't going to make a huge difference in the long
run but including a version of packaging now which gets fixed in a rush
would probably end up muddying the already dark waters of Python
software distribution.

- C

[Antoine Pitrou]

On Tue, 19 Jun 2012 17:46:30 -0400
Éric Araujo <mer...@netwok.org> wrote:
>
> I don’t think (a) would give us enough time; we really want a few
> months (and releases) to hash out the API (most notably with the pip and
> buildout developers) and clean the bdist situation. Likewise (c) would
> require developer (my) time that is currently in short supply. (b) also
> requires time and would make development harder, not to mention probable
> user pain. This leaves (d), after long reflection, as my preferred
> choice, even though I disliked the idea at first (and I fully expect
> Tarek to feel the same way).

The question is what will happen after 3.3. There doesn't seem to be a
lot of activity around the project, does it?

Regards

Antoine.

[Guido van Rossum]

Nick nailed it (again).

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

--
--Guido van Rossum (python.org/~guido)

[Nick Coghlan]

On Wed, Jun 20, 2012 at 11:23 AM, Antoine Pitrou <soli...@pitrou.net> wrote:
> The question is what will happen after 3.3. There doesn't seem to be a
> lot of activity around the project, does it?

I think the desire is there, but there are enough "good enough"
approaches around that people find other more immediately satisfying
things to do with their time (hammering out consensus on packaging
issues takes quite a bit of lead time to fully resolve).

This will make a good guinea pig for my "release alphas early"
proposal, though :)

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Victor Stinner]

What is the status of the third party module on PyPI (distutils2)?
Does it contain all fixes done in the packaging module? Does it have
exactly the same API? Does it support Python 2.5 to 3.3, or maybe also
2.4?

How is the distutils2 module installed? Installed manually? Using pip
or setuptools? Is distutils2 included in some Linux distributions?

If it's simple to install distutils2, it's not a big deal to not have
it in the stdlib.

--

It is sometimes a pain to have a buggy module in Python. For example,
I got a lot of issues with the subprocess module of Python 2.5. I
started to include a copy of the subprocess module from Python 2.7 in
my projects to workaround these issues.

In my previous work we did also backport various modules to get the
last version of the xmlrpc client on Python 2.5 (especially for
HTTP/1.1, to not open a new TCP socket at each request).

I don't want to reopen the discussion "the stdlib should be an
external project". I just want to confirm that it is better to wait
until important users of the packaging API have finished their work
(on porting their project to distutils2, especially pip), before we
can declare the module (and its API) as stable.

By the way, what is the status of "pip using distutils2"?

Victor

[Georg Brandl]

Am 19.06.2012 23:46, schrieb Éric Araujo:

Thanks for the detailed explanation, Éric. Just quoting this paragraph,
since it contains the possibilities to judge:

> With beta coming, a way to deal with that unfortunate situation needs
> to be found. We could (a) grant an exception to packaging to allow
> changes after beta1; (b) keep packaging as it is now under a provisional
> status, with due warnings that many things are expected to change; (c)
> remove the unstable parts and deliver a subset that works (proposed by
> Tarek to the Pyramid author on distutils-sig); (d) not release packaging
> as part of Python 3.3 (I think that was also suggested on distutils-sig
> last month).

(a) and (b) are certainly out of the question. packaging must be solid
when shipped, and there's not enough time. (c) might work (depending on
what features we're talking about), but you say yourself that you won't
be able to spend the time required, so I agree with basically everybody
else that (d) is the way to go (together with a PEP).

Georg

[Dirkjan Ochtman]

On Tue, Jun 19, 2012 at 11:46 PM, Éric Araujo <mer...@netwok.org> wrote:
>  I don’t think (a) would give us enough time; we really want a few months
> (and releases) to hash out the API (most notably with the pip and buildout
> developers) and clean the bdist situation.  Likewise (c) would require
> developer (my) time that is currently in short supply.  (b) also requires
> time and would make development harder, not to mention probable user pain.
>  This leaves (d), after long reflection, as my preferred choice, even though
> I disliked the idea at first (and I fully expect Tarek to feel the same
> way).

It's a pity, but it sounds like the way to go.

This may be crazy, but just idly wondering: is there an opportunity
for the PSF to make things better by throwing some money at it?
Packaging appears to be one of those Hard problems, it might be a good
investment.

Cheers,

Dirkjan

[Donald Stufft]

On Wednesday, June 20, 2012 at 2:36 AM, Victor Stinner wrote:

What is the status of the third party module on PyPI (distutils2)?

Does it contain all fixes done in the packaging module? Does it have

exactly the same API? Does it support Python 2.5 to 3.3, or maybe also

2.4?

How is the distutils2 module installed? Installed manually? Using pip

or setuptools? Is distutils2 included in some Linux distributions?

If it's simple to install distutils2, it's not a big deal to not have

it in the stdlib.

--

It is sometimes a pain to have a buggy module in Python. For example,

I got a lot of issues with the subprocess module of Python 2.5. I

started to include a copy of the subprocess module from Python 2.7 in

my projects to workaround these issues.

In my previous work we did also backport various modules to get the

last version of the xmlrpc client on Python 2.5 (especially for

HTTP/1.1, to not open a new TCP socket at each request).

I don't want to reopen the discussion "the stdlib should be an

external project". I just want to confirm that it is better to wait

until important users of the packaging API have finished their work

(on porting their project to distutils2, especially pip), before we

can declare the module (and its API) as stable.

By the way, what is the status of "pip using distutils2"?

Some students started on a pip2 that was based on distutils2, but I don't think

they've really done much/anything with actually using distutils2 and have mostly

been working on other parts. 

Victor

_______________________________________________

Python-Dev mailing list

Pytho...@python.org

http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/donald.stufft%40gmail.com

["Martin v. Löwis"]

> This may be crazy, but just idly wondering: is there an opportunity
> for the PSF to make things better by throwing some money at it?
> Packaging appears to be one of those Hard problems, it might be a good
> investment.

Only if somebody steps forward to take the money - and somebody who can
be trusted to achieve something, as well.

The general problem is that issues may only occur when packages actually
use the library; so it may even be difficult to fix it in a concerted
effort since that fixing may actually spread over several months (or years).

Regards,
Martin

[Tarek Ziadé]

On 6/19/12 11:46 PM, Éric Araujo wrote:
...

>
> I don’t think (a) would give us enough time; we really want a few
> months (and releases) to hash out the API (most notably with the pip
> and buildout developers) and clean the bdist situation. Likewise (c)
> would require developer (my) time that is currently in short supply.
> (b) also requires time and would make development harder, not to
> mention probable user pain. This leaves (d), after long reflection,
> as my preferred choice, even though I disliked the idea at first (and
> I fully expect Tarek to feel the same way).

Yeah I feel the same way. +1 for (d). I had unfortunately no time
lately. Thanks for picking up things.

We want a solid distutils replacement, and I think we wrote solid PEPs
and seemed to have find consensus for most issues in the past two years.

So I prefer to hold it and have a solid implementation in the stldib.
The only thing I am asking is to retain ourselves to do *anything* in
distutils and continue to declare it frozen, because I know it will be
tempting to do stuff there...

Cheers
Tarek

[Dirkjan Ochtman]

On Wed, Jun 20, 2012 at 10:55 AM, Tarek Ziadé <ta...@ziade.org> wrote:
> So I prefer to hold it and have a solid implementation in the stldib. The
> only thing I am asking is to retain ourselves to do *anything* in distutils
> and continue to declare it frozen, because I know it will be tempting to do
> stuff there...

That policy has been a bit annoying. Gentoo has been carrying patches
forever to improve compilation with C++ stuff (mostly about correctly
passing on environment variables), and forward-porting them on every
release gets tedious, but the packaging/distutils2 effort has made it
harder to get them included in plain distutils. I understand there
shouldn't be crazy patching in distutils, but allowing it to inch
forward a little would make the lives of the Gentoo Python team
easier, at least.

Cheers,

Dirkjan

[Antoine Pitrou]

On Wed, 20 Jun 2012 15:00:52 +1000
Nick Coghlan <ncog...@gmail.com> wrote:
> On Wed, Jun 20, 2012 at 11:23 AM, Antoine Pitrou <soli...@pitrou.net> wrote:
> > The question is what will happen after 3.3. There doesn't seem to be a
> > lot of activity around the project, does it?
>
> I think the desire is there,

What makes you think that, exactly?

Regards

Antoine.

[Antoine Pitrou]

On Wed, 20 Jun 2012 11:05:43 +0200
Dirkjan Ochtman <dir...@ochtman.nl> wrote:
> On Wed, Jun 20, 2012 at 10:55 AM, Tarek Ziadé <ta...@ziade.org> wrote:
> > So I prefer to hold it and have a solid implementation in the stldib. The
> > only thing I am asking is to retain ourselves to do *anything* in distutils
> > and continue to declare it frozen, because I know it will be tempting to do
> > stuff there...
>
> That policy has been a bit annoying. Gentoo has been carrying patches
> forever to improve compilation with C++ stuff (mostly about correctly
> passing on environment variables), and forward-porting them on every
> release gets tedious, but the packaging/distutils2 effort has made it
> harder to get them included in plain distutils. I understand there
> shouldn't be crazy patching in distutils, but allowing it to inch
> forward a little would make the lives of the Gentoo Python team
> easier, at least.

I think the whole idea that distutils should be frozen and improvements
should only go in distutils2 has been misled. Had distutils been
improved instead, many of those enhancements would already have been
available in 3.2 (and others would soon be released in 3.3).

Deciding to remove packaging from 3.3 is another instance of the same
mistake, IMO.

Regards

Antoine.

[Antoine Pitrou]

On Tue, 19 Jun 2012 21:36:35 -0700
Guido van Rossum <gu...@python.org> wrote:
> Nick nailed it (again).

Let's make things clear: packaging is suffering from a lack of
developer involvement, and a lack of user interest.

What makes you think that removing packaging from 3.3, and adding the
constraint of a new PEP to be written, will actually *improve* things?

Regards

Antoine.

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Tarek Ziadé]

On 6/20/12 11:05 AM, Dirkjan Ochtman wrote:
> On Wed, Jun 20, 2012 at 10:55 AM, Tarek Ziadé<ta...@ziade.org> wrote:
>> So I prefer to hold it and have a solid implementation in the stldib. The
>> only thing I am asking is to retain ourselves to do *anything* in distutils
>> and continue to declare it frozen, because I know it will be tempting to do
>> stuff there...
> That policy has been a bit annoying. Gentoo has been carrying patches
> forever to improve compilation with C++ stuff (mostly about correctly
> passing on environment variables), and forward-porting them on every
> release gets tedious, but the packaging/distutils2 effort has made it
> harder to get them included in plain distutils. I understand there
> shouldn't be crazy patching in distutils, but allowing it to inch
> forward a little would make the lives of the Gentoo Python team
> easier, at least.
>
> Cheers,

If distutils gets new features I think it's killing the packaging effort.

Maybe these new features could be implemented in packaging, then bridged
in Distutils ?

the Compilation feature is isolated enough to do this.

In any case, I guess we should have some kind of policy in place where
we list the exceptions when distutils
can be changed.

Maybe in the packaging PEP ?

Cheers
Tarek

[Tarek Ziadé]

On 6/20/12 11:04 AM, Antoine Pitrou wrote:
> On Wed, 20 Jun 2012 15:00:52 +1000
> Nick Coghlan<ncog...@gmail.com> wrote:
>> On Wed, Jun 20, 2012 at 11:23 AM, Antoine Pitrou<soli...@pitrou.net> wrote:
>>> The question is what will happen after 3.3. There doesn't seem to be a
>>> lot of activity around the project, does it?
>> I think the desire is there,
> What makes you think that, exactly?

Maybe because the packaging fatigue occurs around 3 years after you
start fighting that best, and we do have fresh blood working on it ? :)

>
> Regards
>
> Antoine.
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Tarek Ziadé]

On 6/20/12 11:12 AM, Antoine Pitrou wrote:
> On Wed, 20 Jun 2012 11:05:43 +0200
> Dirkjan Ochtman<dir...@ochtman.nl> wrote:
>> On Wed, Jun 20, 2012 at 10:55 AM, Tarek Ziadé<ta...@ziade.org> wrote:
>>> So I prefer to hold it and have a solid implementation in the stldib. The
>>> only thing I am asking is to retain ourselves to do *anything* in distutils
>>> and continue to declare it frozen, because I know it will be tempting to do
>>> stuff there...
>> That policy has been a bit annoying. Gentoo has been carrying patches
>> forever to improve compilation with C++ stuff (mostly about correctly
>> passing on environment variables), and forward-porting them on every
>> release gets tedious, but the packaging/distutils2 effort has made it
>> harder to get them included in plain distutils. I understand there
>> shouldn't be crazy patching in distutils, but allowing it to inch
>> forward a little would make the lives of the Gentoo Python team
>> easier, at least.
> I think the whole idea that distutils should be frozen and improvements
> should only go in distutils2 has been misled. Had distutils been
> improved instead, many of those enhancements would already have been
> available in 3.2 (and others would soon be released in 3.3).

I tried to improve Distutils and I was stopped and told to start
distutils2, because
distutils is so rotten, any *real* change/improvment potentially brakes
the outside world.

This has not changed.

>
> Deciding to remove packaging from 3.3 is another instance of the same
> mistake, IMO.

So what are your suggesting, since you seem to know what's a mistake and
what's not ?

(time-travel machine not allowed)

> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Vinay Sajip]

Antoine Pitrou <solipsis <at> pitrou.net> writes:

>
> Deciding to remove packaging from 3.3 is another instance of the same
> mistake, IMO.
>

What's the rationale for leaving it in, when it's known to be
incomplete/unfinished?

Regards,

Vinay Sajip

[Antoine Pitrou]

On Wed, 20 Jun 2012 11:22:07 +0200
Tarek Ziadé <ta...@ziade.org> wrote:
> I tried to improve Distutils and I was stopped and told to start
> distutils2, because
> distutils is so rotten, any *real* change/improvment potentially brakes
> the outside world.

If distutils was so rotten, distutils2 would not reuse much of its
structure and concepts (and test suite), would it?

Most of the distutils2 improvements (new PEPs, setup.cfg, etc.) were
totally possible in distutils, weren't they?

> > Deciding to remove packaging from 3.3 is another instance of the same
> > mistake, IMO.
> So what are your suggesting, since you seem to know what's a mistake and
> what's not ?

I don't have any suggestion apart from keeping packaging in 3.3.

But I also think it would be better for the community if people were
not delusional when making decisions. Removing packaging from 3.3 is a
big risk: users and potential contributors will be even less interested
than they already are.

Here's a datapoint: distribute (*) is downloaded 100x more times than
distutils2 (**).

(*) http://pypi.python.org/pypi/distribute/
(**) http://pypi.python.org/pypi/Distutils2/

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Antoine Pitrou]

On Wed, 20 Jun 2012 09:51:03 +0000 (UTC)
Vinay Sajip <vinay...@yahoo.co.uk> wrote:
> Antoine Pitrou <solipsis <at> pitrou.net> writes:
>
> >
> > Deciding to remove packaging from 3.3 is another instance of the same
> > mistake, IMO.
> >
>
> What's the rationale for leaving it in, when it's known to be
> incomplete/unfinished?

As an incentive for users to start using the features that are
finished enough, and exercise the migration path from distutils.
The module can be marked "provisional" so as to allow further API
variations.

Regards

Antoine.

[Tarek Ziadé]

On 6/20/12 11:49 AM, Antoine Pitrou wrote:
> On Wed, 20 Jun 2012 11:22:07 +0200
> Tarek Ziadé<ta...@ziade.org> wrote:
>> I tried to improve Distutils and I was stopped and told to start
>> distutils2, because
>> distutils is so rotten, any *real* change/improvment potentially brakes
>> the outside world.
> If distutils was so rotten, distutils2 would not reuse much of its
> structure and concepts (and test suite), would it?

'much' is pretty vague here. distutils2 is a fork of distutils, that has
evolved a *lot*

if you look at the code, beside the compilation part and some commands,
most things are different.

distutils is "rotten" because when you change its internals, you might
break some software
that rely on them.

>
> Most of the distutils2 improvements (new PEPs, setup.cfg, etc.) were
> totally possible in distutils, weren't they?

I started there, remember ? And we ended up saying it was impossible to
continue without
breaking the packaging world.

>>> Deciding to remove packaging from 3.3 is another instance of the same
>>> mistake, IMO.
>> So what are your suggesting, since you seem to know what's a mistake and
>> what's not ?
> I don't have any suggestion apart from keeping packaging in 3.3.
>
> But I also think it would be better for the community if people were
> not delusional when making decisions. Removing packaging from 3.3 is a
> big risk: users and potential contributors will be even less interested
> than they already are.

That's a good point. But if no one works on its polishing *now*, it's
going to be the same effect on people:
they'll likely to be very annoyed if the replacer is not rock solid.

>
> Here's a datapoint: distribute (*) is downloaded 100x more times than
> distutils2 (**).
>
> (*) http://pypi.python.org/pypi/distribute/
> (**) http://pypi.python.org/pypi/Distutils2/

why would you expect a different datapoint ?

- Distutils2 was released as a beta software, and not really promoted yet

- Distribute is downloaded automatically by many stacks out there, and
PyPI does not make a difference
whether the hit was from a human behind pip, or from a stack like
zc.buildout

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Tarek Ziadé]

On 6/20/12 11:54 AM, Antoine Pitrou wrote:
> On Wed, 20 Jun 2012 09:51:03 +0000 (UTC)
> Vinay Sajip<vinay...@yahoo.co.uk> wrote:
>> Antoine Pitrou<solipsis<at> pitrou.net> writes:
>>
>>> Deciding to remove packaging from 3.3 is another instance of the same
>>> mistake, IMO.
>>>
>> What's the rationale for leaving it in, when it's known to be
>> incomplete/unfinished?
> As an incentive for users to start using the features that are
> finished enough, and exercise the migration path from distutils.
> The module can be marked "provisional" so as to allow further API
> variations.

It's true that some modules are quite mature and already useful:

- packaging.version (PEP 386)
- packaging.pypi
- packaging.metadata (PEP 345)
- packaging.database (PEP 386)

the part that is not ready is the installer and some setuptools bridging

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Antoine Pitrou]

On Wed, 20 Jun 2012 12:30:51 +0200
Tarek Ziadé <ta...@ziade.org> wrote:
>
> >
> > Most of the distutils2 improvements (new PEPs, setup.cfg, etc.) were
> > totally possible in distutils, weren't they?
> I started there, remember ? And we ended up saying it was impossible to
> continue without
> breaking the packaging world.

"we" were only certain people, AFAIR.

> why would you expect a different datapoint ?

I wasn't expecting a different datapoint, I'm pointing that shipping
packaging in the stdlib would provide a much better exposure.

Regards

Antoine.


_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Tarek Ziadé]

On 6/20/12 12:39 PM, Antoine Pitrou wrote:
> On Wed, 20 Jun 2012 12:30:51 +0200
> Tarek Ziadé<ta...@ziade.org> wrote:
>>> Most of the distutils2 improvements (new PEPs, setup.cfg, etc.) were
>>> totally possible in distutils, weren't they?
>> I started there, remember ? And we ended up saying it was impossible to
>> continue without
>> breaking the packaging world.
> "we" were only certain people, AFAIR.

That was the BDFL decision after a language summit. Having tried to
innovate in Distutils in the past,
I think it's a very good decision,

[Georg Brandl]

Am 20.06.2012 12:39, schrieb Antoine Pitrou:
> On Wed, 20 Jun 2012 12:30:51 +0200
> Tarek Ziadé <ta...@ziade.org> wrote:
>>
>> >
>> > Most of the distutils2 improvements (new PEPs, setup.cfg, etc.) were
>> > totally possible in distutils, weren't they?
>> I started there, remember ? And we ended up saying it was impossible to
>> continue without
>> breaking the packaging world.
>
> "we" were only certain people, AFAIR.

Yes. The people willing to work on packaging in Python, to be exact.

>> why would you expect a different datapoint ?
>
> I wasn't expecting a different datapoint, I'm pointing that shipping
> packaging in the stdlib would provide a much better exposure.

But not if it's not ready for prime time. (And providing the finished
distutils2 for Python 2 will provide even better exposure at the moment.)

Georg

[Paul Moore]

On 20 June 2012 10:12, Antoine Pitrou <soli...@pitrou.net> wrote:
> I think the whole idea that distutils should be frozen and improvements
> should only go in distutils2 has been misled. Had distutils been
> improved instead, many of those enhancements would already have been
> available in 3.2 (and others would soon be released in 3.3).

The problem seems to be that in the past, any changes in distutils
have been met with cries of "you can't do that", basically because the
lack of a clearly distinguished extension API means that the
assumption is that for any bit of the internals, someone somewhere has
monkeypatched or relied on it. The issue is compounded by the fact
that a lot of distutils is undocumented, or at least badly documented,
so saying "if it's not documented, it's internal" doesn't work :-(

Maybe if we could be a bit more aggressive in saying what counts as
"internal" and resist the cries of "but I use it", modifying distutils
might be a more viable option, But there doesn't seem to be anyone
willing to take and defend that step. IIRC, Tarek proposed
distutils2/packaging after getting frustrated with how little he could
usefully do on distutils itself.

> Deciding to remove packaging from 3.3 is another instance of the same
> mistake, IMO.

I see your point, but without sufficient developer resource, the
question is whether packaging is in a usable state at all. Nobody but
Éric is actually working on packaging (and as he says, even he is not
able to at the moment), so what alternatives are there? I guess one
extra option not mentioned by Éric is to make the packaging issues
into release blocker bugs. That would effectively stall the release of
3.3 until packaging could be brought to an acceptable state,
effectively a form of blackmail. I can't imagine anyone wants to take
that approach. And yet, some of the existing bugs would clearly be
release blockers if they were in any other part of Python.

I think the first question is, do we need an enhanced distutils in the
stdlib? As far as I can see, this one has been answered strongly, in
the affirmative, a few times now. And yet, the need seems to be a
diffuse thing, with no real champion (Tarek and Éric have both tried
to take that role, and both appear to have been unable to invest the
necessary amount of time - which doesn't surprise me, it seems to be a
massive task).

Removing packaging from 3.3, in my mind acknowledges that as it stands
the approach was a failed experiment[1]. Better to get it taken out
before it appears in a released version of Python. We need to rethink
the approach. I see a number of options going forward, all of which
are based round the need to ensure enough developer involvement, so
that Tarek and Éric get help, and don't simply burn out before we have
anything useful.

1. Reconsider the decision to freeze distutils, with a view to
migrating incrementally to the feature set we want from packaging.
That'll be hard as we'd need to take a much stronger line on making
changes that could break existing code (stronger in the sense of "so
we broke your code, tough - you were using undocumented/internal
APIs"). And I suspect Tarek wouldn't be willing to take this route, so
we immediately lose one resource. Maybe the other core developers
could take up the slack, though. For example, Antoine, you seem to be
implying that you would have worked on distutils if this had happened.

2. Free up distutils2 to develop as an external package, and then have
a PEP proposing its inclusion in the stdlib in due course, when it is
ready and has been proven in the wild. The benefit here is that I
assume that as a separate project, becoming a committer would be
easier than becoming a Python core developer, so there's a wider pool
of developers. The downside is that the timescales would be a lot
longer (I doubt we'd see anything in 3.4 this way, and maybe not even
3.5).

3. Write a PEP describing precisely what the packaging module will do,
get consensus/agreement, and then restart development based on a solid
scope and spec. This is the correct route for getting something added
direct to the stdlib, and it's a shame it didn't happen in the first
place for packaging. Having said that, the PEP would likely be huge,
given the scope of packaging, and would require a lot of time from a
champion. There's no guarantee that championing a PEP wouldn't burn
someone out just as rapidly as developing the module itself :-( And
also, given that the packaging documentation is one of its weak areas,
I'd have to say I have concerns as to whether a PEP would come
together in any case... The assumption here, though, is that the PEP
process creates the debate, and results in interested parties coming
together in the discussion. If we can keep that momentum, we get a
pool of interested developers who may well assist in the coding
aspects.

The one option I don't like is taking packaging out, releasing 3.3,
and then putting it straight back in as is, and simply carry on as
now, in the hope that it'll be ready for 3.4. I honestly doubt that
the only issue is that we've run out of time before 3.3. There are
more fundamental problems that need to be addressed as well -
specifically the reliance on one individual to bear all of the load.

Just my thoughts,
Paul.

[1] That reads really harshly. I don't mean to criticise any of the
work that's been done, I'm a huge fan of the idea of packaging, and
its goals. The "experiment" in this case is around process -
developing something as big and important as packaging with limited
developer resource, relatively directly in the core (bringing it in
from distutils2 sooner rather than later) and working from a series of
smaller PEPs focused on particular details, rather than an overall one
covering the whole package.

[Paul Moore]

On 20 June 2012 11:34, Tarek Ziadé <ta...@ziade.org> wrote:
> On 6/20/12 11:54 AM, Antoine Pitrou wrote:
>>
>> On Wed, 20 Jun 2012 09:51:03 +0000 (UTC)
>> Vinay Sajip<vinay...@yahoo.co.uk>  wrote:
>>>
>>> Antoine Pitrou<solipsis<at>  pitrou.net>  writes:
>>>
>>>> Deciding to remove packaging from 3.3 is another instance of the same
>>>> mistake, IMO.
>>>>
>>> What's the rationale for leaving it in, when it's known to be
>>> incomplete/unfinished?
>>
>> As an incentive for users to start using the features that are
>> finished enough, and exercise the migration path from distutils.
>> The module can be marked "provisional" so as to allow further API
>> variations.
>
> It's true that some modules are quite mature and already useful:
>
> - packaging.version     (PEP 386)
> - packaging.pypi
> - packaging.metadata  (PEP 345)
> - packaging.database   (PEP 386)
>
> the part that is not ready is the installer and some setuptools bridging

I've never seen that information mentioned before. So that's (good) news.

A question, then. Why is it not an option to:

1. Rip out all bar those 4 modules.
2. Make sure they are documented and tested solidly (they may already
be, I don't know).
3. Declare that to be what packaging *is* for Python 3.3.

Whether any of those modules are of any use in isolation, is a
slightly more complex question. As is whether the APIs are guaranteed
to be sufficient for further development on "the rest" of packaging,
given that by doing this we commit to API stability and backward
compatibility. Your comment "quite mature and already useful" is not
quite firm enough to reassure me that we're ready to set those modules
in stone (although presumably the 3 relating to the PEPs are, simply
because they implement what the PEPs say).

Paul.

[Hynek Schlawack]

On 06/20, Antoine Pitrou wrote:

> Let's make things clear: packaging is suffering from a lack of
> developer involvement,

Absolutely.

And to be more precise: solid hands-on leadership. Eric wrote it in his
original mail: both packaging maintainers are burned out/busy. That’s a state
that is very unlikely to attract more developers – myself included.

> and a lack of user interest.

Maybe I'm getting you wrong here, but ISTM that proper packaging is in the
short list on nearly everybody’s “things I wish they'd fix in Python”.

[Tarek Ziadé]

The last time I checked:

packaging.version is the implementation of PEP 386, and stable. It's one
building block
that would be helpful as-is in the stdlib. it's completely standalone.

packaging.metadata is the implementation of all metadata versions.
standalone too.

packaging.pypi is the PyPI crawler, and has fairly advanced features. I
defer to Alexis to tell us
is it's completely stable

packaging.database is where PEP 376 is. It has the most innovations,
implements PEP 376

packaging.config is the setup.cfg reader. Ity's awseome because
together with packaging.database
and packaging.markers, it gives you OS-independant data files. see
http://docs.python.org/dev/packaging/setupcfg.html#resources

Yeah maybe this subset could be left in 3.3

and we'd remove packaging-the-installer part (pysetup, commands, compilers)

I think it's a good idea !

[Antoine Pitrou]

On Wed, 20 Jun 2012 13:20:04 +0200
Hynek Schlawack <h...@ox.cx> wrote:
>
> > and a lack of user interest.
>
> Maybe I'm getting you wrong here, but ISTM that proper packaging is in the
> short list on nearly everybody’s “things I wish they'd fix in Python”.

I agree, but I think people have also been burnt by the setuptools
maintenance problem, the setuptools -> distribute migration, the
easy_install / pip duality, and other stuff. I'm not sure they want to
try out "yet another third-party distutils improvement from the
cheeseshop".

Regards

Antoine.

[Antoine Pitrou]

On Wed, 20 Jun 2012 12:11:03 +0100
Paul Moore <p.f....@gmail.com> wrote:
>
> I think the first question is, do we need an enhanced distutils in the
> stdlib?

I would answer a different question: we definitely need a better
distutils/packaging story. Whether it's in the form of distutils
enhancements, or another package, is not clear-cut.

By the way, let me point out that the "distutils freeze" has been
broken to implement PEP 405 (I approve the breakage myself):
http://hg.python.org/cpython/rev/294f8aeb4d4b#l4.1

> As far as I can see, this one has been answered strongly, in
> the affirmative, a few times now. And yet, the need seems to be a
> diffuse thing, with no real champion

Packaging is not a very motivating topic for many developers
(myself included). It's like the build process or the buildbot
fleet :-)

> 2. Free up distutils2 to develop as an external package, and then have
> a PEP proposing its inclusion in the stdlib in due course, when it is

> ready and has been proven in the wild. [...]

> The downside is that the timescales would be a lot
> longer (I doubt we'd see anything in 3.4 this way, and maybe not even
> 3.5).

Agreed, especially if the "proven in the wild" criterion is required
(people won't rush to another third-party distutils replacement, IMHO).

> 3. Write a PEP describing precisely what the packaging module will do,
> get consensus/agreement, and then restart development based on a solid
> scope and spec.

I think it's the best way to sink the project definitively. Our
community isn't organized for such huge monolithic undertakings.

> [1] That reads really harshly. I don't mean to criticise any of the
> work that's been done, I'm a huge fan of the idea of packaging, and
> its goals. The "experiment" in this case is around process -
> developing something as big and important as packaging with limited
> developer resource, relatively directly in the core (bringing it in
> from distutils2 sooner rather than later) and working from a series of
> smaller PEPs focused on particular details, rather than an overall one
> covering the whole package.

I cannot speak for Tarek, but one of the reasons it's been done as a
set of smaller PEPs is that these PEPs were meant to be included in
*distutils*, not distutils2. That is, the module already existed and
the PEPs were individual, incremental improvements.

Regards

Antoine.

[Nick Coghlan]

On Wed, Jun 20, 2012 at 9:31 PM, Tarek Ziadé <ta...@ziade.org> wrote:
> Yeah maybe this subset could be left in 3.3
>
> and we'd remove packaging-the-installer part (pysetup, commands, compilers)
>
> I think it's a good idea !

OK, to turn this into a concrete suggestion based on the packaging docs.

Declare stable, include in 3.3
------------------------------------------
packaging.version — Version number classes
packaging.metadata — Metadata handling
packaging.markers — Environment markers
packaging.database — Database of installed distributions

Maybe needed as dependencies for above?
------------------------------------------------
packaging.errors — Packaging exceptions
packaging.util — Miscellaneous utility functions

It seems to me that stripping the library, docs and tests back to just
these 4 modules and their dependencies shouldn't be much harder than
stripping packaging in its entirety, but my question is what benefit
would we gain from having these (and just these) in the 3.3 stdlib
over having them available on PyPI in distutils2? Third party projects
over the next couple of years are going to want to support more than
just 3.3, so simply depending on distutils2 for this functionality
seems like a far more sensible option.

OTOH, it does send a clear message that progress *is* being made, we
just tried to take too big a jump from implementing these lower level
standards up to "wholesale replacement of distutils" without first
clearly documenting exactly what was wrong with the status quo and
what we wanted to change about it as a sequence of PEPs.

I've broken up the rest of packaging's functionality below into a
series of candidate PEPs that may be more manageable than a single
monolothic "fix packaging" PEP. If we can get multiple interested
parties focusing on different aspects, that may also help with
reducing the burnout factor. Python's current packaging and
distribution story is held together with duct tape and baling wire due
to decisions that were made years ago - unwinding some of those
decisions and replacing them with an alternative that is built on a
solid architectural foundation backed by some level of community
consensus is *not* an easy task, and not one that will be completed
quickly (undue haste will fail the "some level of community consensus"
part, thus missing much of the point of the exercise). That said, I
don't think it's unsolvable either, and there's a reasonable chance to
get something cleaner in place for 3.4.

3.4 PEP: Distutils replacement: Packaging, Distribution & Installation
--------------------------------------------
# This is one of the big balls of mud w.r.t distutils where third
party projects dig deep into the implementation details because that
is the only way to get things done
# It may even be the case that this can be broken up even further
packaging.install — Installation tools
packaging.dist — The Distribution class
packaging.manifest — The Manifest class
packaging.command — Standard Packaging commands
packaging.command.cmd — Abstract base class for Packaging commands

3.4 PEP: Distutils replacement: Compiling Extension Modules
--------------------------------------------
# Another big ball of mud. It sounds like the Gentoo folks may
have some feedback in this space.
packaging.compiler — Compiler classes
packaging.compiler.ccompiler — CCompiler base class
packaging.compiler.extension — The Extension class

3.4 PEP: Standard library package downloader (pysetup)
----------------------------------
# Amongst other things, this needs to have a really good security
story (refusing to install unsigned packages by default, etc)
packaging.depgraph — Dependency graph builder
packaging.pypi — Interface to projects indexes
packaging.pypi.client — High-level query API
packaging.pypi.base — Base class for index crawlers
packaging.pypi.dist — Classes representing query results
packaging.pypi.simple — Crawler using the PyPI “simple” interface
packaging.pypi.xmlrpc — Crawler using the PyPI XML-RPC interface
packaging.tests.pypi_server — PyPI mock server
packaging.fancy_getopt — Wrapper around the getopt module # Why
does this exist?

3.4 PEP: Simple binary package distribution format
--------------------------------------------------------------------------

bdist_simple has been discussed enough times, finally seeing a PEP
for it would be nice :)

I think the main lesson to be learned here is that "fix packaging" is
simply too big a task to be managed sensibly. Smaller goals like
"Standardise versioning", "Fix package metadata", "Support uninstall"
are hard enough to achieve, but also provide concrete milestones along
the way to the larger goal.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Nick Coghlan]

On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou <soli...@pitrou.net> wrote:
> Agreed, especially if the "proven in the wild" criterion is required
> (people won't rush to another third-party distutils replacement, IMHO).

The existence of setuptools means that "proven in the wild" is never
going to fly - a whole lot of people use setuptools and easy_install
happily, because they just don't care about the downsides it has in
terms of loss of control of a system configuration.

> I cannot speak for Tarek, but one of the reasons it's been done as a
> set of smaller PEPs is that these PEPs were meant to be included in
> *distutils*, not distutils2. That is, the module already existed and
> the PEPs were individual, incremental improvements.

That initial set of PEPs were also aimed at defining interoperability
standards that multiple projects could implement independently, even
*without* support in the standard library.

As I wrote in my other email, I think one key aspect of where we went
wrong after that point was in never clearly spelling out just what we
collectively meant by "fix packaging". Most of the burden of
interpreting that phrase thus landed directly on the shoulders of the
distutils2 project lead.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Paul Moore]

On 20 June 2012 13:53, Nick Coghlan <ncog...@gmail.com> wrote:
[...]

> 3.4 PEP: Simple binary package distribution format
> --------------------------------------------------------------------------
>
>    bdist_simple has been discussed enough times, finally seeing a PEP
> for it would be nice :)

I had a PEP for this one part written - Éric had a brief look at it
but I never posted it publicly. Before it'll go anywhere, a bit more
of the "infrastructure PEPs" you mentioned and I trimmed would need to
be completed, but I'd be willing to resurrect it when we get to that
stage...

Paul.

[Alexis Métaireau]

On 20/06/2012 14:53, Nick Coghlan wrote:
> 3.4 PEP: Standard library package downloader (pysetup)
> ----------------------------------
> # Amongst other things, this needs to have a really good security
> story (refusing to install unsigned packages by default, etc)
> packaging.depgraph — Dependency graph builder
> packaging.pypi — Interface to projects indexes
> packaging.pypi.client — High-level query API
> packaging.pypi.base — Base class for index crawlers
> packaging.pypi.dist — Classes representing query results
> packaging.pypi.simple — Crawler using the PyPI “simple” interface
> packaging.pypi.xmlrpc — Crawler using the PyPI XML-RPC interface
> packaging.tests.pypi_server — PyPI mock server
> packaging.fancy_getopt — Wrapper around the getopt module # Why
> does this exist?

I'm okay and willing to work on this part. I started a full review of
the code I wrote years ago, and which clearly needs some cleaning.
Alos, I'm not sure to understand what having a PEP to manage this means:
should I describe all the API in a text document (with examples) so we
can discuss this and validate before doing the changes/cleanings to the API?

Alexis

[Alexis Métaireau]

On 20/06/2012 13:31, Tarek Ziadé wrote:
> packaging.metadata is the implementation of all metadata versions.
> standalone too.
>
> packaging.pypi is the PyPI crawler, and has fairly advanced features.
> I defer to Alexis to tell us
> is it's completely stable

packaging.pypi is functionally working but IMO the API can (and probably
should) be improved (we really lack feedback to know that).

[Nick Coghlan]

On Wed, Jun 20, 2012 at 11:19 PM, Alexis Métaireau <ale...@notmyidea.org> wrote:
> On 20/06/2012 14:53, Nick Coghlan wrote:
>>
>> 3.4 PEP: Standard library package downloader (pysetup)
>> ----------------------------------
>>     # Amongst other things, this needs to have a really good security
>> story (refusing to install unsigned packages by default, etc)
>>     packaging.depgraph — Dependency graph builder
>>     packaging.pypi — Interface to projects indexes
>>     packaging.pypi.client — High-level query API
>>     packaging.pypi.base — Base class for index crawlers
>>     packaging.pypi.dist — Classes representing query results
>>     packaging.pypi.simple — Crawler using the PyPI “simple” interface
>>     packaging.pypi.xmlrpc — Crawler using the PyPI XML-RPC interface
>>     packaging.tests.pypi_server — PyPI mock server
>>     packaging.fancy_getopt — Wrapper around the getopt module  # Why
>> does this exist?
>
> I'm okay and willing to work on this part. I started a full review of the
> code I wrote years ago, and which clearly needs some cleaning.
> Alos, I'm not sure to understand what having a PEP to manage this means:
> should I describe all the API in a text document (with examples) so we can
> discuss this and validate before doing the changes/cleanings to the API?

There would be two main parts to such a PEP:
- defining the command line interface and capabilities (pysetup)
- defining the programmatic API (packaging.pypi and the dependency
graph management)

I would suggest looking at PEP 405 (venv) and PEP 397 (Windows
launcher) to get an idea of the kind of content that might be
appropriate. It's definitely not necessary to reproduce the full API
details verbatim in the PEP text - it's OK to provide highlights and
point to a reference implementation for the full details.

The PEP process can also be a good way to get feedback on an API
design that otherwise may not be forthcoming (cf. the recent
inspect.Signature discussions).

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Alexis Métaireau]

Le mer. 20 juin 2012 15:28:56 CEST, Nick Coghlan a écrit :
> There would be two main parts to such a PEP:
> - defining the command line interface and capabilities (pysetup)
> - defining the programmatic API (packaging.pypi and the dependency
> graph management)

Okay. I don't think that the command line has anything to do with
packaging.pypi and dependency management tools.
One is dealing with the hole cli for different tings (install / remove /
search etc.) while the other one is only how to communicate with the
indexes and build dependency graphs from there.

We probably should put the cli part in a separate PEP, as the scopes
aren't the same that the ones I see for packaging.pypi / depgraph

> I would suggest looking at PEP 405 (venv) and PEP 397 (Windows
> launcher) to get an idea of the kind of content that might be
> appropriate. It's definitely not necessary to reproduce the full API
> details verbatim in the PEP text - it's OK to provide highlights and
> point to a reference implementation for the full details.

Thanks for the pointers, will read them and try to come back with a PEP
proposal.

Alexis

[Paul Moore]

On 20 June 2012 14:16, Alexis Métaireau <ale...@notmyidea.org> wrote:
> On 20/06/2012 13:31, Tarek Ziadé wrote:
>>
>> packaging.metadata is the implementation of all metadata versions.
>> standalone too.
>>
>> packaging.pypi is the PyPI crawler, and has fairly advanced features. I
>> defer to Alexis to tell us
>> is it's completely stable
>
>
> packaging.pypi is functionally working but IMO the API can (and probably
> should) be improved (we really lack feedback to know that).

I wasn't aware of this - I've had a look and my first thought is that
the documentation needs completing. At the moment, there's a lot that
isn't documented, and we should avoid getting into the same situation
as with distutils where people have to use undocumented APIs to get
anything done. There are a lot of examples, but not so much formal API
documentation.

I don't mean to pick on this one module - unless things have changed a
lot, the same is probably true of much of the rest of packaging. Lack
of documentation is the #1 criticism I've seen.

Are there people willing to do some serious documentation work to get
the docs for the "agreed stable" parts of packaging complete? There's
more time to do this (doc changes don't have to be done before the
beta), but by deciding to retain parts of packaging, we *are* making a
commitment to complete that documentation, in my view.

Paul.

PS packaging.pypi is nice - I wish I'd known of its existence for a
bit of work I was doing a little while ago...

[Bill Janssen]

Nick Coghlan <ncog...@gmail.com> wrote:

> On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou <soli...@pitrou.net> wrote:
> > Agreed, especially if the "proven in the wild" criterion is required
> > (people won't rush to another third-party distutils replacement, IMHO).
>
> The existence of setuptools means that "proven in the wild" is never
> going to fly - a whole lot of people use setuptools and easy_install
> happily, because they just don't care about the downsides it has in
> terms of loss of control of a system configuration.

Maybe not "happily" :-). Speaking for myself, I'd love to find an
alternative, but setuptools seems to be the only system that knows how
to build shared libraries across all my platforms. I've got little
interest in a packaging module that doesn't include the compiler magic
to do that.

Bill

[Éric Araujo]

Hi all,

Sorry I can’t take the time to reply to all messages, this week I’m
fully busy with work and moving out.

To answer or correct a few things:

- I am lacking time these months, but that’s because I’m still getting
used to having a full-time job and being settled into a new country.
With the feedback we’ve been getting from people recently, I am
motivated, not burned out.

- I have started building a group of distutils2 contributors here in
Montreal. People are motivated, but it takes time to learn the codebase
and tackle on the big things.

- The four modules identified as minimal, standalone, good subset all
have big problems (the PEPs have open issues, and the modules' APIs need
improvements).

- Tarek, Georg and Guido have pronounced. With all the respect I have
for Antoine’s opinion, and the valid concerns he raises and that I don’t
answer here, I consider option (d) accepted and will scrap one hour to
do it before b1.

Regards

[Georg Brandl]

Am 20.06.2012 17:34, schrieb Éric Araujo:
> Hi all,
>
> Sorry I can’t take the time to reply to all messages, this week I’m
> fully busy with work and moving out.
>
> To answer or correct a few things:
>
> - I am lacking time these months, but that’s because I’m still getting
> used to having a full-time job and being settled into a new country.
> With the feedback we’ve been getting from people recently, I am
> motivated, not burned out.
>
> - I have started building a group of distutils2 contributors here in
> Montreal. People are motivated, but it takes time to learn the codebase
> and tackle on the big things.
>
> - The four modules identified as minimal, standalone, good subset all
> have big problems (the PEPs have open issues, and the modules' APIs need
> improvements).

Tarek seems to think otherwise... looks like in the end, this subset could
only be included as "provisional".

Georg

[Tarek Ziadé]

On 6/20/12 5:44 PM, Georg Brandl wrote:
> Am 20.06.2012 17:34, schrieb Éric Araujo:
>> Hi all,
>>
>> Sorry I can’t take the time to reply to all messages, this week I’m
>> fully busy with work and moving out.
>>
>> To answer or correct a few things:
>>
>> - I am lacking time these months, but that’s because I’m still getting
>> used to having a full-time job and being settled into a new country.
>> With the feedback we’ve been getting from people recently, I am
>> motivated, not burned out.
>>
>> - I have started building a group of distutils2 contributors here in
>> Montreal. People are motivated, but it takes time to learn the codebase
>> and tackle on the big things.
>>
>> - The four modules identified as minimal, standalone, good subset all
>> have big problems (the PEPs have open issues, and the modules' APIs need
>> improvements).
> Tarek seems to think otherwise... looks like in the end, this subset could
> only be included as "provisional".

I defer to Eric -- My answers are probably missing recent changes he knows

>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev

> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[PJ Eby]

On Wed, Jun 20, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:

On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou <soli...@pitrou.net> wrote:
> Agreed, especially if the "proven in the wild" criterion is required
> (people won't rush to another third-party distutils replacement, IMHO).

The existence of setuptools means that "proven in the wild" is never
going to fly - a whole lot of people use setuptools and easy_install
happily, because they just don't care about the downsides it has in
terms of loss of control of a system configuration.

Um, this may be a smidge off topic, but what "loss of control" are we talking about here?  AFAIK, there isn't anything it does that you can't override with command line options or the config file.  (In most cases, standard distutils options or config files.)  Do you just mean that most people use the defaults and don't care about there being other options?  And if that's the case, which other options are you referring to?

If the long-term goal is to draw setuptools users over to packaging, then AFAIK the packaging effort is still missing a few things, like build-time dependencies and alternatives to setuptools' entry points and "extras", as well as the ability to integrate version control for building sdists (without requiring the sdist's recipient to *also* have the version control integration in order to build the package or recreate a new sdist).

These are just the missing features that I know of, from recent distutils-sig discussions; I don't know how complete a list this is.  While no single one of these features is directly used by every project or even a majority of such projects, there is a correlation between size of a project and the likelihood that they are depending on one or more of these features.  i.e., the bigger and more widely-used the project, the more likely it is to either use one of these features, or depend on a project that does.

Some of these features could be built on top of packaging, in more or less the same way setuptools is built on top of distutils.  But whether they're done inside or outside of the packaging library, somebody's going to have to do them, for people to be able to migrate off of setuptools.

[Nick Coghlan]

On Thu, Jun 21, 2012 at 3:29 AM, PJ Eby <p...@telecommunity.com> wrote:
> On Wed, Jun 20, 2012 at 9:02 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>>
>> On Wed, Jun 20, 2012 at 9:46 PM, Antoine Pitrou <soli...@pitrou.net>
>> wrote:
>> > Agreed, especially if the "proven in the wild" criterion is required
>> > (people won't rush to another third-party distutils replacement, IMHO).
>>
>> The existence of setuptools means that "proven in the wild" is never
>> going to fly - a whole lot of people use setuptools and easy_install
>> happily, because they just don't care about the downsides it has in
>> terms of loss of control of a system configuration.
>
>
> Um, this may be a smidge off topic, but what "loss of control" are we
> talking about here?  AFAIK, there isn't anything it does that you can't
> override with command line options or the config file.  (In most cases,
> standard distutils options or config files.)  Do you just mean that most
> people use the defaults and don't care about there being other options?  And
> if that's the case, which other options are you referring to?

No, I mean there are design choices in setuptools that explain why
many people don't like it and are irritated when software they want to
use depends on it without a good reason. Clearly articulating the
reasons that "just include setuptools" is no longer being considered
as an option should be one of the goals of any PEPs associated with
adding packaging back for 3.4.

The reasons I'm personally aware of:
- it's a unilateral runtime fork of the standard library that bears a
lot of responsibility for the ongoing feature freeze in distutils.
Standard assumptions about the behaviour of site and distutils cease
to be valid once setuptools is installed
- overuse of "*.pth" files and the associated sys.path changes for all
Python programs running on a system. setuptools gleefully encourages
the inclusion of non-trivial code snippets in *.pth files that will be
executed by all programs.
- advocacy for the "egg" format and the associated sys.path changes
that result for all Python programs running on a system
- too much magic that is enabled by default and is hard to switch off
(e.g. http://rhodesmill.org/brandon/2009/eby-magic/)

System administrators (and developers that think like system
administrators when it comes to configuration management) *hate* what
setuptools (and setuptools based installers) can do to their systems.
It doesn't matter that package developers don't *have* to do those
things - what matters is that the needs and concerns of system
administrators simply don't appear to have been anywhere on the radar
when setuptools was being designed. (If those concerns actually were
taken into account at some point, it's sure hard to tell from the end
result and the choices of default behaviour)

setuptools is a masterful achievement built on shaky foundations that
will work most of the time. However, when it doesn't work, you're
probably screwed, and as soon as it's present on a system, you know
that your assumptions about understanding the Python interpreter's
startup sequences are probably off. The efforts around
distutils2/packaging have been focused on taking the time to *fix the
foundations first* rather than accepting the inevitable shortcomings
of trying to build something in the middle of a swamp.

> If the long-term goal is to draw setuptools users over to packaging, then
> AFAIK the packaging effort is still missing a few things, like build-time
> dependencies and alternatives to setuptools' entry points and "extras", as
> well as the ability to integrate version control for building sdists
> (without requiring the sdist's recipient to *also* have the version control
> integration in order to build the package or recreate a new sdist).

Right - clearly enumerating the features that draw people to use
setuptools over just using distutils should be a key element in any
PEP for 3.4

I honestly think a big part of why packaging ended up being incomplete
for 3.3 is that we still don't have a clearly documented answer to two
critical questions:
1. Why do people choose setuptools over distutils?
2. What's wrong with setuptools that meant the idea of including it
directly in the stdlib was ultimately dropped and eventually replaced
with the goal of incorporating distutils2?

I imagine there are answers to both of those questions embedded in
past python-dev, distutils-sig, setuptools and distutils2 mailing list
discussions, but that's no substitute for having them clearly
documented in a PEP (or PEPs, given the scope of the questions).

We've tried to shortcircuit this process twice now, first with "just
include setuptools" back around 2.5, and again now with "just include
distutils2 as packaging" for 3.3. It hasn't worked, so maybe it's time
to try doing it properly and clearly articulating the desired end
result. If the end goal is "the bulk of the setuptools feature set
without the problematic features and default behaviours that make
system administrators break out the torches and pitchforks", then we
should *write that down* (and spell out the implications) rather than
assuming that everyone knows the purpose of the exercise.

Regards,

[Chris McDonough]

All of these are really pretty minor issues compared with the main
benefit of not needing to ship everything with everything else. The
killer feature is that developers can specify dependencies and users can
have those dependencies installed automatically in a cross-platform way.
Everything else is complete noise if this use case is not served.

IMO, the second and third things you mention above (use of pth files and
eggs) are actually features when compared against the result of
something like pip, which installs things using
--single-version-externally-managed and then tries to manage the
resulting potentially-intertwined directories. Eggs are *easier* to
manage than files potentially overlapping files and directories
installed into some other directory. Either they exist or they don't.
Either they're mentioned in a .pth file or they aren't. It's not really
that hard.

In any case, any tool that tries to manage distribution installation
will need somewhere to keep distribution metadata. It's a minor mystery
to me why people think it could be done much better than in something
very close to egg format.

> System administrators (and developers that think like system
> administrators when it comes to configuration management) *hate* what
> setuptools (and setuptools based installers) can do to their systems.
> It doesn't matter that package developers don't *have* to do those
> things - what matters is that the needs and concerns of system
> administrators simply don't appear to have been anywhere on the radar
> when setuptools was being designed. (If those concerns actually were
> taken into account at some point, it's sure hard to tell from the end
> result and the choices of default behaviour)

I think you mean easy_install here. And I guess you mean managing .pth
files. Note that if you use pip, neither thing needs to happen. And
even easy_install lets you install a distribution that way (with
--single-version-externally-managed). So I think, as you mention, this
is a matter of defaults (tool and or flag defaults) rather than core
functionality.

> setuptools is a masterful achievement built on shaky foundations that
> will work most of the time. However, when it doesn't work, you're
> probably screwed, and as soon as it's present on a system, you know
> that your assumptions about understanding the Python interpreter's
> startup sequences are probably off.

It's true setuptools is based on shaky foundations. The rest of the
stuff you say above is pretty darn specious, I think.

> The efforts around
> distutils2/packaging have been focused on taking the time to *fix the
> foundations first* rather than accepting the inevitable shortcomings
> of trying to build something in the middle of a swamp.
>
>> If the long-term goal is to draw setuptools users over to packaging, then
>> AFAIK the packaging effort is still missing a few things, like build-time
>> dependencies and alternatives to setuptools' entry points and "extras", as
>> well as the ability to integrate version control for building sdists
>> (without requiring the sdist's recipient to *also* have the version control
>> integration in order to build the package or recreate a new sdist).
>
> Right - clearly enumerating the features that draw people to use
> setuptools over just using distutils should be a key element in any
> PEP for 3.4
>
> I honestly think a big part of why packaging ended up being incomplete
> for 3.3 is that we still don't have a clearly documented answer to two
> critical questions:
> 1. Why do people choose setuptools over distutils?

Because it supports automated installation of dependencies. Almost
everything else is noise (although some of the other things that
setuptools provides, like entry points and console scripts, is important
noise).

> 2. What's wrong with setuptools that meant the idea of including it
> directly in the stdlib was ultimately dropped and eventually replaced
> with the goal of incorporating distutils2?

Because distutils sucks and setuptools is based on distutils. It's
horrible to need to hack on.

Setuptools also has documentation which is effectively deltas to the
distutils docs. As a result, it's very painful to try to follow the
setuptools docs. IMO, it's not that the ideas in setuptools are bad,
it's that setuptools requires a *lot* more docs to be consumable by
normal humans, and those docs need to be a lot more accessible.

> I imagine there are answers to both of those questions embedded in
> past python-dev, distutils-sig, setuptools and distutils2 mailing list
> discussions, but that's no substitute for having them clearly
> documented in a PEP (or PEPs, given the scope of the questions).
>
> We've tried to shortcircuit this process twice now, first with "just
> include setuptools" back around 2.5, and again now with "just include
> distutils2 as packaging" for 3.3. It hasn't worked, so maybe it's time
> to try doing it properly and clearly articulating the desired end
> result. If the end goal is "the bulk of the setuptools feature set
> without the problematic features and default behaviours that make
> system administrators break out the torches and pitchforks", then we
> should *write that down* (and spell out the implications) rather than
> assuming that everyone knows the purpose of the exercise.

There's all kinds of built in conflict here wrt to those pitchforks.
Most of it is stupid.

System admininstrators tend to be stuck in a "one package to rule them
all" model of deployment and that model *just cant work* on a system
where you need repeatable deployments of multiple pieces of Python-based
software which may require mutually exclusive different Python and
library versions. Trying to pretend it can work is just plain madness.
Telling developers they must work on an exact replica of the
production system in order to develop the software is also a terrible,
unproductive idea. This is a hopeless, 1990s waterfall model of
deployment and devlopment.

This is why packages like virtualenv and buildout are so popular. Using
them gets developers what they need. Developers get repeatable
cross-platform deployments without requiring special privilege, and this
allows for a *reduction* in the system administrator's role in
deployment. Sometimes a certain type of system administrator can be a
hindrance to deployment and maintenance, like sometimes a DBA can be a
hindrance to a developer who just needs to add a damn table.

With the tools available today (Fabric, buildout, salt, virtualenv,
pip), it's a heck of a lot easier to script a cross-platform deployment
that will work simultaneously on Debian, Red Hat, BSD, and Mac OS X than
it is to build system-level packages for multiple platforms or even
*one* platform. And to be honest, if a system administrator can't cope
with the notion that he may need to forsake his system-level package
installer and instead follow the instructions we give to him to type
four or five commands to get a completely working system deployed or
updated, he probably should not be a system administrator. His job is
going to quickly be taken by folks who *can* cope with such deployment
mechanisms like any cloud service: all the existing Python cloud
deployment services handle distutils/setuptools installs just fine and
these tend to be the *only* way you can get Python software installed
into a system on them.

- C

[Nick Coghlan]

On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough <chr...@plope.com> wrote:
> All of these are really pretty minor issues compared with the main benefit
> of not needing to ship everything with everything else. The killer feature
> is that developers can specify dependencies and users can have those
> dependencies installed automatically in a cross-platform way.  Everything
> else is complete noise if this use case is not served.

Cool. This is the kind of thing we need recorded in a PEP - there's a
lot of domain knowledge floating around in the heads of packaging
folks that needs to be captured so we can know *what the addition of
packaging to the standard library is intended to fix*.

And, like it or not, setuptools has a serious PR problem due to the
fact it monkeypatches the standard library, uses *.pth files to alter
sys.path for every installed application by default, actually *uses*
the ability to run code in *.pth files and has hard to follow
documentation to boot. I *don't* trust that I fully understand the
import system on any machine with setuptools installed, because it is
demonstrably happy to install state to the file system that will
affect *all* Python programs running on the machine.

A packaging PEP needs to explain:
- what needs to be done to eliminate any need for monkeypatching
- what's involved in making sure that *.pth are *not* needed by default
- making sure that executable code in implicitly loaded *.pth files
isn't used *at all*

I *think* trying to achieve this is actually the genesis of the
original distribute fork, that subsequently became distutils2 as Tarek
discovered how much of the complexity in setuptools was actually due
to the desire to *not* officially fork distutils (and instead
monkeypatch it, effectively creating a runtime fork).

However, for those of us that weren't directly involved, this is all
still a strange mystery dealt with by other people. I've cribbed
together bits and pieces just from following the fragments of the
discussions that have happened on python-dev and at PyCon US, but if
we want the madness to ever stop, then *the problems with the status
quo* need to be written down so that other core developers can
understand them.

In fact, I just remembered that Tarek *has* written a lot of this
down, just not in PEP form: http://www.aosabook.org/en/packaging.html

Cheers,

Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Dag Sverre Seljebotn]

David Cournapeau's Bento project takes the opposite approach, everything
is explicit and without any magic.

http://cournape.github.com/Bento/

It had its 0.1.0 release a week ago.

Please, I don't want to reopen any discussions about Bento here --
distutils2 vs. Bento discussions have been less than constructive in the
past -- I just wanted to make sure everybody is aware that distutils2
isn't the only horse in this race. I don't know if there are others too?

--
Dag Sverre Seljebotn

[David Cournapeau]

On Thu, Jun 21, 2012 at 9:45 AM, Nick Coghlan <ncog...@gmail.com> wrote:

On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough <chr...@plope.com> wrote:
> All of these are really pretty minor issues compared with the main benefit
> of not needing to ship everything with everything else. The killer feature
> is that developers can specify dependencies and users can have those
> dependencies installed automatically in a cross-platform way.  Everything
> else is complete noise if this use case is not served.

Cool. This is the kind of thing we need recorded in a PEP - there's a
lot of domain knowledge floating around in the heads of packaging
folks that needs to be captured so we can know *what the addition of
packaging to the standard library is intended to fix*.

And, like it or not, setuptools has a serious PR problem due to the
fact it monkeypatches the standard library, uses *.pth files to alter
sys.path for every installed application by default, actually *uses*
the ability to run code in *.pth files and has hard to follow
documentation to boot. I *don't* trust that I fully understand the
import system on any machine with setuptools installed, because it is
demonstrably happy to install state to the file system that will
affect *all* Python programs running on the machine.

A packaging PEP needs to explain:
- what needs to be done to eliminate any need for monkeypatching
- what's involved in making sure that *.pth are *not* needed by default
- making sure that executable code in implicitly loaded *.pth files
isn't used *at all*

It is not a PEP, but here are a few reasons why extending distutils is difficult (taken from our experience in the scipy community, which has by far the biggest extension of distutils AFAIK):

http://cournape.github.com/Bento/html/faq.html#why-not-extending-existing-tools-distutils-etc

While I believe setuptools has been a net negative for the scipy community because of the way it works and for the reason you mentioned, I think it is fair to say it is not really possible to do any differently if you rely on distutils.

If specifying install dependencies is the killer feature of setuptools, why can't we have a very simple module that adds the necessary 3 keywords to record it, and let 3rd party tools deal with it as they wish ? That would not even require speciying the format, and would let us more time to deal with the other, more difficult questions.

David

[Chris McDonough]

On 06/21/2012 04:45 AM, Nick Coghlan wrote:
> On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough<chr...@plope.com> wrote:
>> All of these are really pretty minor issues compared with the main benefit
>> of not needing to ship everything with everything else. The killer feature
>> is that developers can specify dependencies and users can have those
>> dependencies installed automatically in a cross-platform way. Everything
>> else is complete noise if this use case is not served.
>
> Cool. This is the kind of thing we need recorded in a PEP - there's a
> lot of domain knowledge floating around in the heads of packaging
> folks that needs to be captured so we can know *what the addition of
> packaging to the standard library is intended to fix*.
>
> And, like it or not, setuptools has a serious PR problem due to the
> fact it monkeypatches the standard library, uses *.pth files to alter
> sys.path for every installed application by default, actually *uses*
> the ability to run code in *.pth files and has hard to follow
> documentation to boot. I *don't* trust that I fully understand the
> import system on any machine with setuptools installed, because it is
> demonstrably happy to install state to the file system that will
> affect *all* Python programs running on the machine.

I don't know about Red Hat but both Ubuntu and Apple put all kinds of
stuff on the default sys.path of the system Python of the box that's
related to their software's concerns only. I don't understand why
people accept this but get crazy about the fact that installing a
setuptools distribution using easy_install changes the default sys.path.

Installing a distribution will change behavior whether or not sys.path
is changed as a result. That's its purpose. The code that runs in the
.pth *file* (there's only one that matters: easy_install.pth) just
mutates sys.path. The end result is this: if you understand how
sys.path works, you understand how eggs work. Each egg is addded to
sys.path. That's all there is to it. It's the same as manually
mutating a global PYTHONPATH, except you don't need to do it.

And note that this is not "setuptools" in general. It's easy_install in
particular. Everything you've brought up so far I think is limited to
easy_install. It doesn't happen when you use pip. I think it's a
mistake that pip doesn't do it, but I think you have to make more
accurate distinctions.

> A packaging PEP needs to explain:
> - what needs to be done to eliminate any need for monkeypatching
> - what's involved in making sure that *.pth are *not* needed by default
> - making sure that executable code in implicitly loaded *.pth files
> isn't used *at all*

I'll note that these goals are completely sideways to any actual
functional goal. It'd be a shame to have monkeypatching going on, but
the other stuff I don't think are reasonable goals. Instead they
represent fears, and those fears just need to be managed.

> I *think* trying to achieve this is actually the genesis of the
> original distribute fork, that subsequently became distutils2 as Tarek
> discovered how much of the complexity in setuptools was actually due
> to the desire to *not* officially fork distutils (and instead
> monkeypatch it, effectively creating a runtime fork).
>
> However, for those of us that weren't directly involved, this is all
> still a strange mystery dealt with by other people. I've cribbed
> together bits and pieces just from following the fragments of the
> discussions that have happened on python-dev and at PyCon US, but if
> we want the madness to ever stop, then *the problems with the status
> quo* need to be written down so that other core developers can
> understand them.

It'd also be useful if other core developers actually tried to use
setuptools in anger. That'd be a good start towards understanding some
of its tradeoffs. People can write this stuff down til they're blue in
the face, but if core devs don't try the stuff, they'll always fear it.

> In fact, I just remembered that Tarek *has* written a lot of this
> down, just not in PEP form: http://www.aosabook.org/en/packaging.html

Cool.

- C

[Tarek Ziadé]

On 6/21/12 11:08 AM, Dag Sverre Seljebotn wrote:
> ...

> David Cournapeau's Bento project takes the opposite approach,
> everything is explicit and without any magic.
>
> http://cournape.github.com/Bento/
>
> It had its 0.1.0 release a week ago.
>
> Please, I don't want to reopen any discussions about Bento here --
> distutils2 vs. Bento discussions have been less than constructive in
> the past -- I just wanted to make sure everybody is aware that
> distutils2 isn't the only horse in this race. I don't know if there
> are others too?
>

That's *exactly* the kind of approach that has made me not want to
continue.

People are too focused on implementations, and 'how distutils sucks'
'how setuptools sucks' etc 'I'll do better' etc

Instead of having all the folks involved in packaging sit down together
and try to fix the issues together by building PEPs describing what
would be a common set of standards, they want to create their own tools
from scratch.

That will not work. And I will say here again what I think we should do
imho:

1/ take all the packaging PEPs and rework them until everyone is happy
(compilation sucks in distutils ? write a PEP !!!)

2/ once we have a consensus, write as many tools as you want, if they
rely on the same standards => interoperability => win.

But I must be naive because everytime I tried to reach people that were
building their own tools to ask them to work with us on the PEPs, all I
was getting was "distutils sucks!'

It worked with the OS packagers guys though, we have built a great data
files managment system in packaging + the versions (386)

> --
> Dag Sverre Seljebotn
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:

> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Nick Coghlan]

On Thu, Jun 21, 2012 at 7:28 PM, David Cournapeau <cour...@gmail.com> wrote:
> If specifying install dependencies is the killer feature of setuptools, why
> can't we have a very simple module that adds the necessary 3 keywords to
> record it, and let 3rd party tools deal with it as they wish ? That would
> not even require speciying the format, and would let us more time to deal
> with the other, more difficult questions.

That low level role is filled by PEP 345 (the latest PyPI metadata
format, which adds the new fields), PEP 376 (local installation
database) and PEP 386 (version numbering schema).

The corresponding packaging submodules are the ones that were being
considered for retention as a reference implementation in 3.3, but are
still slated for removal along with the rest of the package (the
reference implementations will remain available as part of distutils2
on PyPI).

Whatever UI a Python packaging solution presents to a user, it needs
to support those 3 PEPs on the back end for interoperability with
other tools (including, eventually, the packaging module in the
standard library).

Your feedback on the commands/compilers design sounds valuable, and I
would be very interested in seeing a PEP targeting that aspect of the
new packaging module (if you look at the start of this thread, the
failure to improve the compiler API is one of the reasons for pulling
the code from 3.3).

If python-dev ends up playing referee on multiple competing PEPs,
that's not necessarily a bad thing. If a consensus solution doesn't
meet the needs of key parties that aren't well served by existing
approaches (specifically, the scientific community, and enterprise
users that want to be able to translate the plethora of language
specific packaging systems to a common format for internal use to
simplify system administration and configuration management and
auditing), then we may as well not bother and let the status quo
continue indefinitely.

[Tarek Ziadé]

On 6/20/12 2:53 PM, Nick Coghlan wrote:
> On Wed, Jun 20, 2012 at 9:31 PM, Tarek Ziadé<ta...@ziade.org> wrote:
>> Yeah maybe this subset could be left in 3.3
>>
>> and we'd remove packaging-the-installer part (pysetup, commands, compilers)
>>
>> I think it's a good idea !
> OK, to turn this into a concrete suggestion based on the packaging docs.



> Declare stable, include in 3.3
> ------------------------------------------
> packaging.version — Version number classes
> packaging.metadata — Metadata handling
> packaging.markers — Environment markers
> packaging.database — Database of installed distributions

I think that's a good subset.

+1 on all of the things you said after

If you succeed on getting the sci people working on "PEP: Distutils
replacement: Compiling Extension Modules" it will be a big win.

[Oscar Benjamin]

On 21 June 2012 12:48, Chris McDonough <chr...@plope.com> wrote:

On 06/21/2012 04:45 AM, Nick Coghlan wrote:

On Thu, Jun 21, 2012 at 2:44 PM, Chris McDonough<chr...@plope.com>  wrote:

All of these are really pretty minor issues compared with the main benefit
of not needing to ship everything with everything else. The killer feature
is that developers can specify dependencies and users can have those
dependencies installed automatically in a cross-platform way.  Everything
else is complete noise if this use case is not served.

Cool. This is the kind of thing we need recorded in a PEP - there's a
lot of domain knowledge floating around in the heads of packaging
folks that needs to be captured so we can know *what the addition of
packaging to the standard library is intended to fix*.

And, like it or not, setuptools has a serious PR problem due to the
fact it monkeypatches the standard library, uses *.pth files to alter
sys.path for every installed application by default, actually *uses*
the ability to run code in *.pth files and has hard to follow
documentation to boot. I *don't* trust that I fully understand the
import system on any machine with setuptools installed, because it is
demonstrably happy to install state to the file system that will
affect *all* Python programs running on the machine.

I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff on the default sys.path of the system Python of the box that's related to their software's concerns only.  I don't understand why people accept this but get crazy about the fact that installing a setuptools distribution using easy_install changes the default sys.path.

I don't like the particular way that easy_install modifies sys.path so that it can no longer be overridden by PYTHONPATH. For a discussion, see:

http://stackoverflow.com/questions/5984523/eggs-in-path-before-pythonpath-environment-variable

The fact that ubuntu does this for some system ubuntu packages has never bothered me, but the fact that it happens for packages that I install with easy_install has. The typical scenario would be that I:

1) Install some package X with easy_install.

2) Find a bug or some aspect of X that I want to change and checkout the latest version from e.g. github.

3) Try to use PYTHONPATH to test the checked out version and find that easy_install's path modification prevents me from doing so.

4) Run the quickfix script in the stackoverflow question above and consider not using easy_install for X in future.

Oscar

[David Cournapeau]

On Thu, Jun 21, 2012 at 12:58 PM, Nick Coghlan <ncog...@gmail.com> wrote:

On Thu, Jun 21, 2012 at 7:28 PM, David Cournapeau <cour...@gmail.com> wrote:
> If specifying install dependencies is the killer feature of setuptools, why
> can't we have a very simple module that adds the necessary 3 keywords to
> record it, and let 3rd party tools deal with it as they wish ? That would
> not even require speciying the format, and would let us more time to deal
> with the other, more difficult questions.

That low level role is filled by PEP 345 (the latest PyPI metadata
format, which adds the new fields), PEP 376 (local installation
database) and PEP 386 (version numbering schema).

The corresponding packaging submodules are the ones that were being
considered for retention as a reference implementation in 3.3, but are
still slated for removal along with the rest of the package (the
reference implementations will remain available as part of distutils2
on PyPI).

I understand the code is already implemented, but I meant that it may be a good idea to have a simple, self-contained module that does just provide the necessary bits for the "setuptools killer feature", and let competing tools deal with it as they please.

Whatever UI a Python packaging solution presents to a user, it needs
to support those 3 PEPs on the back end for interoperability with
other tools (including, eventually, the packaging module in the
standard library).

Your feedback on the commands/compilers design sounds valuable, and I
would be very interested in seeing a PEP targeting that aspect of the
new packaging module (if you look at the start of this thread, the
failure to improve the compiler API is one of the reasons for pulling
the code from 3.3).

The problem with compilation is not just the way the compiler classes work. It it how they interact with commands and the likes, which ends up being most of the original distutils code. What's wrong with  distutils is the whole underlying model, if one can call that. No PEP will fix the issue if the premise is to work within that model.

There are similar kind of arguments around the extensibility of distutils: it is not just about monkey-patching, but what kind of API you offer to allow for extensibility, and I think the only way to design this sensibly is to work on real packages and iterate, not writing a PEP as a first step.

David

[Nick Coghlan]

On Thu, Jun 21, 2012 at 9:48 PM, Chris McDonough <chr...@plope.com> wrote:
> On 06/21/2012 04:45 AM, Nick Coghlan wrote:
>> And, like it or not, setuptools has a serious PR problem due to the
>> fact it monkeypatches the standard library, uses *.pth files to alter
>> sys.path for every installed application by default, actually *uses*
>> the ability to run code in *.pth files and has hard to follow
>> documentation to boot. I *don't* trust that I fully understand the
>> import system on any machine with setuptools installed, because it is
>> demonstrably happy to install state to the file system that will
>> affect *all* Python programs running on the machine.
>
>
> I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff
> on the default sys.path of the system Python of the box that's related to
> their software's concerns only.  I don't understand why people accept this
> but get crazy about the fact that installing a setuptools distribution using
> easy_install changes the default sys.path.

Because the vendor gets to decide what goes into the base install of
the OS. If I'm using the system Python, then I expect sys.path to
contain the system paths, just as I expect gcc to be able to see the
system include paths. If I don't want that, I'll use virtualenv or a
completely separate Python installation.

However, when I install a new Python package into site-packages it
*should* just sit there and have zero impact on other Python
applications that don't import that package. As soon as someone
installs a *.pth file, however, that's *no longer the case* - every
Python application on that machine will now be scanning additional
paths for modules whether it wants to or not. It's unnecessary
coupling between components that *should* be completely independent of
each other.

Now, *.pth support in the interpreter certainly cannot be blamed on
setuptools, but encouraging use of a packaging format that effectively
requires them certainly can be.

It's similar to the reason why monkeypatching and global environment
variable modifications (including PYTHONPATH) are a problem: as soon
as you start doing that kind of thing, you're introducing coupling
that *shouldn't exist*. If there is no better solution, then sure, do
it as a near term workaround, but that isn't the same as accepting it
as the long term answer.

> Installing a distribution will change behavior whether or not sys.path is
> changed as a result.  That's its purpose.

No it won't. An ordinary package will only change the behaviour of
Python applications that import a package by that name. Other Python
applications will be completely unaffected (as it should be).

> The code that runs in the .pth
> *file* (there's only one that matters: easy_install.pth) just mutates
> sys.path.  The end result is this: if you understand how sys.path works, you
> understand how eggs work.  Each egg is addded to sys.path.  That's all there
> is to it.  It's the same as manually mutating a global PYTHONPATH, except
> you don't need to do it.

Yes, it's the same as mutating PYTHONPATH. That's a similarly bad
system global change. Individual libraries do not have the right to
change the sys.path seen on initialisation by every other Python
application on that system.

> And note that this is not "setuptools" in general.  It's easy_install in
> particular.  Everything you've brought up so far I think is limited to
> easy_install.  It doesn't happen when you use pip.  I think it's a mistake
> that pip doesn't do it, but I think you have to make more accurate
> distinctions.

What part of "PR problem" was unclear? setuptools and easy_install are
inextricably linked in everyone's minds, just like pip and distribute.

>> A packaging PEP needs to explain:
>> - what needs to be done to eliminate any need for monkeypatching
>> - what's involved in making sure that *.pth are *not* needed by default
>> - making sure that executable code in implicitly loaded *.pth files
>> isn't used *at all*
>
> I'll note that these goals are completely sideways to any actual functional
> goal.  It'd be a shame to have monkeypatching going on, but the other stuff
> I don't think are reasonable goals.  Instead they represent fears, and those
> fears just need to be managed.

No, they reflect the mindset of someone with configuration management
and auditing responsibilities for shared systems with multiple
applications installed which may be written in a variety of languages,
not just Python. You may not care about those people, but I do.

> It'd also be useful if other core developers actually tried to use
> setuptools in anger.  That'd be a good start towards understanding some of
> its tradeoffs.  People can write this stuff down til they're blue in the
> face, but if core devs  don't try the stuff, they'll always fear it.

setuptools (or, perhaps, easy_install, although I've seen enough posts
about eggs being uploaded to PyPI to suspect otherwise), encourages
the deployment of system configuration changes that alter the runtime
environment of every single Python application executed on the system.
That's simply not cool.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Nick Coghlan]

On Thu, Jun 21, 2012 at 10:19 PM, David Cournapeau <cour...@gmail.com> wrote:
>
>
> On Thu, Jun 21, 2012 at 12:58 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>>
>> On Thu, Jun 21, 2012 at 7:28 PM, David Cournapeau <cour...@gmail.com>
>> wrote:
>> > If specifying install dependencies is the killer feature of setuptools,
>> > why
>> > can't we have a very simple module that adds the necessary 3 keywords to
>> > record it, and let 3rd party tools deal with it as they wish ? That
>> > would
>> > not even require speciying the format, and would let us more time to
>> > deal
>> > with the other, more difficult questions.
>>
>> That low level role is filled by PEP 345 (the latest PyPI metadata
>> format, which adds the new fields), PEP 376 (local installation
>> database) and PEP 386 (version numbering schema).
>>
>> The corresponding packaging submodules are the ones that were being
>> considered for retention as a reference implementation in 3.3, but are
>> still slated for removal along with the rest of the package (the
>> reference implementations will remain available as part of distutils2
>> on PyPI).
>
>
> I understand the code is already implemented, but I meant that it may be a
> good idea to have a simple, self-contained module that does just provide the
> necessary bits for the "setuptools killer feature", and let competing tools
> deal with it as they please.

If you're genuinely interested in that prospect, I suggest
collaborating with the distutils2 team to extract the four identified
modules (and any necessary support code) as a "distmeta" project on
PyPI:

distmeta.version — Version number classes
distmeta.metadata — Metadata handling
distmeta.markers — Environment markers
distmeta.database — Database of installed distributions

That will allow faster iteration on the core interoperability
standards prior to reincorporation in 3.4, and explicitly decouple
them from the higher level (more contentious) features.

>> Whatever UI a Python packaging solution presents to a user, it needs
>> to support those 3 PEPs on the back end for interoperability with
>> other tools (including, eventually, the packaging module in the
>> standard library).
>>
>> Your feedback on the commands/compilers design sounds valuable, and I
>> would be very interested in seeing a PEP targeting that aspect of the
>> new packaging module (if you look at the start of this thread, the
>> failure to improve the compiler API is one of the reasons for pulling
>> the code from 3.3).
>
>
> The problem with compilation is not just the way the compiler classes work.
> It it how they interact with commands and the likes, which ends up being
> most of the original distutils code. What's wrong with  distutils is the
> whole underlying model, if one can call that. No PEP will fix the issue if
> the premise is to work within that model.

I don't accept the premise that the 3.4 packaging solution must be
restricted to the distutils semantic model. However, no alternative
strategy has been formally presented to python-dev.

Regards,

[Dag Sverre Seljebotn]

On 06/21/2012 01:56 PM, Tarek Ziadé wrote:
> On 6/21/12 11:08 AM, Dag Sverre Seljebotn wrote:
>> ...
>> David Cournapeau's Bento project takes the opposite approach,
>> everything is explicit and without any magic.
>>
>> http://cournape.github.com/Bento/
>>
>> It had its 0.1.0 release a week ago.
>>
>> Please, I don't want to reopen any discussions about Bento here --
>> distutils2 vs. Bento discussions have been less than constructive in
>> the past -- I just wanted to make sure everybody is aware that
>> distutils2 isn't the only horse in this race. I don't know if there
>> are others too?
>>
> That's *exactly* the kind of approach that has made me not want to
> continue.
>
> People are too focused on implementations, and 'how distutils sucks'
> 'how setuptools sucks' etc 'I'll do better' etc
>
> Instead of having all the folks involved in packaging sit down together
> and try to fix the issues together by building PEPs describing what
> would be a common set of standards, they want to create their own tools
> from scratch.

Guido was asked about build issues and scientific software at PyData
this spring, and his take was that "if scientific users have concerns
that are that special, perhaps you just need to go and do your own
thing". Which is what David is doing.

Trailing Q&A session here: http://www.youtube.com/watch?v=QjXJLVINsSA

Generalizing a bit I think it's "web developers" and "scientists"
typically completely failing to see each others' usecases. I don't know
if that bridge can be crossed through mailing list discussion alone. I
know that David tried but came to a point where he just had to
unsubscribe to distutils-sig.

Sometimes design by committee is just what you want, and sometimes
design by committee doesn't work. ZeroMQ, for instance, is a great piece
of software resulting from dropping out of the AQMP committee.

>
> That will not work. And I will say here again what I think we should do
> imho:
>
> 1/ take all the packaging PEPs and rework them until everyone is happy
> (compilation sucks in distutils ? write a PEP !!!)

I think the only way of making scientists happy is to make the build
tool choice arbitrary (and allow the use of waf, scons, cmake, jam, ant,
etc. for the build). After all, many projects contains more C++ and
Fortran code than Python code. (Of course, one could make a PEP saying
that.)

Right now things are so horribly broken for the scientific community
that I'm not sure if one *can* sanely specify PEPs. It's more a question
of playing around and throwing things at the wall and see what sticks --
5 years from now one is perhaps in a position where the problem is
really understood and one can write PEPs.

Perhaps the "web developers" are at the PEP-ing stage already. Great for
you. But the usecases are really different.

Anyway: I really don't want to start a flame-war here. So let's accept
up front that we likely won't agree here; I just wanted to clarify my
position.

(Some context: I might have funding to work 2 months full-time on
distributing Python software on HPC clusters this autumn. It's not
really related to Bento (or distutils though, more of a client tool
using those libraries)

Dag Sverre Seljebotn
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Chris McDonough]

On 06/21/2012 08:21 AM, Nick Coghlan wrote:
>
>> Installing a distribution will change behavior whether or not sys.path is
>> changed as a result. That's its purpose.
>
> No it won't. An ordinary package will only change the behaviour of
> Python applications that import a package by that name. Other Python
> applications will be completely unaffected (as it should be).

If a Python application is effected by a change to sys.path which
doesn't impact modules it uses, then that Python application is plain
broken, because the developer of that application cannot make
assumptions about what a user does to sys.path unrelated to the modules
it requires. This is completely independent of easy_install.

Any Python application is going to be effected by the installation of a
distribution that does impact modules it imports, whether sys.path is
used to change the working set of modules or not.

So what concrete situation are we actually talking about here?

>> The code that runs in the .pth
>> *file* (there's only one that matters: easy_install.pth) just mutates
>> sys.path. The end result is this: if you understand how sys.path works, you
>> understand how eggs work. Each egg is addded to sys.path. That's all there
>> is to it. It's the same as manually mutating a global PYTHONPATH, except
>> you don't need to do it.
>
> Yes, it's the same as mutating PYTHONPATH. That's a similarly bad
> system global change. Individual libraries do not have the right to
> change the sys.path seen on initialisation by every other Python
> application on that system.

Is it reasonable to even assume there is only
one-sys.path-to-rule-them-all? And that users install "the set of
libraries they need" into a common place? This quickly turns into
failure, because Python is used for many, many tasks, and those tasks
sometimes *require conflicting versions of libraries*. This is the root
cause of why virtualenv exists and is popular.

The reason it's disappointing to see OS vendors mutating the default
sys.path is because they put *very old versions of very common
non-stdlib packages* (e.g. zope.interface, lxml) on sys.path by default.
The path is tainted out of the box for anyone who wants to use the
system Python for development of newer software. So at some point they
invariably punt to virtualenv or a virtualenv-like system where the
OS-vendor-provided path is not present.

If Python supported the installation of multiple versions of the same
module and versioned imports, both PYTHONPATH and virtualenv would be
much less important. But given lack of enthusiasm for that, I don't
think it's reasonable to assume there is only one sys.path on every system.

I sympathize, however, with Oscar's report that PYTHONPATH can't the
setuptools-derived path. That's indeed a mistake that a future tool
should not make.

>> And note that this is not "setuptools" in general. It's easy_install in
>> particular. Everything you've brought up so far I think is limited to
>> easy_install. It doesn't happen when you use pip. I think it's a mistake
>> that pip doesn't do it, but I think you have to make more accurate
>> distinctions.
>
> What part of "PR problem" was unclear? setuptools and easy_install are
> inextricably linked in everyone's minds, just like pip and distribute.

Hopefully for the purposes of the discussion, folks here can make the
mental separation between setuptools and easy_install. We can't help
what other folks think in the meantime, certainly not solely by making
technological compromises anyway.

>>> A packaging PEP needs to explain:
>>> - what needs to be done to eliminate any need for monkeypatching
>>> - what's involved in making sure that *.pth are *not* needed by default
>>> - making sure that executable code in implicitly loaded *.pth files
>>> isn't used *at all*
>>
>> I'll note that these goals are completely sideways to any actual functional
>> goal. It'd be a shame to have monkeypatching going on, but the other stuff
>> I don't think are reasonable goals. Instead they represent fears, and those
>> fears just need to be managed.
>
> No, they reflect the mindset of someone with configuration management
> and auditing responsibilities for shared systems with multiple
> applications installed which may be written in a variety of languages,
> not just Python. You may not care about those people, but I do.

I care about deploying Python-based applications to many platforms. You
care about deploying multilanguage-based applications to a single
platform. There's going to be conflict there.

My only comment on that is this: Since this is a problem related to the
installation of Python distributions, it should deal with the problems
that Python developers have more forcefully than non-Python developers
and non-programmers.

>> It'd also be useful if other core developers actually tried to use
>> setuptools in anger. That'd be a good start towards understanding some of
>> its tradeoffs. People can write this stuff down til they're blue in the
>> face, but if core devs don't try the stuff, they'll always fear it.
>
> setuptools (or, perhaps, easy_install, although I've seen enough posts
> about eggs being uploaded to PyPI to suspect otherwise), encourages
> the deployment of system configuration changes that alter the runtime
> environment of every single Python application executed on the system.
> That's simply not cool.

Again, it would help if you tried it in anger. What's the worst that
could happen? You might like it! ;-)

- C

[Tarek Ziadé]

On 6/21/12 2:45 PM, Dag Sverre Seljebotn wrote:
>
> Guido was asked about build issues and scientific software at PyData
> this spring, and his take was that "if scientific users have concerns
> that are that special, perhaps you just need to go and do your own
> thing". Which is what David is doing.
>
> Trailing Q&A session here: http://www.youtube.com/watch?v=QjXJLVINsSA

if you know what you want and have a tool that does it, why bother using
distutils ?

But then, what your community will do with the guy that create packages
with distutils ? just tell him he suck ?

The whole idea is *interoperability*, not the tool used.

>
> Generalizing a bit I think it's "web developers" and "scientists"
> typically completely failing to see each others' usecases. I don't
> know if that bridge can be crossed through mailing list discussion
> alone. I know that David tried but came to a point where he just had
> to unsubscribe to distutils-sig.

I was there, and sorry to be blunt, but he came to tell us we had to
drop distutils because it sucked, and left because we did not follow
that path

>
> Sometimes design by committee is just what you want, and sometimes
> design by committee doesn't work. ZeroMQ, for instance, is a great
> piece of software resulting from dropping out of the AQMP committee.
>
>>
>> That will not work. And I will say here again what I think we should do
>> imho:
>>
>> 1/ take all the packaging PEPs and rework them until everyone is happy
>> (compilation sucks in distutils ? write a PEP !!!)
>
> I think the only way of making scientists happy is to make the build
> tool choice arbitrary (and allow the use of waf, scons, cmake, jam,
> ant, etc. for the build). After all, many projects contains more C++
> and Fortran code than Python code. (Of course, one could make a PEP
> saying that.)
>
> Right now things are so horribly broken for the scientific community
> that I'm not sure if one *can* sanely specify PEPs. It's more a
> question of playing around and throwing things at the wall and see
> what sticks -- 5 years from now one is perhaps in a position where the
> problem is really understood and one can write PEPs.
>
> Perhaps the "web developers" are at the PEP-ing stage already. Great
> for you. But the usecases are really different.

If you sit down and ask your self: "what are the information a python
project should give me so I can compile its extensions ?" I think this
has nothing to do with the tools/implementations.

And if we're able to write down in a PEP this, e.g. the information a
compiler is looking for to do its job, then any tool out there waf,
scons, cmake, jam, ant, etc, can do the job, no ?

>
> Anyway: I really don't want to start a flame-war here. So let's accept
> up front that we likely won't agree here; I just wanted to clarify my
> position.

After 4 years I still don't understand what "we won't agree" means in
this context. *NO ONE* ever ever came and told me : here's what I want
a Python project to describe for its extensions.

Just "we won't agree" or "distutils sucks" :)


Gosh I hope we will overcome this lock one day, and move forward :D

[Nick Coghlan]

On Thu, Jun 21, 2012 at 10:51 PM, Chris McDonough <chr...@plope.com> wrote:
> Is it reasonable to even assume there is only one-sys.path-to-rule-them-all?
> And that users install "the set of libraries they need" into a common place?
>  This quickly turns into failure, because Python is used for many, many
> tasks, and those tasks sometimes *require conflicting versions of
> libraries*.  This is the root cause of why virtualenv exists and is popular.

And why I'm very happy to see pyvenv make it's way into the standard library :)

> I care about deploying Python-based applications to many platforms.  You
> care about deploying multilanguage-based applications to a single platform.
>  There's going to be conflict there.
>
> My only comment on that is this: Since this is a problem related to the
> installation of Python distributions, it should deal with the problems that
> Python developers have more forcefully than non-Python developers and
> non-programmers.

Thanks to venv, there's an alternative available that may be able to
keep both of us happy: split the defaults. For system installs, adopt
a vendor-centric, multi-language,
easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding
*.pth files by unpacking eggs to the file system). For venv installs,
do whatever is most convenient for pure Python developers (e.g.
leaving eggs packed and using *.pth files to extend sys.path within
the venv).

One of Python's great virtues is its role as a glue language, and part
of being an effective glue language is playing well with others. That
should apply to packaging & distribution as well, not just to runtime
bindings to tools written in other languages.

When we add the scientific users into the mix, we're actually getting
to a *third* audience: multi-language developers that want to use
*Python's* packaging utilities for their source and binary
distribution formats.

The Python community covers a broad spectrum of use cases, and I
suspect that's one of the big reasons packaging can get so contentious
- the goals end up being in direct conflict. Currently, I've
identified at least half a dozen significant communities with very
different needs (the names aren't meant to be all encompassing, just
good representatives of each category, and many individuals will span
multiple categories depending on which hat they're wearing at the
time):

Library authors: just want to quickly and easily publish their work on
the Python package index in a way that is discoverable by others and
allows feedback to reach them at their development site

Web developers: creators of Python applications, relying primarily on
other Python software and underlying OS provided functionality,
potentially with some native extensions, that may need to run on
multiple platforms, but can require installation using a language
specific mechanism by technical staff

Rich client developers: creators of Python applications relying
primarily on other Python software and underlying OS provided
functionality, potentially with native extensions, that need to run on
multiple platforms, but must be installed using standard system
utilities for the benefit of non-technical end users

Enterprise developers: creators of Python or mixed language
applications that need to integrate with corporate system
administration policies (including packaging, auditing and
configuration management)

Scientists: creators of Python data analysis and modelling
applications, with complex dependencies on software written in a
variety of other languages and using various build systems

Python embedders: developers that embed a Python runtime inside a
larger application

>> setuptools (or, perhaps, easy_install, although I've seen enough posts
>> about eggs being uploaded to PyPI to suspect otherwise), encourages
>> the deployment of system configuration changes that alter the runtime
>> environment of every single Python application executed on the system.
>> That's simply not cool.
>
> Again, it would help if you tried it in anger.  What's the worst that could
> happen?  You might like it! ;-)

Oh, believe me, if I ever had distribution needs that required the
power and flexibility of setuptools, I would reach for it in a
heartbeat (in fact, I already use it today, albeit for tasks that
ordinary distutils could probably handle). That said, I do get to
cheat though - since I don't need to worry about cross-platform
deployment, I can just use the relevant RPM hooks directly :)

You're right that most of my ire should be directed at the default
behaviour of easy_install rather than at setuptools itself, though. I
shall moderate my expressed opinions accordingly.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[PJ Eby]

On Wed, Jun 20, 2012 at 11:57 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>
> Right - clearly enumerating the features that draw people to use
> setuptools over just using distutils should be a key element in any
> PEP for 3.4
>
> I honestly think a big part of why packaging ended up being incomplete
> for 3.3 is that we still don't have a clearly documented answer to two
> critical questions:
> 1. Why do people choose setuptools over distutils?

Some of the reasons:

* Dependencies
* Namespace packages
* Less boilerplate in setup.py (revision control, data files support, find_packages(), etc.)
* Entry points system for creating extensible applications and frameworks that need runtime plugin discovery
* Command-line script wrappers
* Binary plugin installation system for apps (i.e. dump eggs in a directory and let pkg_resources figure out what to put on sys.path)
* "Test" command
* Easy distribution of (and runtime access to) static data resources

Of these, automatic dependency resolution with as close to 100% backward compatibility for installing other projects on PyPI was almost certainly the #1 factor driving setuptools' initial adoption.  The 20% that drives the 80%, as it were.  The rest are the 80% that brings in the remaining 20%.

 
>
> 2. What's wrong with setuptools that meant the idea of including it
> directly in the stdlib was ultimately dropped and eventually replaced
> with the goal of incorporating distutils2?

Based on the feedback from Python-Dev, I withdrew setuptools from 2.5 because of what I considered valid concerns raised regarding:

1. Lack of available persons besides myself familiar with the code base and design
2. Lack of design documents to remedy #1
3. Lack of unified end-user documentation

And there was no time for me to fix all of that before 2.5 came out, although I did throw together the EggFormats documentation.  After that, the time window where I was being paid (by OSAF) for setuptools improvements came to an end, and other projects started taking precedence.

Since then, setuptools *itself* has become stable legacy code in much the same way that the distutils has: pip, buildout, and virtualenv all built on top of it, as it built on top of the distutils.  Problem #3 remains, but at least now there are other people working on the codebase.

>   If the end goal is "the bulk of the setuptools feature set
> without the problematic features and default behaviours that make
> system administrators break out the torches and pitchforks", then we
> should *write that down* (and spell out the implications) rather than
> assuming that everyone knows the purpose of the exercise.

That's why I brought this up.  ISTM that far too much of the knowledge of what those use cases and implications are, has been either buried in my head or spread out among diverse user communities in the past.

Luckily, a lot of people from those communities are now getting considerably more involved in this effort.  At the time of, say, the 2.5 setuptools question, there wasn't anybody around but me who was able to argue the "why eggs are good and useful" side of the discussion, for example.

(If you look back to the early days of setuptools, I often asked on distutils-sig for people who could help assemble specs for various things...  which I ended up just deciding for myself, because nobody was there to comment on them.  It took *years* of setuptools actually being in the field and used before enough people knew enough to *want* to take part in the design discussions.  The versioning and metadata PEPs were things I asked about many years prior, but nobody knew what they wanted yet, or even knew yet why they should care.)

Similarly, in the years since then, MvL -- who originally argued against all things setuptools at 2.5 time -- actually proposed the original namespace package PEP.

So I don't think it's unfair to say that, seven years ago, the ideas in setuptools were still a few years ahead of their "time".  Today, console script generation, virtual environments, namespace packages, entry point discovery, setup.py-driven testing tools, static file inclusion, etc. are closer to "of course we should have that/everybody uses that" features, rather than esoteric oddities.

That being said, setuptools *itself* is not such a good thing.  It was originally a *private* add-on to distutils (like numpy's distutils extensions) and a prototyping sandbox for additions to the distutils.  (E.g. setuptools features were added to distutils in 2.4 and 2.5.)  I honestly didn't think at the time that I was writing those features (or even the egg stuff), that the *long term* goal would be for those things to be maintained in a separate package.

Instead, I (rather optimistically) assumed that the value of the approaches would be self-evident, and copied the way the other setuptools features were.  (To this day, there are an odd variety of other little experimental "future distutils enhancements" still living in the setuptools code base, like support for building shared libraries to be used in common between multiple C extensions.)

By the way, for an overview of setuptools' components and use cases, and what happened with 2.5, see here:

  http://mail.python.org/pipermail/python-dev/2006-April/064145.html

The plan I proposed was to phase out setuptools and merge its functionality into distutils for 2.6, but as I mentioned above, my available bandwidth to work on the project essentially vanished shortly after the above post; setuptools was pretty much "good enough" for OSAF's needs at the time, and they had other development priorities for my time.

So, if we are to draw any lesson from the past, it would seem to be, "make sure that the people who'll be doing the work are actually going to be available through to the next Python version".

After all, if they are not, it may not much matter whether the code is in the stdlib or not.  ;-)

[Vinay Sajip]

Chris McDonough <chrism <at> plope.com> writes:

> On 06/21/2012 04:45 AM, Nick Coghlan wrote:
> > A packaging PEP needs to explain:
> > - what needs to be done to eliminate any need for monkeypatching
> > - what's involved in making sure that *.pth are *not* needed by default
> > - making sure that executable code in implicitly loaded *.pth files
> > isn't used *at all*
>
> I'll note that these goals are completely sideways to any actual
> functional goal. It'd be a shame to have monkeypatching going on, but
> the other stuff I don't think are reasonable goals. Instead they
> represent fears, and those fears just need to be managed.

Managed how? Whose functional goals? It's good to have something that works here
and now, but surely there's more to it. Presumably distutils worked for some
value of "worked" up until the point where it didn't, and setuptools needed to
improve on it. Oscar's example shows how setuptools is broken for some use
cases. Nor does it consider, for example, the goals of OS distro packagers in
the same way that packaging has tried to. You're encouraging core devs to use
setuptools, but as most seem to agree that distutils is (quick-)sand and
setuptools is built on sand, it's hard to see setuptools as anything other than
a stopgap, the best we have until something better can be devised.

The command-class based design of distutils and hence setuptools doesn't seem to
be something to bet the future on. As an infrastructure concern, this area of
functionality definitely needs to be supported in the stdlib, even if it's a
painful process getting there. The barriers seem more social than technical, but
hopefully the divide-and-conquer-with-multiple-PEPs approach will prevail.

Regards,

Vinay Sajip

[Barry Warsaw]

On Jun 21, 2012, at 07:48 AM, Chris McDonough wrote:

>I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff
>on the default sys.path of the system Python of the box that's related to
>their software's concerns only. I don't understand why people accept this
>but get crazy about the fact that installing a setuptools distribution using
>easy_install changes the default sys.path.

Frankly, I've long thought that distros like Debian/Ubuntu which rely so much
on Python for essential system functions should basically have two Python
stacks. One would be used for just those system functions and the other would
be for application deployment. OTOH, I often hear from application developers
on Ubuntu that they basically have to build up their own stack *anyway* if
they want to ensure they've got the right suite of dependencies. This is
where tools like virtualenv and buildout on the lower end and chef/puppet/juju
on the higher end come into play.

-Barry

[Nick Coghlan]

On Thu, Jun 21, 2012 at 11:31 PM, PJ Eby <p...@telecommunity.com> wrote:
> So, if we are to draw any lesson from the past, it would seem to be, "make
> sure that the people who'll be doing the work are actually going to be
> available through to the next Python version".

Thanks for that write-up - I learned quite a few things I didn't know,
even though I was actually around for 2.5 development (the fact I had
less of a vested interest in packaging issues then probably made a big
difference, too).

> After all, if they are not, it may not much matter whether the code is in
> the stdlib or not.  ;-)

Yeah, I think Tarek had the right idea with working through the slow
painful process of reaching consensus from the bottom up, feature by
feature - we just got impatient and tried to skip to the end without
working through the rest of the list.

It's worth reflecting on the progress we've made so far, and looking
ahead to see what else remains

In the standard library for 3.3:
- native namespace packages (PEP 420)
- native venv support (PEP 405)

Packaging tool interoperability standards as Accepted PEPs (may still
require further tweaks):
- updated PyPI metadata standard (PEP 345)
- PyPI enforced orderable dist versioning standard (PEP 386)
- common dist installation database format (PEP 376)

As I noted earlier in the thread, it would be good to see the
components of distutils2/packaging aimed at this interoperability
level split out as a separate utility library that can more easily be
shared between projects (distmeta was my suggested name for such a
PyPI project)

Other components where python-dev has a role to play as an
interoperability clearing house:

- improved command and compiler extension API

Other components where python-dev has a role to play in smoothing the
entry of beginners into the Python ecosystem:
- a package installer shipped with Python to reduce bootstrapping issues
- a pypi client for the standard library
- dependency graph builder
- reduced boilerplate in package definition (setup.cfg should help there)

Other components where standard library inclusion is a "nice-to-have"
but not critical:
- most of the other convenience features in setuptools

Cheers,

[Alex Clark]

Hi,

On 6/21/12 7:56 AM, Tarek Ziadé wrote:
> On 6/21/12 11:08 AM, Dag Sverre Seljebotn wrote:
>> ...
>> David Cournapeau's Bento project takes the opposite approach,
>> everything is explicit and without any magic.
>>
>> http://cournape.github.com/Bento/
>>
>> It had its 0.1.0 release a week ago.
>>
>> Please, I don't want to reopen any discussions about Bento here --
>> distutils2 vs. Bento discussions have been less than constructive in
>> the past -- I just wanted to make sure everybody is aware that
>> distutils2 isn't the only horse in this race. I don't know if there
>> are others too?
>>
> That's *exactly* the kind of approach that has made me not want to
> continue.
>
> People are too focused on implementations, and 'how distutils sucks'
> 'how setuptools sucks' etc 'I'll do better' etc
>
> Instead of having all the folks involved in packaging sit down together
> and try to fix the issues together by building PEPs describing what
> would be a common set of standards, they want to create their own tools
> from scratch.
>
> That will not work.

But you can't tell someone or some group of folks that, and expect them
to listen. Most times NIH is pejorative[1], but sometimes something
positive comes out of it.

> And I will say here again what I think we should do
> imho:
>
> 1/ take all the packaging PEPs and rework them until everyone is happy
> (compilation sucks in distutils ? write a PEP !!!)
>
> 2/ once we have a consensus, write as many tools as you want, if they
> rely on the same standards => interoperability => win.
>
> But I must be naive because everytime I tried to reach people that were
> building their own tools to ask them to work with us on the PEPs, all I
> was getting was "distutils sucks!'

And that's the best you can do: give your opinion. I understand the
frustration, but we have to let people succeed and/or fail on their own[2].

>
> It worked with the OS packagers guys though, we have built a great data
> files managment system in packaging + the versions (386)

Are you referring to "the" packaging/distutils2 or something else?


Alex


[1] http://en.wikipedia.org/wiki/Not_invented_here
[2]
http://docs.pythonpackages.com/en/latest/advanced.html#buildout-easy-install-vs-virtualenv-pip

>
>
>
>
>> --
>> Dag Sverre Seljebotn
>> _______________________________________________
>> Python-Dev mailing list
>> Pytho...@python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com
>

--
Alex Clark · http://pythonpackages.com

[Barry Warsaw]

On Jun 21, 2012, at 08:51 AM, Chris McDonough wrote:

>The reason it's disappointing to see OS vendors mutating the default sys.path
>is because they put *very old versions of very common non-stdlib packages*
>(e.g. zope.interface, lxml) on sys.path by default. The path is tainted out
>of the box for anyone who wants to use the system Python for development of
>newer software. So at some point they invariably punt to virtualenv or a
>virtualenv-like system where the OS-vendor-provided path is not present.
>
>If Python supported the installation of multiple versions of the same module
>and versioned imports, both PYTHONPATH and virtualenv would be much less
>important. But given lack of enthusiasm for that, I don't think it's
>reasonable to assume there is only one sys.path on every system.

This is really the key insight that should be driving us IMO. From the system
vendor point of view, my job is to ensure the *system* works right, and that
everything written in Python that provides system functionality is compatible
with whatever versions of third party Python packages I provide in a
particular OS version. That's already a hard enough problem, that frankly any
illusions that I can also provide useful versions for higher level
applications that people will deploy on my OS is just madness.

This is why I get lots of people requesting versioned imports, or simply
resorting to venv/buildout/chef/puppet/juju to deploy *their* applications on
the OS. There's just no other sane way to do it.

I do think Python could do better, but obviously it's a difficult problem.
I suspect that having venv support out of the box in 3.3 will go a long way to
solving some class of these problems. I don't know if that will be the *only*
answer.

-Barry

[Chris McDonough]

On 06/21/2012 09:29 AM, Nick Coghlan wrote:
>> My only comment on that is this: Since this is a problem related to the
>> installation of Python distributions, it should deal with the problems that
>> Python developers have more forcefully than non-Python developers and
>> non-programmers.
>
> Thanks to venv, there's an alternative available that may be able to
> keep both of us happy: split the defaults. For system installs, adopt
> a vendor-centric, multi-language,
> easy-to-translate-to-language-neutral-packaging mindset (e.g. avoiding
> *.pth files by unpacking eggs to the file system). For venv installs,
> do whatever is most convenient for pure Python developers (e.g.
> leaving eggs packed and using *.pth files to extend sys.path within
> the venv).

I'd like to agree with this, but I think there's a distinction that
needs to be made here that's maybe not obvious to everyone.

A tool to generate an OS-specific system package from a Python library
project should be unrelated to a Python distribution *installer*.
Instead, you'd use related tools that understood how to unpack the
distribution packaging format to build one or more package structures.
The resulting structures will be processed and then eventually installed
by native OS install tools. But the Python distribution installer (e.g
easy_install, pip, or some future similar tool) would just never come
into play to create those structures. The Python distribution installer
and the OS-specific build tool might share code to introspect and unpack
files from the packaging format, but they'd otherwise have nothing to do
with one another.

This seems like the most reasonable separation of concerns to me anyway,
and I'd be willing to work on the code that would be shared by both the
Python-level installer and by OS-level packaging tools.

> One of Python's great virtues is its role as a glue language, and part
> of being an effective glue language is playing well with others. That

> should apply to packaging& distribution as well, not just to runtime

I think we'll also need to put some limits on the goal independent of
the union of everything all the audiences require.

Here's some scope suggestions that I believe could be shared by all of
the audiences you list above except for embedders; I think that use case
is pretty much separate. It might also leave "rich client developers"
wanting, but no more than they're already wanting.

- Install code that can *later be imported*. This could be pure Python
code or C code which requires compilation. But it's not for the
purpose of compiling and installing completely arbitrary C code to
arbitrary locations, it's ust written for the purpose of compiling C
code which then *lives in the installed distribution* to
provide an importable Python module that lives in the same
distribution with logic.

- Install "console scripts" which are shell-scripts/batch-files
that cause some logic written in Python to get run as a result.
These console scripts are written to sys.prefix + '/{bin/Scripts}'
depending on the platform.

- Install "package resources", which are non-Python source files that
happen to live in package directories.

IOW, an installer should be about installing Python libraries and
supporting files to a well-known location defined by the interpreter or
venv that runs it, not full applications-that-require-persistent-state
which just happen to be written in Python and which require deployment
to arbitrary locations.

You shouldn't expect the Python packaging tools to install an instance
of an application on a system, you should expect them to install enough
code that would allow you to *generate* an instance of such an
application. Most tools make that possible by installing a console
script which can generate a sandbox that can be used to keep application
state.

Hopefully this is preaching to the choir.

>>> setuptools (or, perhaps, easy_install, although I've seen enough posts
>>> about eggs being uploaded to PyPI to suspect otherwise), encourages
>>> the deployment of system configuration changes that alter the runtime
>>> environment of every single Python application executed on the system.
>>> That's simply not cool.
>>
>> Again, it would help if you tried it in anger. What's the worst that could
>> happen? You might like it! ;-)
>
> Oh, believe me, if I ever had distribution needs that required the
> power and flexibility of setuptools, I would reach for it in a
> heartbeat (in fact, I already use it today, albeit for tasks that
> ordinary distutils could probably handle). That said, I do get to
> cheat though - since I don't need to worry about cross-platform
> deployment, I can just use the relevant RPM hooks directly :)

Ideally this is all you'd ever need to care deeply about in an ideal
world, too, given the separation of installer vs.
system-packaging-support-tools outlined above.

> You're right that most of my ire should be directed at the default
> behaviour of easy_install rather than at setuptools itself, though. I
> shall moderate my expressed opinions accordingly.

Woot!

- C

[Nick Coghlan]

On Thu, Jun 21, 2012 at 11:57 PM, Barry Warsaw <ba...@python.org> wrote:
> On Jun 21, 2012, at 07:48 AM, Chris McDonough wrote:
>
>>I don't know about Red Hat but both Ubuntu and Apple put all kinds of stuff
>>on the default sys.path of the system Python of the box that's related to
>>their software's concerns only.  I don't understand why people accept this
>>but get crazy about the fact that installing a setuptools distribution using
>>easy_install changes the default sys.path.
>
> Frankly, I've long thought that distros like Debian/Ubuntu which rely so much
> on Python for essential system functions should basically have two Python
> stacks.  One would be used for just those system functions and the other would
> be for application deployment.  OTOH, I often hear from application developers
> on Ubuntu that they basically have to build up their own stack *anyway* if
> they want to ensure they've got the right suite of dependencies.  This is
> where tools like virtualenv and buildout on the lower end and chef/puppet/juju
> on the higher end come into play.

Yeah, I liked Hynek's method for blending a Python-centric application
development approach with a system packaging centric configuration
management approach: take an entire virtualenv and package *that* as a
single system package.

Another strategy that can work is application specific system package
repos, but you have to be very committed to a particular OS and
packaging system for that approach to make a lot of sense :)

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Dag Sverre Seljebotn]

I'm not sure if I understand. A project can't "give the information
needed to build it". The build system is an integrated piece of the code
and package itself. Making the build of library X work on some ugly HPC
setup Y is part of the development of X.

To my mind a solution looks something like (and Bento is close to this):

Step 1) "Some standard" to do configuration of a package (--prefix and
other what-goes-where options, what libraries to link with, what
compilers to use...)

Step 2) Launch the package's custom build system (may be Unix shell
script or makefile in some cases (sometimes portability is not a goal),
may be a waf build)

Step 3) "Some standard" to be able to cleanly
install/uninstall/upgrade the product of step 2)

An attempt to do Step 2) in a major way in the packaging framework
itself, and have the package just "declare" its C extensions, would not
work. It's fine to have a way in the packaging framework that works for
trivial cases, but it's impossible to create something that works for
every case.

>
> And if we're able to write down in a PEP this, e.g. the information a
> compiler is looking for to do its job, then any tool out there waf,
> scons, cmake, jam, ant, etc, can do the job, no ?
>
>
>>
>> Anyway: I really don't want to start a flame-war here. So let's accept
>> up front that we likely won't agree here; I just wanted to clarify my
>> position.
> After 4 years I still don't understand what "we won't agree" means in
> this context. *NO ONE* ever ever came and told me : here's what I want a
> Python project to describe for its extensions.

That's unfortunate. To be honest, it's probably partly because it's
easier to say what won't work than come with a constructive suggestion.
A lot of people (me included) just use waf/cmake/autotools, and forget
about making the code installable through PyPI or any of the standard
Python tools. Just because that works *now* for us, but we don't have
any good ideas for how to make this into something that works on a wider
scale.

I think David is one of the few who has really dug into the matter and
tried to find something that can both do builds and work through
standard install mechanisms. I can't answer for why you haven't been
able to understand one another.

It may also be an issue with how much one can constructively do on
mailing lists. Perhaps the only route forward is to to bring people
together in person and walk distutils2 people through some hairy
scientific HPC builds (and vice versa).

> Just "we won't agree" or "distutils sucks" :)
>
>
> Gosh I hope we will overcome this lock one day, and move forward :D

Well, me too.

Dag

[Nick Coghlan]

Right, but if the standard library grows a dist installer (and I think
it eventually should), we're going to need to define how it should
behave when executed with the *system* Python.

That will give at least 3 mechanisms for Python code to get onto a system:

1. Python dist -> converter -> system package -> system Python path

2. Python dist -> system Python installer -> system Python path

3. Python dist -> venv Python installer -> venv Python path

While I agree that path 2 should be discouraged for production
systems, I don't think it should be prevented altogether (since it can
be very convenient on personal systems).

As far as the scope of the packaging utilities and what they can
install goes, I think the distutils2 folks have done a pretty good job
of defining that with their static metadata format:
http://alexis.notmyidea.org/distutils2/setupcfg.html#files

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Chris McDonough]

On 06/21/2012 10:30 AM, Nick Coghlan wrote:
>> A tool to generate an OS-specific system package from a Python library
>> project should be unrelated to a Python distribution *installer*. Instead,
>> you'd use related tools that understood how to unpack the distribution
>> packaging format to build one or more package structures. The resulting
>> structures will be processed and then eventually installed by native OS
>> install tools. But the Python distribution installer (e.g easy_install,
>> pip, or some future similar tool) would just never come into play to create
>> those structures. The Python distribution installer and the OS-specific
>> build tool might share code to introspect and unpack files from the
>> packaging format, but they'd otherwise have nothing to do with one another.
>>
>> This seems like the most reasonable separation of concerns to me anyway, and
>> I'd be willing to work on the code that would be shared by both the
>> Python-level installer and by OS-level packaging tools.
>
> Right, but if the standard library grows a dist installer (and I think
> it eventually should), we're going to need to define how it should
> behave when executed with the *system* Python.
>
> That will give at least 3 mechanisms for Python code to get onto a system:
>
> 1. Python dist -> converter -> system package -> system Python path
>
> 2. Python dist -> system Python installer -> system Python path
>
> 3. Python dist -> venv Python installer -> venv Python path
>
> While I agree that path 2 should be discouraged for production
> systems, I don't think it should be prevented altogether (since it can
> be very convenient on personal systems).

I'm not sure under what circumstance 2 and 3 wouldn't do the same thing.
Do you have a concrete idea?

> As far as the scope of the packaging utilities and what they can
> install goes, I think the distutils2 folks have done a pretty good job
> of defining that with their static metadata format:
> http://alexis.notmyidea.org/distutils2/setupcfg.html#files

Yeah definitely a good start.

- C

[Zooko Wilcox-O'Hearn]

On Thu, Jun 21, 2012 at 12:57 AM, Nick Coghlan <ncog...@gmail.com> wrote:
>
> Standard assumptions about the behaviour of site and distutils cease to be valid once setuptools is installed
…

> - advocacy for the "egg" format and the associated sys.path changes that result for all Python programs running on a system

…

> System administrators (and developers that think like system administrators when it comes to configuration management) *hate* what setuptools (and setuptools based installers) can do to their systems.

I have extensive experience with this, including quite a few bug
reports and a few patches in setuptools and distribute, plus
maintaining my own fork of setuptools to build and deploy my own
projects, plus interviewing quite a few Python developers about why
they hated setuptools, plus supporting one of them who hates
setuptools even though he and I use it in a build system
(https://tahoe-lafs.org).

I believe that 80% to 90% of the hatred alluded to above is due to a
single issue: the fact that setuptools causes your Python interpreter
to disrespect the PYTHONPATH, in violation of the documentation in
http://docs.python.org/release/2.7.2/install/index.html#inst-search-path
, which says:

"""
The PYTHONPATH variable can be set to a list of paths that will be
added to the beginning of sys.path. For example, if PYTHONPATH is set
to /www/python:/opt/py, the search path will begin with
['/www/python', '/opt/py']. (Note that directories must exist in order
to be added to sys.path; the site module removes paths that don’t
exist.)
"""

Fortunately, this issue is fixable! I opened a bug report and I and a
others have provided patches that makes setuptools stop doing this
behavior. This makes the above documentation true again. The negative
impact on features or backwards-compatibility doesn't seem to be
great.

http://bugs.python.org/setuptools/issue53

Philip J. Eby provisionally approved of one of the patches, except for
some specific requirement that I didn't really understand how to fix
and that now I don't exactly remember:

http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html

Regards,

Zooko

[Antoine Pitrou]

On Thu, 21 Jun 2012 12:02:58 -0300
"Zooko Wilcox-O'Hearn" <zo...@zooko.com> wrote:
>
> Fortunately, this issue is fixable! I opened a bug report and I and a
> others have provided patches that makes setuptools stop doing this
> behavior. This makes the above documentation true again. The negative
> impact on features or backwards-compatibility doesn't seem to be
> great.
>
> http://bugs.python.org/setuptools/issue53
>
> Philip J. Eby provisionally approved of one of the patches, except for
> some specific requirement that I didn't really understand how to fix
> and that now I don't exactly remember:
>
> http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html

These days, I think you should really target distribute, not setuptools.

Regards

Antoine.

[Chris Lambacher]

Nick Coghlan <ncoghlan <at> gmail.com> writes:

>
> The Python community covers a broad spectrum of use cases, and I
> suspect that's one of the big reasons packaging can get so contentious
> - the goals end up being in direct conflict. Currently, I've
> identified at least half a dozen significant communities with very
> different needs (the names aren't meant to be all encompassing, just
> good representatives of each category, and many individuals will span
> multiple categories depending on which hat they're wearing at the
> time):
>

One set of users not covered by your list is people who need to Cross-Compile
Python to another CPU architecture (i.e. x86 to ARM/PowerPC) for use with
embedded computers. Distutils does not handle this very well. If you want a
recent overview of what these users go through you should see my talk from
PyCon 2012:
http://pyvideo.org/video/682/cross-compiling-python-c-extensions-for-embedde

-Chris

[PJ Eby]

On Jun 21, 2012 11:02 AM, "Zooko Wilcox-O&apos;Hearn" <zo...@zooko.com> wrote:
>
> Philip J. Eby provisionally approved of one of the patches, except for
> some specific requirement that I didn't really understand how to fix
> and that now I don't exactly remember:
>
> http://mail.python.org/pipermail/distutils-sig/2009-January/010880.html
>

I don't remember either; I just reviewed the patch and discussion, and I'm not finding what the holdup was, exactly.  Looking at it now, it looks to me like a good idea...  oh wait, *now* I remember the problem, or at least, what needs reviewing.

Basically, the challenge is that it doesn't allow an .egg in a PYTHONPATH directory to take precedence over that *specific* PYTHONPATH directory.

With the perspective of hindsight, this was purely a transitional concern, since it only *really* mattered for site-packages; anyplace else you could just delete the legacy package if it was a problem.  (And your patch works fine for that case.)

However, for setuptools as it was when you proposed this, it was a potential backwards-compatibility problem.  My best guess is that I was considering the approach for 0.7...  which never got any serious development time.

(It may be too late to fix the issue, in more than one sense.  Even if the problem ceased to be a problem today, nobody's going to re-evaluate their position on setuptools, especially if their position wasn't even based on a personal experience with the issue.)

[PJ Eby]

On Jun 21, 2012 10:12 AM, "Chris McDonough" <chr...@plope.com> wrote:
> - Install "package resources", which are non-Python source files that
>  happen to live in package directories.

I love this phrasing, by the way ("non-Python source files").

A pet peeve of mine is the insistence by some people that such files are "data" and don't belong in package directories, despite the fact that if you gave them a .py extension and added data="""...""" around them, they'd be considered part of the code.  A file's name and internal format aren't what distinguishes code from data; it's the way it's *used* that matters.

I think "packaging" has swung the wrong way on this particular point, and that resources and data files should be distinguished in setup.cfg, with sysadmins *not* being given the option to muck about with resources -- especially not to install them in locations where they might be mistaken for something editable.

[Nick Coghlan]

On Fri, Jun 22, 2012 at 12:59 AM, Chris McDonough <chr...@plope.com> wrote:
> On 06/21/2012 10:30 AM, Nick Coghlan wrote:
>> That will give at least 3 mechanisms for Python code to get onto a system:
>>
>> 1. Python dist ->  converter ->  system package ->  system Python path
>>
>> 2. Python dist ->  system Python installer ->  system Python path
>>
>> 3. Python dist ->  venv Python installer ->  venv Python path
>>
>> While I agree that path 2 should be discouraged for production
>> systems, I don't think it should be prevented altogether (since it can
>> be very convenient on personal systems).
>
>
> I'm not sure under what circumstance 2 and 3 wouldn't do the same thing.  Do
> you have a concrete idea?

Yep, this is what I was talking about in terms of objecting to
installation of *.pth files: I think automatically installing *.pth
files into the system Python path is *wrong* (just like globally
editing PYTHONPATH), and that includes any *.pth files needed for egg
installation.

In a venv however, I assume the entire thing is application specific,
so using *.pth files and eggs for ease of management makes a lot of
sense and I would be fine with using that style of installation by
default.

If the *same* default was going to the used in both places, my
preference would be to avoid *.pth files by default and require them
to be explicitly requested regardless of the nature of the target
environment. I really just wanted to be clear that I don't mind *.pth
files at all in the venv case, because they're not affecting the
runtime state of other applications.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[Chris McDonough]

On 06/21/2012 11:37 AM, PJ Eby wrote:
>
> On Jun 21, 2012 11:02 AM, "Zooko Wilcox-O&apos;Hearn" <zo...@zooko.com

A minor backwards incompat here to fix that issue would be appropriate,
if only to be able to say "hey, that issue no longer exists" to folks
who condemn the entire ecosystem based on that bug. At least, that is,
if there will be another release of setuptools. Is that likely?

- C

[Chris McDonough]

On 06/21/2012 11:45 AM, PJ Eby wrote:
>
> On Jun 21, 2012 10:12 AM, "Chris McDonough" <chr...@plope.com

+1. A good number of the "package resource" files we deploy are not
data files at all. In particular, a lot of them are files which
represent HTML templates. These templates are exclusively the domain of
the software being installed, and considering them explicitly "more
editable" than the Python source they sit next to in the package
structure is a grave mistake. They have exactly the same editability
candidacy as the Python source files they are mixed in with.

[Tarek Ziadé]

On 6/21/12 5:50 PM, Chris McDonough wrote:
> A minor backwards incompat here to fix that issue would be
> appropriate, if only to be able to say "hey, that issue no longer
> exists" to folks who condemn the entire ecosystem based on that bug.
> At least, that is, if there will be another release of setuptools. Is
> that likely?

or simply do that fix in distribute since it's Python 3 compatible --
and have setuptools officially discontinued for the sake of clarity.

[PJ Eby]

Yes. At the very least, there will be updated development snapshots (which are what buildout uses anyway).

(Official releases are in a bit of a weird holding pattern.  distribute's versioning scheme leads to potential confusion: if I release e.g. 0.6.1, then it sounds like it's a lesser version than whatever distribute is up to now.  OTOH, releasing a later version number than distribute implies that I'm supporting their feature enhancements, and I really don't want to add new features to 0.6...  but don't have time right now to clean up all the stuff I started in the 0.7 line either, since I've been *hoping* that the work on packaging would make 0.7 unnecessary.  And let's not even get started on the part where system-installed copies of distribute can prevent people from downloading or installing setuptools in the first place.)

Anyway, changing this in a snapshot release shouldn't be a big concern; the main user of snapshots is buildout, and buildout doesn't use .pth files anyway, it just writes scripts that do sys.path manipulation.  (A better approach, for everything except having stuff importable from the standard interpreter.)

Of course, the flip side is that it means there won't be many people testing the fix.

[Chris McDonough]

On 06/21/2012 12:26 PM, PJ Eby wrote:
> On Thu, Jun 21, 2012 at 11:50 AM, Chris McDonough <chr...@plope.com
> <mailto:chr...@plope.com>> wrote:
>
> On 06/21/2012 11:37 AM, PJ Eby wrote:
>
>
> On Jun 21, 2012 11:02 AM, "Zooko Wilcox-O&apos;Hearn"
> <zo...@zooko.com <mailto:zo...@zooko.com>

> <mailto:zo...@zooko.com <mailto:zo...@zooko.com>>> wrote:
> >
> > Philip J. Eby provisionally approved of one of the patches,
> except for
> > some specific requirement that I didn't really understand how
> to fix
> > and that now I don't exactly remember:
> >
> >

> http://mail.python.org/__pipermail/distutils-sig/2009-__January/010880.html

Welp, I don't want to get in the middle of that whole mess. But maybe
the distribute folks would be kind enough to do a major version bump in
their next release; e.g. 1.67 instead of 0.67. That said, I don't think
anyone would be confused by overlapping version numbers between the two
projects. It's known that they have been diverging for a while.

[Tarek Ziadé]

On 6/21/12 6:44 PM, Chris McDonough wrote:
>
>>
>> Yes. At the very least, there will be updated development snapshots
>> (which are what buildout uses anyway).
>>
>> (Official releases are in a bit of a weird holding pattern.
>> distribute's versioning scheme leads to potential confusion: if I
>> release e.g. 0.6.1, then it sounds like it's a lesser version than
>> whatever distribute is up to now. OTOH, releasing a later version
>> number than distribute implies that I'm supporting their feature
>> enhancements, and I really don't want to add new features to 0.6... but
>> don't have time right now to clean up all the stuff I started in the 0.7
>> line either, since I've been *hoping* that the work on packaging would
>> make 0.7 unnecessary. And let's not even get started on the part where
>> system-installed copies of distribute can prevent people from
>> downloading or installing setuptools in the first place.)
>
>
> Welp, I don't want to get in the middle of that whole mess. But maybe
> the distribute folks would be kind enough to do a major version bump
> in their next release; e.g. 1.67 instead of 0.67. That said, I don't
> think anyone would be confused by overlapping version numbers between
> the two projects.

Oh yeah no problem, if Philip backports all the things we've done like
Py3 compat, and bless more people to maintain setuptools, we can even
discontinue distribute !

If not, I think you are just being joking here -- we don't want to go
back into the lock situation we've suffered for many years were PJE is
the only maintainer then suddenly disappears for a year, telling us no
one that is willing to maintain setuptools is able to do so. (according
to him)

> It's known that they have been diverging for a while.

Yeah the biggest difference is Py3 compat, other than that afaik I don't
think any API has been removed or modified.


In my opinion, distribute is the only project that should go forward
since it's actively maintained and does not suffer from the bus factor.

[Alex Clark]

Hi,

+1. I can't help but cringe when I read this (sorry, PJ Eby!):

"Official releases are in a bit of a weird holding pattern." due to
distribute.

Weren't they in a bit of a weird holding pattern before distribute?
Haven't they always been in a bit of a weird holding pattern?

Let's let setuptools be setuptools and distribute be distribute i.e. as
long as distribute exists, I don't care at all about setuptools' release
schedule (c.f. PIL/Pillow) and I like it that way :-). If one day
setuptools or packaging/distutils2 comes along and fixes everything,
then distribute can cease to exist.



Alex

--
Alex Clark · http://pythonpackages.com

[PJ Eby]

On Thu, Jun 21, 2012 at 1:20 PM, Tarek Ziadé <ta...@ziade.org> wrote:

telling us no one that is willing to maintain setuptools is able to do so. (according to him)

Perhaps there is some confusion or language barrier here: what I said at that time was that the only people who I already *knew* to be capable of taking on full responsibility for *continued development* of setuptools, were not available/interested in the job, to my knowledge.

Specifically, the main people I had in mind were Ian Bicking and/or Jim Fulton, both of whom had developed extensions to or significant chunks of setuptools' functionality themselves, during which they demonstrated exemplary levels of understanding both of the code base and the wide variety of scenarios in which that code base had to operate.  They also both demonstrated conservative, user-oriented design choices, that made me feel comfortable that they would not do anything to disrupt the existing user base, and that if they made any compatibility-breaking changes, they would do so in a way that avoided disruption.  (I believe I also gave Philip Jenvey as an example of someone who, while not yet proven at that level, was someone I considered a good potential candidate as well.)

This was not a commentary on anyone *else's* ability, only on my then-present *knowledge* of clearly-suitable persons and their availability, or lack thereof.

I would guess that the pool of qualified persons is even larger now, but the point is moot: my issue was never about who would "maintain" setuptools, but who would *develop* it.

And I expect that we would at this point agree that future *development* of setuptools is not something either of us are seeking. Rather, we should be seeking to develop tools that can properly supersede it.

This is why I participated in Distutils-SIG discussion of the various packaging PEPs, and hope to see more of them there.

[Chris McDonough]

I'm not too interested in the drama/history of the fork situation so I
don't care whether setuptools has the fix or distribute has it or both
have it, but being able to point at some package which doesn't prevent
folks from overriding sys.path ordering using PYTHONPATH would be a good
thing.

- C

[Tarek Ziadé]

On 6/21/12 7:49 PM, PJ Eby wrote:

On Thu, Jun 21, 2012 at 1:20 PM, Tarek Ziadé <ta...@ziade.org> wrote:

telling us no one that is willing to maintain setuptools is able to do so. (according to him)

Perhaps there is some confusion or language barrier here: what I said at that time was that the only people who I already *knew* to be capable of taking on full responsibility for *continued development* of setuptools, were not available/interested in the job, to my knowledge.

Specifically, the main people I had in mind were Ian Bicking and/or Jim Fulton, both of whom had developed extensions to or significant chunks of setuptools' functionality themselves, during which they demonstrated exemplary levels of understanding both of the code base and the wide variety of scenarios in which that code base had to operate.  They also both demonstrated conservative, user-oriented design choices, that made me feel comfortable that they would not do anything to disrupt the existing user base, and that if they made any compatibility-breaking changes, they would do so in a way that avoided disruption.  (I believe I also gave Philip Jenvey as an example of someone who, while not yet proven at that level, was someone I considered a good potential candidate as well.)

This was not a commentary on anyone *else's* ability, only on my then-present *knowledge* of clearly-suitable persons and their availability, or lack thereof.

Yes, so I double-checked my sentence, I think we are in agreement: you would not let folks that *wanted* to maintain it back then, do it.  Sorry if this was not clear to you.

But let's forget about this, old story I guess.

I would guess that the pool of qualified persons is even larger now, but the point is moot: my issue was never about who would "maintain" setuptools, but who would *develop* it.

And I expect that we would at this point agree that future *development* of setuptools is not something either of us are seeking. Rather, we should be seeking to develop tools that can properly supersede it.

This is why I participated in Distutils-SIG discussion of the various packaging PEPs, and hope to see more of them there.

I definitely agree, and I think your feedback on the various PEPs were very important.

My point is just that, we could (and *should*) in my opinion, merge back setuptools and distribute, just to have a py3-enabled setuptools that is in maintenance mode,

and work on the new stuff in packaging besides it.

the merged setuptools/distribute project could also be the place were we start to do the work to be compatible with the new standards

That's my proposal.


Tarek

[Tarek Ziadé]

On 6/21/12 4:26 PM, Dag Sverre Seljebotn wrote:
>
>> project should give me so I can compile its extensions ?" I think this
>> has nothing to do with the tools/implementations.
> If you sit down and ask your self: "what are the information a python
>
> I'm not sure if I understand. A project can't "give the information
> needed to build it". The build system is an integrated piece of the
> code and package itself. Making the build of library X work on some
> ugly HPC setup Y is part of the development of X.
>
> To my mind a solution looks something like (and Bento is close to this):
>
> Step 1) "Some standard" to do configuration of a package (--prefix
> and other what-goes-where options, what libraries to link with, what
> compilers to use...)
>
> Step 2) Launch the package's custom build system (may be Unix shell
> script or makefile in some cases (sometimes portability is not a
> goal), may be a waf build)
>
> Step 3) "Some standard" to be able to cleanly
> install/uninstall/upgrade the product of step 2)
>
> An attempt to do Step 2) in a major way in the packaging framework
> itself, and have the package just "declare" its C extensions, would
> not work. It's fine to have a way in the packaging framework that
> works for trivial cases, but it's impossible to create something that
> works for every case.

I think we should, as you proposed, list a few projects w/ compilation
needs -- from the simplest to the more complex, then see how a standard
*description* could be used by any tool

>
>>
>> And if we're able to write down in a PEP this, e.g. the information a
>> compiler is looking for to do its job, then any tool out there waf,
>> scons, cmake, jam, ant, etc, can do the job, no ?
>>
>>
>>>
>>> Anyway: I really don't want to start a flame-war here. So let's accept
>>> up front that we likely won't agree here; I just wanted to clarify my
>>> position.
>> After 4 years I still don't understand what "we won't agree" means in
>> this context. *NO ONE* ever ever came and told me : here's what I want a
>> Python project to describe for its extensions.
>
> That's unfortunate. To be honest, it's probably partly because it's
> easier to say what won't work than come with a constructive
> suggestion. A lot of people (me included) just use
> waf/cmake/autotools, and forget about making the code installable
> through PyPI or any of the standard Python tools. Just because that
> works *now* for us, but we don't have any good ideas for how to make
> this into something that works on a wider scale.
>
> I think David is one of the few who has really dug into the matter and
> tried to find something that can both do builds and work through
> standard install mechanisms. I can't answer for why you haven't been
> able to understand one another.
>
> It may also be an issue with how much one can constructively do on
> mailing lists. Perhaps the only route forward is to to bring people
> together in person and walk distutils2 people through some hairy
> scientific HPC builds (and vice versa).

Like versions scheme, I think it's fine if you guys have a more complex
system to build software. But there should be a way to share a common
standard for complation, even if people that uses distutils2 or xxx, are
just doing the dumbest things, like simple C libs compilation.

>
>> Just "we won't agree" or "distutils sucks" :)
>>
>>
>> Gosh I hope we will overcome this lock one day, and move forward :D
>
> Well, me too.

The other thing is, the folks in distutils2 and myself, have zero
knowledge about compilers. That's why we got very frustrated not to see
people with that knowledge come and help us in this area.

So, I reiterate my proposal, and it could also be expressed like this:

1/ David writes a PEP where he describes how Bento interact with a
project -- metadata, description files, etc
2/ Someone from distutils2 completes the PEP by describing how setup.cfg
works wrt Extensions
3/ we see if we can have a common standard even if it's a subset of
bento capabilities

>
> Dag
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:

> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Tarek Ziadé]

On 6/21/12 7:56 PM, Chris McDonough wrote:
> ...

>> think any API has been removed or modified.
>>
>>
>> In my opinion, distribute is the only project that should go forward
>> since it's actively maintained and does not suffer from the bus factor.
> Yeah the biggest difference is Py3 compat, other than that afaik I don't
>
> I'm not too interested in the drama/history of the fork situation

You are the one currently adding drama by asking for a new setuptools
release and saying distribute is diverging.

> so I don't care whether setuptools has the fix or distribute has it or
> both have it, but being able to point at some package which doesn't
> prevent folks from overriding sys.path ordering using PYTHONPATH would
> be a good thing.
>

It has to be in Distribute if we want it in most major Linux distros.

And as I proposed to PJE I think the best thing would be to have a
single project code base, working with Py3 and receiving maintenance
fixes with several maintainers.

Since it's clear we're not going to add feature in any of the projects,
I think we can safely trust a larger list of maintainers, and just keep
the project working until the replacement is used

> - C
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:

> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Paul Moore]

Can I take a step back and make a somewhat different point.

Developer requirements are very relevant, sure. But the most important
requirements are those of the end user. The person who simply wants to
*use* a distribution, couldn't care less how it was built, whether it
uses setuptools, or whatever.

End users should not need packaging tools on their machines.

At the moment, to install from source requires the tools the developer
chooses to use for his convenience (distribute/setuptools,
distutils2/packaging, bento) to be installed on the target machine.
And binary installers are only normally available for code that needs
a C extension, and in that case the developer's choice is still
visible in terms of the binary format provided.

I would argue that we should only put *end user* tools in the stdlib.
- A unified package format, suitable for binaries, but also for pure
Python code that wants to ship that way.
- Installation management tools (download, install, remove, list, and
dependency management) that handle the above package format
- Maybe support in the package format and/or installation tools for
managing "wrapper executables" for executable scripts in distributions

Development tools like distutils2, distribute/setuptools, bento would
*only* be needed on developer machines, and would be purely developer
choice. They would all interact with end users via the
stdlib-supported standard formats. They could live outside the stdlib,
and developers could use whichever tool suited them.

This is a radical idea in that it does not cater for the "zipped up
development directory as a distribution format" mental model that
current Python uses. That model could still work, but only if all the
tools generated a stdlib-supported build definition (which could
simply be a Python script that runs the various compile/copy commands,
plus some compiler support classes in the stdlib) in the same way that
lex/yacc generate C, and projects often distribute the generated C
along with the grammar files.

Legacy support in the form of distutils, converters from bdist_xxx
formats to the new binary format, and maybe pip-style "hide the
madness under a unified interface" tools could support this, either in
the stdlib or as 3rd party tools.

I realise this is probably too radical to happen, but at least, it
might put the debate into context if people try to remember that end
users, as well as package developers, are affected by this (and there
are a lot more end users than package developers...).

Paul.

PS I know that setuptools includes some end-user aspects -
multi-versioning, entry points and optional dependencies, for example.
Maybe these are needed - personally, I have never had a need for any
of these, so I'm not the best person to comment.

[PJ Eby]

On Thu, Jun 21, 2012 at 4:01 PM, Paul Moore <p.f....@gmail.com> wrote:

End users should not need packaging tools on their machines.

Well, unless they're developers.  ;-)  Sometimes, the "end user" is a developer making use of a library.
 

Development tools like distutils2, distribute/setuptools, bento would
*only* be needed on developer machines, and would be purely developer
choice. They would all interact with end users via the
stdlib-supported standard formats. They could live outside the stdlib,
and developers could use whichever tool suited them.

AFAIK, this was the goal behind setup.cfg in packaging, and it's a goal I agree with.

 

This is a radical idea in that it does not cater for the "zipped up
development directory as a distribution format" mental model that
current Python uses. That model could still work, but only if all the
tools generated a stdlib-supported build definition

Again, packaging's setup.cfg is, or should be, this.  I think there are some technical challenges with the current state of setup.cfg, but AFAIK they aren't anything insurmountable.

(Background: the general idea is that setup.cfg contains "hooks", which name Python callables to be invoked at various stages of the process.  These hooks can dynamically add to the setup.cfg data, e.g. to list newly-built files, binaries, etc., as well as to do any actual building.)
 

PS I know that setuptools includes some end-user aspects -
multi-versioning, entry points and optional dependencies, for example.
Maybe these are needed - personally, I have never had a need for any
of these, so I'm not the best person to comment.

Entry points are a developer tool, and cross-project co-ordination facility.  They allow packages to advertise classes, modules, functions, etc. that other projects may wish to import and use in a programmatic way.  For example, a web framework may say, "if you want to provide a page template file format, register an entry point under this naming convention, and we will automatically use it when a template has a matching file extension."  So entry points are not really consumed by end users; libraries and frameworks use them as ways to dynamically co-ordinate with other installed libraries, plugins, etc.

Optional dependencies ("extras"), OTOH, are for end-user convenience: they allow an author to suggest configurations that might be of interest.  Without them, people have to do things like this:

  http://pypi.python.org/pypi/celery-with-couchdb

in order to advertise what else should be installed.  If Celery were instead to list its couchdb and SQLAlchemy requirements as "extras" in setup.py, then one could "easy_install celery[couchdb]" or "easy_install celery[sqla]" instead of needing to register separate project names on PyPI for each of these scenarios.

As it happens, however, two of the most popular setuptools add-ons (pip and buildout) either did not or still do not support "extras", because they were not frequently used.  Unfortunately, this meant that projects had to do things like setup dummy projects on PyPI, because the popular tools didn't support the scenario.

In short, nobody's likely to mourn the passing of extras to any great degree.  They're a nice idea, but hard to bootstrap into use due to the chicken-and-egg problem.  If you don't know what they're for, you won't use them, and without common naming conventions (like mypackage[c_speedups] or mypackage[test_support]), nobody will get used to asking for them.  I think at some point we will end up reinventing them, but essentially the challenge is that they are a generalized solution to a variety of small problems that are not individually very motivating to anybody.  They were only motivating to me in the aggregate because I saw lots of individual people being bothered by their particular variation on the theme of auxiliary dependencies or recommended options.

As for multi-versioning, it's pretty clearly a dead duck, a proof-of-concept that was very quickly obsoleted by buildout and virtualenv.  Buildout is a better implementation of multi-versioning for actual scripts, and virtualenvs work fine for people who haven't yet discovered the joys of buildout.  (I'm a recent buildout convert, in case you can't tell.  ;-) )

[Dag Sverre Seljebotn]

It's not clear to me what you mean by description. Package metadata,
install information or description of what/how to build?

I hope you don't mean the latter, that would be insane...it would
effectively amount to creating a build tool that's both more elegant and
more powerful than any option that's currently already out there.

Assuming you mean the former, that's what David did to create Bento.
Reading and understanding Bento and the design decisions going into it
would be a better use of time than redoing a discussion, and would at
least be a very good starting point.

But anyway, some project types from simple to advanced:

- Simple library using Cython + NumPy C API
- Wrappers around HPC codes like mpi4py, petsc4py
- NumPy
- SciPy (uses Fortran compilers too)
- Library using code generation, Cython, NumPy C API, Fortran 90 code,
some performance tuning with CPU characteristics (instruction set, cache
size, optimal loop structure) decided compile-time

Here's the flip side: If you have zero knowledge about compilers, it's
going to be almost impossible to have a meaningful discussion about a
compilation PEP. It's very hard to discuss standards unless everybody
involved have the necessary prerequisite knowledge. You don't go
discussing details of the Linux kernel without some solid C experience
either.

The necessary prerequisites in this case is not merely "knowledge of
compilers". To avoid repeating mistakes of the past, the prerequisites
for a meaningful discussion is years of hard-worn experience building
software in various languages, on different platforms, using different
build tools.

Look, these problems are really hard to deal with. Myself I have
experience with building 2-3 languages using 2-3 build tools on 2
platforms, and I consider myself a complete novice and usually decide to
trust David's instincts over trying to make up an opinion of my own --
simply because I know he's got a lot more experience than I have.

Theoretically it is possible to separate and isolate concerns so that
one set of people discuss build integration and another set of people
discuss installation. Problem is that all the problems tangle -- in
particular when the starting point is distutils!

That's why *sometimes*, not always, design by committee is the wrong
approach, and one-man-shows is what brings technology forwards.

> So, I reiterate my proposal, and it could also be expressed like this:
>
> 1/ David writes a PEP where he describes how Bento interact with a
> project -- metadata, description files, etc
> 2/ Someone from distutils2 completes the PEP by describing how setup.cfg
> works wrt Extensions
> 3/ we see if we can have a common standard even if it's a subset of
> bento capabilities

bento isn't a build tool, it's a packaging tool, competing directly with
distutils2. It can deal with simple distutils-like builds using a
bundled build tool, and currently has integration with waf for
complicated builds; integration with other build systems will presumably
be added later as people need it (the main point is that bento is
designed for it).

Dag
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev

Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Tarek Ziadé]

On 6/21/12 10:46 PM, Dag Sverre Seljebotn wrote:
...

>> I think we should, as you proposed, list a few projects w/ compilation
>> needs -- from the simplest to the more complex, then see how a standard
>> *description* could be used by any tool
>
> It's not clear to me what you mean by description. Package metadata,
> install information or description of what/how to build?
>
> I hope you don't mean the latter, that would be insane...it would
> effectively amount to creating a build tool that's both more elegant
> and more powerful than any option that's currently already out there.
>
> Assuming you mean the former, that's what David did to create Bento.
> Reading and understanding Bento and the design decisions going into it
> would be a better use of time than redoing a discussion, and would at
> least be a very good starting point.

What I mean is : what would it take to use Bento (or another tool) as
the compiler in a distutils-based project, without having to change the
distutils metadata.

>
> But anyway, some project types from simple to advanced:
>
> - Simple library using Cython + NumPy C API
> - Wrappers around HPC codes like mpi4py, petsc4py
> - NumPy
> - SciPy (uses Fortran compilers too)
> - Library using code generation, Cython, NumPy C API, Fortran 90
> code, some performance tuning with CPU characteristics (instruction
> set, cache size, optimal loop structure) decided compile-time

I'd add:

- A Distutils project with a few ExtensionsThe other thing is, the folks

in distutils2 and myself, have zero
>> knowledge about compilers. That's why we got very frustrated not to see
>> people with that knowledge come and help us in this area.
>
> Here's the flip side: If you have zero knowledge about compilers, it's
> going to be almost impossible to have a meaningful discussion about a
> compilation PEP. It's very hard to discuss standards unless everybody
> involved have the necessary prerequisite knowledge. You don't go
> discussing details of the Linux kernel without some solid C experience
> either.

Consider me as the end user that want to have his 2 C modules compiled
in their Python project.

>
> The necessary prerequisites in this case is not merely "knowledge of
> compilers". To avoid repeating mistakes of the past, the prerequisites
> for a meaningful discussion is years of hard-worn experience building
> software in various languages, on different platforms, using different
> build tools.
>
> Look, these problems are really hard to deal with. Myself I have
> experience with building 2-3 languages using 2-3 build tools on 2
> platforms, and I consider myself a complete novice and usually decide
> to trust David's instincts over trying to make up an opinion of my own
> -- simply because I know he's got a lot more experience than I have.
>
> Theoretically it is possible to separate and isolate concerns so that
> one set of people discuss build integration and another set of people
> discuss installation. Problem is that all the problems tangle -- in
> particular when the starting point is distutils!
>
> That's why *sometimes*, not always, design by committee is the wrong
> approach, and one-man-shows is what brings technology forwards.

I am not saying this should be designed by a commitee, but rather - if
such a tool can be made compatible with simple Distutils project, the
guy behind this tool can probably help on a PEP with feedback from a
larger audience than the sci community.

What bugs me is to say that we live in two separate worlds and cannot
build common pieces. This is not True.

>
>> So, I reiterate my proposal, and it could also be expressed like this:
>>
>> 1/ David writes a PEP where he describes how Bento interact with a
>> project -- metadata, description files, etc
>> 2/ Someone from distutils2 completes the PEP by describing how setup.cfg
>> works wrt Extensions
>> 3/ we see if we can have a common standard even if it's a subset of
>> bento capabilities
>
> bento isn't a build tool, it's a packaging tool, competing directly
> with distutils2. It can deal with simple distutils-like builds using a
> bundled build tool, and currently has integration with waf for
> complicated builds; integration with other build systems will
> presumably be added later as people need it (the main point is that
> bento is designed for it).

I am not interested in Bento-the-tool. I am interested in what such a
tool needs from a project to use it =>

"It can deal with simple distutils-like builds using a bundled build

tool" => If I understand this correctly, does that mean that Bento can
build a distutils project with the distutils Metadata ?

If this is the case it means that there a piece of function that
translates Distutils metadata into something Bento deals with.

That's the part I am interested in for interoperability.

>
> Dag
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:

> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com

[Donald Stufft]

On Thursday, June 21, 2012 at 4:01 PM, Paul Moore wrote:

End users should not need packaging tools on their machines.

Sort of riffing on this idea, I cannot seem to find a specification for what a Python

package actually is. Maybe the first effort should focus on this instead of arguing one

implementation or another.  

As a packager:

    I should not (in general) care what tool (pip, pysetup, easy_install, buildout, whatever) is used

    to install my package, My package should just describe what to do to install itself.

As a end user:

   I should not (in general) care what tool was used to create a package (setuptools, bento, distutils,

   whatever). My tool of choice should look at the package and preform the operations that the package

   says are needed for install.

Ideally the package could have some basic primitives that are enough to tell the package installer

tool what to do to install it, These primitives should be enough to cover the common cases (pure python

modules at the very least, maybe additionally some C modules). Now as others have remarked it would

be insane to attempt to do this in every case as it would involve writing a build system that is more

advanced than anything else existing, so a required primitive would be something that allows calling out

to a specific package decided build system (waf, make, whatever) to handle the build configuration.

The eventual end goal here being to make a package from something that varies from implementation

to implementation to a standardized format that any number of tools can build on top of. It would likely

include some things defining where metadata MUST be defined.

For instance, if metadata in setuptools was "compiled" down to static file, and easy_install, pip et;al

used that static file to install from instead of executing setup.py, then the end user would not have

required setup tools installed and instead any number of tools could have been created that utilized

that data.

[David Cournapeau]

On Thu, Jun 21, 2012 at 10:04 PM, Tarek Ziadé <ta...@ziade.org> wrote:

On 6/21/12 10:46 PM, Dag Sverre Seljebotn wrote:
...

I think we should, as you proposed, list a few projects w/ compilation
needs -- from the simplest to the more complex, then see how a standard
*description* could be used by any tool

It's not clear to me what you mean by description. Package metadata, install information or description of what/how to build?

I hope you don't mean the latter, that would be insane...it would effectively amount to creating a build tool that's both more elegant and more powerful than any option that's currently already out there.

Assuming you mean the former, that's what David did to create Bento. Reading and understanding Bento and the design decisions going into it would be a better use of time than redoing a discussion, and would at least be a very good starting point.

What I mean is : what would it take to use Bento (or another tool) as the compiler in a distutils-based project, without having to change the distutils metadata.

I think there is a misunderstanding of what bento is: bento is not a compiler or anything like that. It is a set of libraries that work together to configure, build and install a python project.

Concretely, in bento, there is

  -  a part that build a packager description (Distribution-like in distutils-parlance) from a bento.info (a bite like setup.cfg)

  - a set of tools of commands around this package description. 

  - a set of "backends" to e.g. use waf to build C extension with full and automatic dependency analysis (rebuild this if this other thing is out of date), parallel builds and configuration. Bento scripts build numpy more efficiently and reliable while being 50 % shorter than our setup.py.

  - a small library to build a distutils-compatible Distribution so that you can write a 3 lines setup.py that takes all its info from bento.info and allow for pip to work.

Now, you could produce a similar package description from the setup.cfg to be fed to bento, but I don't really see the point since AFAIK, bento.info is strictly more powerful as a format than setup.cfg.

Another key point is that the commands around this package description are almost entirely decoupled from each other: this is the hard part, and something that is not really possible to do with the current distutils design in an incremental way.

  - Command don't know about each other and dependencies between commands are *external* to commands. You say command "build" depends on command "configure", those dependencies are resolved at runtime. This allows for 3rd parties to insert new command without interfering with each other.

  - options are registered and handled outside command as well: each command can query any other command options. I believe something similar is now available in distutils2, though. Bento allow to add arbitrary configure options to customize library directories (ala autoconf).

  - bento internally has an explicit "database" of built files, with associated categories, and the build command produces a build "manifest". The build manifest + the build tree defines completely the input for install and installers command. The different binary installers use the same build manifest, and the build manifest is actually designed as to allow lossless convertion between different installers (e.g. wininst <-> msi, egg <-> mpkg on mac, etc…). This is what allows in principle to use make, gyp, etc… to produce this build manifest 

 

"It can deal with simple distutils-like builds using a bundled build tool"  => If I understand this correctly, does that mean that Bento can build a distutils project with the distutils Metadata ?

I think Dag meant that bento has a system where you can basically do

# setup.py

from distutils.core import setup

import bento.distutils

bento.distutils.monkey_patch()

setup()

and this setup.py will automatically build a distutils Distribution populated from bento.info. This allows a bento package to be installable with pip or anything that expected a setup.py

This allows for interoperability without having to depend on all the distutils issues.

David