[Python-Dev] BDFL delegation for PEP 426 (PyPI metadata 1.3)
Nick Coghlan
noticing confusing problems with the current spec that mean I want to
become a *co-author* on the spec, rather than explaining to the
current authors the aspects I object to, until they produce a version
that I'm happy with (this is frustrating for the authors as well,
since several of the problems have been inherited from the previous
version of the spec rather than being new in the current version).
Now, Guido's authored and accepted his own PEPs in the past, but to
date we've avoided letting anyone else do that. Since I *definitely*
want to co-author the new metadata PEP (mostly to address issues with
the version specifier section and to include the *rationale* for
changes, rather than merely listing them as previous versions of the
metadata PEPs have done), that means one of the following needs to
happen:
- someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)
- I get clear approval (perhaps from Guido?) to be both co-author
*and* BDFL-Delegate for PEP 426
Regards,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Guido van Rossum
> In doing the detailed review of PEP 426 as BDFL-Delegate, I keep
> noticing confusing problems with the current spec that mean I want to
> become a *co-author* on the spec, rather than explaining to the
> current authors the aspects I object to, until they produce a version
> that I'm happy with (this is frustrating for the authors as well,
> since several of the problems have been inherited from the previous
> version of the spec rather than being new in the current version).
>
> Now, Guido's authored and accepted his own PEPs in the past, but to
> date we've avoided letting anyone else do that. Since I *definitely*
> want to co-author the new metadata PEP (mostly to address issues with
> the version specifier section and to include the *rationale* for
> changes, rather than merely listing them as previous versions of the
> metadata PEPs have done), that means one of the following needs to
> happen:
>
> - someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)
> - I get clear approval (perhaps from Guido?) to be both co-author
> *and* BDFL-Delegate for PEP 426
right. If you are uncomfortable being PEP-uncle *and* -author, find
another author or another uncle. But since it doesn't affect the
language or library, it's fine with me if you are both. :-)
--
--Guido van Rossum (python.org/~guido)
Nick Coghlan
> I don't know or care much about PyPI metadata, so do what you feel is
> right. If you are uncomfortable being PEP-uncle *and* -author, find
> another author or another uncle. But since it doesn't affect the
> language or library, it's fine with me if you are both. :-)
specific people like Vinay Sajip (distlib), and MvL or Richard Jones
(PyPI) to at least glance over it, as well as giving distutils-sig,
catalog-sig and python-dev one last chance to comment on my revised
draft.
Then, wearing my BDFL-Delegate hat, I'd decide what feedback I
considered worth addressing and what could be safely ignored (or
postponed to the next time the metadata gets updated).
I don't expect anything I want to do to be particularly controversial,
but I think it's worth trying to get it right (even if it delays wheel
support in pip for a few more weeks).
Cheers,
Antoine Pitrou
Nick Coghlan <ncog...@gmail.com> wrote:
> On Sun, Feb 3, 2013 at 4:37 PM, Guido van Rossum <gu...@python.org> wrote:
> > I don't know or care much about PyPI metadata, so do what you feel is
> > right. If you are uncomfortable being PEP-uncle *and* -author, find
> > another author or another uncle. But since it doesn't affect the
> > language or library, it's fine with me if you are both. :-)
>
> To clarify what will happen if I handle both jobs: I'll bug a few
> specific people like Vinay Sajip (distlib), and MvL or Richard Jones
> (PyPI) to at least glance over it, as well as giving distutils-sig,
> catalog-sig and python-dev one last chance to comment on my revised
> draft.
>
> Then, wearing my BDFL-Delegate hat, I'd decide what feedback I
> considered worth addressing and what could be safely ignored (or
> postponed to the next time the metadata gets updated).
>
> I don't expect anything I want to do to be particularly controversial,
> but I think it's worth trying to get it right (even if it delays wheel
> support in pip for a few more weeks).
Regards
Antoine.
Paul Moore
>> I don't expect anything I want to do to be particularly controversial,
>> but I think it's worth trying to get it right (even if it delays wheel
>> support in pip for a few more weeks).
>
> Will wheel be implemented in distutils?
particularly given the general "distutils is frozen" principle). To
clarify:
- The wheel project provides a bdist_wheel command to build wheels,
but this depends on setuptools/distribute. I don't know how hard it
would be to write a version that didn't depend on them.
- The wheel project provides standalone commands to install wheels,
and to convert eggs and wininst installers to wheels.
The pip integration is basically to allow pip to find wheels on PyPI
or any local indexes you have, and to install them via the "pip
install" command. It also incidentally includes pip support for the
new dist-info metadata format (I can't recall which PEP that is,
sorry) simply because that's the format that wheel uses rather than
the legacy egg-info format.
So it's perfectly possible to use wheels right now, without the pip
integration. But the pip developers don't want to integrate the wheel
format just because it exists - they want the assurance that it's an
accepted format supported by PEPs, hence the interest in getting the 3
wheel PEPs (of which the metadata PEP is the first) accepted.
Paul
Antoine Pitrou
Paul Moore <p.f....@gmail.com> wrote:
>
> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.
to integrate support in the stdlib, it doesn't make much sense.
I vote for removing the "distutils is frozen" principle. It was already
destructive enough when packaging was still alive - bug fixes were
actively turned down for the dubious reason that fixing bugs would
break compatibility. Its rationale has become completely obsolete now
that packaging has failed.
Regards
Antoine.
Nick Coghlan
> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.
that before PEP 427 will be accepted, it must either switch to using
S/MIME for signatures and drop support for Java Web Signatures
completely, or else it must contain a compelling rationale for why we
should even be considering a signature scheme that isn't yet an IETF
standard. I take the disclaimer the IETF put on their drafts
seriously: "It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." MvL raised
this concern last time the wheel format was discussed, and, to date,
nothing has happened to address it. JWS *does* look like a neat piece
of technology, but it's just too young to be basing our binary
distribution infrastructure on it (especially as new crypto is, by
default, bad crypto - that's why NIST/NSA hold their multi-year
competitions when they need to come up with new crypto related
algorithms).
The other two PEPs (the new metadata and the version compatibility
tags) are in a much better place. Most of the issues with PEP 426 have
been inherited from the previous version of the metadata, rather than
being related to the changes Daniel needed for the wheel format, and
I've just completed a new draft that should address most of those
problems. It's been a while since I looked closely at the
compatibility tag PEP, but I don't recall their being any significant
problems with it last time it was discussed.
Nick Coghlan
> On Sun, 3 Feb 2013 12:34:36 +0000
> Paul Moore <p.f....@gmail.com> wrote:
>>
>> So it's perfectly possible to use wheels right now, without the pip
>> integration. But the pip developers don't want to integrate the wheel
>> format just because it exists - they want the assurance that it's an
>> accepted format supported by PEPs, hence the interest in getting the 3
>> wheel PEPs (of which the metadata PEP is the first) accepted.
>
> But it's an "accepted format" by whom? If we accept the PEP but refuse
> to integrate support in the stdlib, it doesn't make much sense.
>
> I vote for removing the "distutils is frozen" principle. It was already
> destructive enough when packaging was still alive - bug fixes were
> actively turned down for the dubious reason that fixing bugs would
> break compatibility. Its rationale has become completely obsolete now
> that packaging has failed.
That rationale still holds.
The most likely current candidate for integration into Python 3.4 is
distlib, which, as was proposed when packaging was dropped, is built
incrementally as components are standardised, rather than being a
mish-mash of updated code and flawed design inherited from distutils.
Regards,
Antoine Pitrou
Nick Coghlan <ncog...@gmail.com> wrote:
> On Sun, Feb 3, 2013 at 10:48 PM, Antoine Pitrou <soli...@pitrou.net> wrote:
> > On Sun, 3 Feb 2013 12:34:36 +0000
> > Paul Moore <p.f....@gmail.com> wrote:
> >>
> >> So it's perfectly possible to use wheels right now, without the pip
> >> integration. But the pip developers don't want to integrate the wheel
> >> format just because it exists - they want the assurance that it's an
> >> accepted format supported by PEPs, hence the interest in getting the 3
> >> wheel PEPs (of which the metadata PEP is the first) accepted.
> >
> > But it's an "accepted format" by whom? If we accept the PEP but refuse
> > to integrate support in the stdlib, it doesn't make much sense.
> >
> > I vote for removing the "distutils is frozen" principle. It was already
> > destructive enough when packaging was still alive - bug fixes were
> > actively turned down for the dubious reason that fixing bugs would
> > break compatibility. Its rationale has become completely obsolete now
> > that packaging has failed.
>
> The rationale for the distutils freeze is "don't break setuptools".
> That rationale still holds.
without breaking distribute. What's more, distribute is actually
maintained and can evolve to accomodate the fixes.
As for setuptools (as opposed to distribute), I don't think we should
care anymore.
Regards
Antoine.
Nick Coghlan
> MvL raised this concern last time the wheel format was discussed, and, to date,
> nothing has happened to address it.
previous discussion. Daniel assures me MvL was objecting primarily to
the use of one particular crypto algorithm (which was subsequently
removed from the PEP), rather than the use of JWS in general.
That said, my own concerns about relying on an Internet Draft (when
the IETF specifically say not to do that) remain. It just means
they're a new issue, rather than one continuing over from the previous
discussion.
Tres Seaver
Hash: SHA1
On 02/03/2013 08:09 AM, Antoine Pitrou wrote:
> I'm sure it is perfectly possible to evolve and bugfix distutils
> without breaking distribute. What's more, distribute is actually
> maintained and can evolve to accomodate the fixes.
rabbit hole.
> As for setuptools (as opposed to distribute), I don't think we should
> care anymore.
setuptools remain largely interchangeable, which is the only thing that
makes distribute viable, in the ecosystem sense.
Tres.
- --
===================================================================
Tres Seaver +1 540-429-0999 tse...@palladion.com
Palladion Software "Excellence by Design" http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with undefined - http://www.enigmail.net/
iEYEARECAAYFAlEOZakACgkQ+gerLs4ltQ5N7ACgv/v0GTs7xyQ7I/jvrcu/s2i5
kHcAoJUKigWKZTiVr1ARMLt2xtQF+TAL
=mjAm
-----END PGP SIGNATURE-----
Daniel Holth
The neat thing about wheel is that you can install them without having the software used to build them. So we might try to provide a very simple wheel installer script with Python that did not even depend on DistUtils. You would be able to install pip etc with that tool. There is no need to put wheel directly into DistUtils.
Antoine Pitrou
Daniel Holth <dho...@gmail.com> wrote:
> The neat thing about wheel is that you can install them without having the
> software used to build them. So we might try to provide a very simple wheel
> installer script with Python that did not even depend on DistUtils. You
> would be able to install pip etc with that tool. There is no need to put
> wheel directly into DistUtils.
useful for everyone if that facility was bundled with Python.
Regards
Antoine.
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Paul Moore
> The neat thing about wheel is that you can install them without having the
> software used to build them. So we might try to provide a very simple wheel
> installer script with Python that did not even depend on DistUtils. You
> would be able to install pip etc with that tool. There is no need to put
> wheel directly into DistUtils.
distutils should *also* be simple. As command classes are essentially
independent code, it's not even really going to violate whatever
remains of the "distutils is frozen" philosophy.
Is there any mileage to submitting a patch to the Python stdlib that
implements a bdist_wheel distutils subcommand and a wheel installer in
the stdlib? I'm willing to write that patch based on Daniel's existing
wheel code if no-one else is willing to, but only if there's likely to
be some level of support from python-dev (for example, if Antoine and
Nick are in general agreement on the principle, I'd be willing to
write the code and we could see where the subsequent debate leads).
Paul
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Nick Coghlan
On 4 Feb 2013 00:54, "Paul Moore" <p.f....@gmail.com> wrote:
>
> On 3 February 2013 14:41, Daniel Holth <dho...@gmail.com> wrote:
> > The neat thing about wheel is that you can install them without having the
> > software used to build them. So we might try to provide a very simple wheel
> > installer script with Python that did not even depend on DistUtils. You
> > would be able to install pip etc with that tool. There is no need to put
> > wheel directly into DistUtils.
>
> The other side of the equation, adding a bdist_wheel command to
> distutils should *also* be simple. As command classes are essentially
> independent code, it's not even really going to violate whatever
> remains of the "distutils is frozen" philosophy.
>
> Is there any mileage to submitting a patch to the Python stdlib that
> implements a bdist_wheel distutils subcommand and a wheel installer in
> the stdlib? I'm willing to write that patch based on Daniel's existing
> wheel code if no-one else is willing to, but only if there's likely to
> be some level of support from python-dev (for example, if Antoine and
> Nick are in general agreement on the principle, I'd be willing to
> write the code and we could see where the subsequent debate leads).
Adding a bdist_wheel command makes sense to me, and I agree it is the kind of change that shouldn't cause setuptools any grief. The original trigger for the distutils freeze was a ban on internal refactoring, so new features that don't alter internal APIs should be fine.
My one real concern at this point is the signature scheme. We definitely need one, so dropping it from the PEP isn't an option, but the choice of JWS over S/MIME worries me. The two acceptable options I currently see are to wait until JWS is approved by the IETF, or wait until an S/MIME based wheel implementation is available. I'd love to be able to just accept the wheel format as it stands, but absent a compelling pro-JWS story in the PEP, it isn't going to happen :(
I'd also hold off on a wheel downloader for now, that side of things is likely to be affected if/when distlib gets proposed for 3.4.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Barry Warsaw
>- someone else volunteers to be BDFL-Delegate for PEP 426 (MvL, perhaps?)
I'm not volunteering though, due to lack of time.
Cheers,
-Barry
Éric Araujo
I don’t have a problem with Nick improving the PEP and deciding on its
acceptance.
Le 03/02/2013 07:48, Antoine Pitrou a écrit :
> On Sun, 3 Feb 2013 12:34:36 +0000
> Paul Moore <p.f....@gmail.com> wrote:
>> So it's perfectly possible to use wheels right now, without the pip
>> integration. But the pip developers don't want to integrate the wheel
>> format just because it exists - they want the assurance that it's an
>> accepted format supported by PEPs, hence the interest in getting the 3
>> wheel PEPs (of which the metadata PEP is the first) accepted.
>
> But it's an "accepted format" by whom? If we accept the PEP but refuse
> to integrate support in the stdlib, it doesn't make much sense.
There’s a chicken and egg problem: python-dev could have waited a year
or two to see the wheel format being used and fixed/improved before
deciding to do something in the stdlib, but the pip maintainers wanted
an official blessing first. Accepting the wheel PEP would mean that
python-dev and the community think wheels are an interesting way
forward, and after some time needed to find out what fixes and
improvements are needed, python-dev is ready to add stable, official
support in the stdlib.
(I know Daniel Holth is frustrated by the wait for what he sees as a
simple format that’s been stable since last year, but it’s really not
long for stdlib inclusion, and there *are* things to sort out.)
> I vote for removing the "distutils is frozen" principle.
I’ve also been thinking about that. There have been two exceptions to
the freeze, for ABI flags in extension module names and for pycache
directories. When the stable ABI was added and MvL wanted to change
distutils (I don’t know to do what exactly), Tarek stood firm on the
freeze and asked for any improvement to go into distutils2, and after
MvL said that he would not contibute to an outside project, we merged d2
into the stdlib. Namespace packages did not impact distutils either.
Now that we’ve removed packaging from the stdlib, we have two Python
features that are not supported in the standard packaging system, and I
agree that it is a bad thing for our users.
I’d like to propose a reformulation of the freeze:
- refactorings for the sake of cleanup are still shunned
- fixes to really old bugs that have become the expected behavior are
still avoided
- fixes to follow OS changes are still allowed (we’ve had a number for
Debian multiarch, Apple moving stuff around, Windows manifest options
changes)
- support for Python evolutions that involve totally new code, commands
or setup parameters are now possible (this enables stable API support
as well as a new bdist format)
- behavior changes to track Python behavior changes are now possible
(this enables recognizing namespace packages, unless we decide they
need a new setup parameter)
We’ll probably need to talk this over at PyCon (FYI I won’t be at the
language summit but I’ll take part in the packaging mini-summit planned
thanks to Nick).
Regards
Daniel Holth
My position is that these days distutils doesn't belong in the standard library any more than Django does. So I am mildly opposed to supporting it when you should be using better designed third party tools like Bento or setuptools. Wheel makes it possible for Python to get out of the build tool business. Just install your preferred tools with a concise bootstrap installer.
Éric Araujo
Le 03/02/2013 13:57, Daniel Holth a écrit :
> My position is that these days distutils doesn't belong in the standard
> library any more than Django does.
You can install anything you want, but first you need an installer. I
think that a language needs packaging formats and basic build and
install tools in its stdlib. You can use different tools for
specialized needs or tastes, but I think years and years of frustration
show that people need a basic thing working out of the box.
> So I am mildly opposed to supporting it when you should be using better
> designed third party tools like Bento
Aside: can we keep the discussion civil instead of constantly bashing
distutils and heaping praise on one tool? (It happened with
easy_install before bento.) I’m the first one to recognize that
distutils has deep flaws, but I respect the time and energy that many
people have put into it.
> Wheel makes it possible for Python to get out of the build tool
> business. Just install your preferred tools with a concise bootstrap
> installer.
Looks like we agree that a basic tool able to bootstrap the packaging
story is needed :)
Regards
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com
Ralf Schmitt
> Wheel makes it possible for Python to get out of the build tool
> business. Just install your preferred tools with a concise bootstrap
> installer.
didn't happen. Why do you think it will happen now or am I missing
something?
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Chris Jerdonek
> Le 03/02/2013 07:48, Antoine Pitrou a écrit :
>> I vote for removing the "distutils is frozen" principle.
> I’ve also been thinking about that. There have been two exceptions to
> the freeze, for ABI flags in extension module names and for pycache
> directories. When the stable ABI was added and MvL wanted to change
> distutils (I don’t know to do what exactly), Tarek stood firm on the
> freeze and asked for any improvement to go into distutils2, and after
> MvL said that he would not contibute to an outside project, we merged d2
> into the stdlib. Namespace packages did not impact distutils either.
> Now that we’ve removed packaging from the stdlib, we have two Python
> features that are not supported in the standard packaging system, and I
> agree that it is a bad thing for our users.
>
> I’d like to propose a reformulation of the freeze:
freeze spelled out somewhere?
--Chris
Daniel Holth
Bento is the only available packaging tool to heap praise onto and it is impressive. I am reacting to all the hate heaped on setup tools when I think the underlying DistUtils design is a big part of the problem. My feeling is that stdlib packaging tools should be for bootstrapping and reference, more like wsgiref than django.
Wheel is very similar to egg but tries to solve the original setup tools sin of installing something you don't want (the entire setuptools build system) by omitting the runtime component. The result is a very reviewable couple hundred lines bootstrap installer or "unzip" in a pinch. Its also built upon Python standards that were not available when easy_install was introduced.
Simon Cross
> Bento is the only available packaging tool to heap praise onto and it is
> impressive.
in the Python ecosystem? Not necessarily by incorporating it into
stdlib, but perhaps by officially sanctioning it in other ways
(documentation, PyPI, perhaps getting some helpful hooks / tweaks to
Python itself)?
I don't know the answer to these questions, but if there is a good
solution out there, it would be cool to throw our community weight
behind it.
Schiavo
Simon
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Brian Curtin
<hodgestar...@gmail.com> wrote:
> On Sun, Feb 3, 2013 at 9:42 PM, Daniel Holth <dho...@gmail.com> wrote:
>> Bento is the only available packaging tool to heap praise onto and it is
>> impressive.
>
> If Bento is cool, is there some way we can help it gain more traction
> in the Python ecosystem? Not necessarily by incorporating it into
> stdlib, but perhaps by officially sanctioning it in other ways
> (documentation, PyPI, perhaps getting some helpful hooks / tweaks to
> Python itself)?
>
> I don't know the answer to these questions, but if there is a good
> solution out there, it would be cool to throw our community weight
> behind it.
start telling others to use bento on their own, that's fine. For the
core team to get behind it would probably require a lot of work to
safely stamp it as "the new way...that we don't actually have anything
to do with"
If python-dev officially says "hey, use bento", then it has all sorts
of problems and ends up dying in 6 months, we will look like idiots.
Marcus Smith
The pip integration is basically to allow pip to find wheels on PyPI
or any local indexes you have, and to install them via the "pip
install" command.
PJ Eby
> The rationale for the distutils freeze is "don't break setuptools".
> That rationale still holds.
distutils refactoring broke setuptools, but that it did so in what was
supposed to be a bugfix-only release of Python, via a change to what
appeared to be a public/documented method.
This rationale doesn't apply to major/minor releases of Python, and
the freeze was only supposed to apply to 2.x in any event. (IIRC,
some of the breakage was the result of backporting 3.x distutils
changes to 2.x.)
Anyway, adding features or refactoring distutils is not the problem:
doing it in bugfix releases is. With normal releases, third-party
extenders of distutils at least have the opportunity to test and issue
updates in a timely fashion.
Daniel Holth
$ python wheel-0.14.0-py2.py3-none-any.whl/wheel -h
usage: wheel [-h] {keygen,sign,verify,unpack,install,convert,help} ...
Vinay Sajip
> So it's perfectly possible to use wheels right now, without the pip
> integration. But the pip developers don't want to integrate the wheel
> format just because it exists - they want the assurance that it's an
> accepted format supported by PEPs, hence the interest in getting the 3
> wheel PEPs (of which the metadata PEP is the first) accepted.
once it has been accepted as a standard and the open issues (such as signature
scheme) have been resolved.
Regards,
Vinay Sajip
Daniel Holth
They can be signed with pypi detached signatures already. It works now exactly as for sdist.
The innovation was supposed to be in convenience for the signer, in allowing keys to be trusted per package and for a list of dependencies and the expected signing keys to be shared easily. Does anyone have a concise pure py3 s/mime implementation?
Vinay Sajip
> Bento is the only available packaging tool to heap praise onto and it is
> impressive. I am reacting to all the hate heaped on setup tools when I think
> the underlying DistUtils design is a big part of the problem. My feeling is
> that stdlib packaging tools should be for bootstrapping and reference, more
> like wsgiref than django.
somewhat idiosyncratic source style, David Cournapeau himself points to what he
regards as weaknesses in it[1]. It is no doubt a better build tool than
distutils for NumPy/SciPy users, and that's great. However, from what I could
see, the documentation is not very detailed, making it hard to evaluate, and
much of the build internals seem to be in some other tool called yaku, for which
I couldn't find much documentation.
When I see a sentiment suggesting that "Python should get out of the build tools
business" (paraphrasing what Daniel said, not quoting him), I can certainly see
why the distutils compiler infrastructure would make one think that, if it was
impossible to come up with something better. However, I do feel there is room
for a battery included in this area, as long as it is simple and easier to build
on than distutils has been. Although I have not started to add anything in this
area to distlib, I have spent some time thinking about the problems, and I have
some experimental work in mind.
I don't believe that "hate" is the right word when talking about why setuptools
isn't loved more by python-dev. Since it was extending the flawed distutils
design, it doesn't make sense to have it getting any closer to the stdlib, and
it could perhaps be argued that distutils2/packaging also failed to make
sufficient progress for the same reason.
Certainly, with distlib I've tried to incorporate a lot of the setuptools
functionality which developers find useful - in-package data, package exports
(entry points), wrapping callables with scripts and so on.
Regards,
Vinay Sajip
[1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento
_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Simon Cross
> Bento is interesting, but I wouldn't jump to heap praise onto it. Apart from the
> somewhat idiosyncratic source style, David Cournapeau himself points to what he
> regards as weaknesses in it[1].
[1] http://bento.readthedocs.org/en/latest/faq.html#why-shouldn-t-i-use-bento
Schiavo
Simon
Vinay Sajip
> Looks like we agree that a basic tool able to bootstrap the packaging
> story is needed :)
Agreed. Just because distutils can't easily/reliably build things that are
better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have
the ability to build pure-Python distributions and distributions including C
libs and extensions, with the ability to extend easily by third-party tools. It
just needs to be done in a way which is easy to build on, so the included
battery stays small and simple. Easier said than done, I know :-)
Regards,
Vinay Sajip
Vinay Sajip
> For the record, all the reasons listed at [1] appear trivial.
"at a lower level, a lot of code leaves to be desired" may be trivial if David
is just being self-deprecating, but what if he isn't? Or perhaps that part of
the page is out of date, and needs updating? I can certainly agree with the
"Weak documentation" part of the assessment, but this makes it hard to assess
the package as a whole. Note that I'm not sniping - writing good documentation
is hard work.
Regards,
Vinay Sajip
Daniel Holth
He is being self deprecating. Its also true that python dev can't recommend bento wholesale. That is fine with me.
David Cournapeau
> Simon Cross <hodgestar+pythondev <at> gmail.com> writes:
>
>> For the record, all the reasons listed at [1] appear trivial.
>
> In Bento's author's own words - "Weak documentation", "Mediocre code quality",
> "at a lower level, a lot of code leaves to be desired" may be trivial if David
> is just being self-deprecating, but what if he isn't? Or perhaps that part of
> the page is out of date, and needs updating? I can certainly agree with the
> "Weak documentation" part of the assessment, but this makes it hard to assess
> the package as a whole. Note that I'm not sniping - writing good documentation
> is hard work.
written: it is stated that the focus is on the general architecture
and low-coupling which are the main issues I saw with distutils. Bento
is designed to use multiple build backends (it can use distutils to
build C extensions, or waf, the latter being how numpy/scipy is being
built with bento).
FWIW, I am not in favor of having bento blessed (or any other tool for
that matter). The fundamental mistake of the previous attempts at
packaging has been to formalize too early, or impose de-facto
standards without much specification. That's why wheel and similar
efforts are the way forward: they tackle a narrow but well defined
sub-problem of packaging. Thus, they can be reused by other libraries
to build higher abstractions. They are also less prone to the
'fatigue' that often arise in packaging efforts.
David
Nick Coghlan
Another couple of key pieces are "Setup-Requires-Dist" and the extension mechanism in PEP 426 - they're intended to make it easier to communicate the use of third party tools for building, packaging and installation, as well as making it easier to provide additional metadata for the benefit of those tools.
Cheers,
Nick.
>
> David
> _______________________________________________
> Python-Dev mailing list
> Pytho...@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Vinay Sajip
> You are putting the words out of the context in which those were
> written: it is stated that the focus is on the general architecture
Regards,
Vinay Sajip
David Cournapeau
> David Cournapeau <cournape <at> gmail.com> writes:
>
>> You are putting the words out of the context in which those were
>> written: it is stated that the focus is on the general architecture
>
> OK, no offence was meant. Thanks for the clarification.
David
M.-A. Lemburg
+1 on lifting the freeze from me.
--
Marc-Andre Lemburg
eGenix.com
Professional Python Services directly from the Source (#1, Feb 04 2013)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________
::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
http://www.egenix.com/company/contact/
Chris Jerdonek
<chris.j...@gmail.com> wrote:
> On Sun, Feb 3, 2013 at 10:33 AM, Éric Araujo <mer...@netwok.org> wrote:
>> Le 03/02/2013 07:48, Antoine Pitrou a écrit :
>>> I vote for removing the "distutils is frozen" principle.
>> I’ve also been thinking about that. There have been two exceptions to
>> the freeze, for ABI flags in extension module names and for pycache
>> directories. When the stable ABI was added and MvL wanted to change
>> distutils (I don’t know to do what exactly), Tarek stood firm on the
>> freeze and asked for any improvement to go into distutils2, and after
>> MvL said that he would not contibute to an outside project, we merged d2
>> into the stdlib. Namespace packages did not impact distutils either.
>> Now that we’ve removed packaging from the stdlib, we have two Python
>> features that are not supported in the standard packaging system, and I
>> agree that it is a bad thing for our users.
>>
>> I’d like to propose a reformulation of the freeze:
>
> This could be common knowledge, but is the current formulation of the
> freeze spelled out somewhere?
somewhere like in a PEP? If not, can someone state it precisely (e.g.
what's allowed to change and what's not)?
Thanks,
Éric Araujo
> I asked this earlier, but didn't see a response. Is the freeze
> stated somewhere like in a PEP?
It’s part in notes from the PyCon 2010 Language Summit, part in
unwritten policy in the heads of people involved in distutils bugs these
last years.
> If not, can someone state it precisely (e.g.
> what's allowed to change and what's not)?
Bug fixes are possible; changes to keep working with Python (i.e. pyc
files are generated in pycache directories after 3.2); changes to be
able to build on current OSes (e.g. following Mac dev tools location
change, introduction of Debian multiarch, etc.).
Some bugs have been here for so long that everybody depends on them or
work around them, or they would be extremely painful to fix (e.g. in the
option parsing code) for little benefit, so they are wontfix.
Cleanups, refactorings and improvements were banned by the feature freeze.
Regards
Erik Bray
> Éric Araujo <merwok <at> netwok.org> writes:
>
>> Looks like we agree that a basic tool able to bootstrap the packaging
>> story is needed :)
>
> Agreed. Just because distutils can't easily/reliably build things that are
> better built with SCons/WAF/tup/whatever, doesn't mean that we shouldn't have
> the ability to build pure-Python distributions and distributions including C
> libs and extensions, with the ability to extend easily by third-party tools. It
> just needs to be done in a way which is easy to build on, so the included
> battery stays small and simple. Easier said than done, I know :-)
>
> Regards,
>
> Vinay Sajip
Sorry to revive an old-ish discussion--I'm just catching up on things.
But I just wanted to add that distutils is still pretty okay for
building reasonably complex projects. Although it does not rise to
the level of complexity of Numpy or SciPy, the Astropy project
(https://github.com/astropy/astropy) has managed to put together a
pretty nice build system on top of mostly-plain distutils (it does use
distribute but primarily just for 2to3 support).
This has necessitated a number of hacks to overcome shortcomings and
bugs in distutils, but most of those shortcomings could probably be
fixed in distutils within the framework of a slightly lifted freeze.
But in any case I haven't found it worthwhile to switch to something
like bento when the batteries included in the stdlib have been mostly
Good Enough. Having fewer installation dependencies has also made it
significantly easier for non-advanced users to install. Even the
distribute requirement doesn't add too much overhead, as most users
have it on their systems by default now, and for those who don't
distribute_setup.py works okay.
TL;DR, strong -1 on the stdlib "getting out of the build business".
Also as I think Nick already mentioned one of the wins of
Setup-Requires-Dist is to have a standard way to bring in extra build
requirements (such as bento) so if we have better support for a
feature like that it's not necessary to "bless" any preferred tool.
Erik
Daniel Holth
Nick Coghlan
> TL;DR, strong -1 on the stdlib "getting out of the build business".
> Also as I think Nick already mentioned one of the wins of
> Setup-Requires-Dist is to have a standard way to bring in extra build
> requirements (such as bento) so if we have better support for a
> feature like that it's not necessary to "bless" any preferred tool.
of Python 3.4.
Please, if anyone has any comments on the below, send them to me via
private email. I am only publishing this half-baked version to allay
any concerns people may have that distutils itself might be going away
in the foreseeable future (such concerns are perfectly understandable,
given the many harsh things that have been said about distutils and
the use of setup.py to drive the build process).
(I plan to flesh this out further before the packaging & distribution
mini-summit at PyCon US, but it already reflects the general scheme I
had in the back of my mind when I decided to step up as BDFL-delegate
for Daniel's three PEPs related to the wheel format. There are
obviously lots of details to be worked out on distutils-sig and
catalog-sig, but the big advantage it has over the approach tried with
distutils2 is that individual projects shouldn't have to change much
of anything - this scheme is designed to work so long as I can
convince at least the setuptools, distribute, distutils2, pip, bento
and zc.buildout developers to support it. End users would just need to
update to recent versions of their preferred tools or, in the case of
current users of plain distutils, upgrade to setuptools/distribute or
use the "pip wheel" subcommand to get wheel support in older versions
of Python)
Components:
Archiver: creates sdist source archives for distribution
Builder: creates wheel binary archives for installation or distribution
Uploader: tool for publishing to PyPI and private index servers
Installer: retrieves archives (and their dependencies) and
installs them on a target system
Proposed flow for source distribution:
Development System:
-> Source checkout
-> Installer -> Setup-Requires-Dist dependencies
-> Archiver -> sdist
-> Uploader -> PyPI (or private index server)
Target System:
-> Installer -> sdist + Setup-Requires-Dist dependencies
-> Builder -> wheel
-> Installer -> installed software + Requires-Dist dependencies
Proposed flow for binary distribution:
Development System:
-> Source checkout
-> Installer -> Setup-Requires-Dist dependencies
-> Archiver -> sdist
-> Builder -> wheel
-> Uploader -> PyPI (or private index server)
Target System:
-> Installer -> installed software + Requires-Dist dependencies
Standard library (3.4):
distlib: tools to create Archivers, Uploaders, Builders and Installers
pydist: Uploader/Installer CLI
distutils: Archiver/Builder CLI (via setup.py)
(setup.py will also continue to function as a limited
Uploader/Installer CLI)
Alternate Archiver/Builder tools:
setuptools, distribute, distutils2, bento (and various custom
distutils derivatives)
Alternate Installer tools:
pip, easy_install, zc.buildout
The pydist CLI would likely be deliberately limited to installation
from wheel binary archives. The officially recommended approach to
supporting installation from sdist on systems which do not provide pip
preinstalled would then be "pydist install pip".
Regards,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Nick Coghlan
> I don't know or care much about PyPI metadata, so do what you feel is
> right. If you are uncomfortable being PEP-uncle *and* -author, find
> another author or another uncle. But since it doesn't affect the
> language or library, it's fine with me if you are both. :-)
Based on this, and due to the fact that I think most everyone else
involved in reviewing PEP 426 would have already accepted it by now,
I've reclaimed the BDFL-Delegate role for PEP 426.
I'll be posting the latest draft to distutils-sig for a final sanity
check shortly. (Do people see any point in posting it here as well?
Everyone that will need to support it near-term is over on
distutils-sig, but may not be on python-dev)
Cheers,
Chris Withers
>> As for setuptools (as opposed to distribute), I don't think we should
>> care anymore.
>
> Yes, you need to care. It is *still* true today that distribute and
> setuptools remain largely interchangeable, which is the only thing that
> makes distribute viable, in the ecosystem sense.
+ sys.maxint.
I was very unpopular with the attitude I took to distribute when it
arrived, including it's poor choice of name ;-)
Now, I know my tone was bad, but we are where I feared we're be, to
paraphrase Ned Batchelder's "two problems" job page: I had a problem
with setuptools, so I thought I'd create distribute, now I have two
problems...
I'm excited by wheel and I still have vague hopes that one day the
stdlib might be a collection of packages shipped as a "big distribution"
for those who need it and separately upgradeable for the rest of us...
...but let's make sure we keep caring about the tools that people really
use, which includes both setuptools and distribute.
...or deliberately break them both, at the same time as we fix the
security issues, and provide a workable alternative. People will quickly
migrate ;-)
cheers,
Chris
--
Simplistix - Content Management, Batch Processing & Python Consulting
- http://www.simplistix.co.uk
Nick Coghlan
> ...but let's make sure we keep caring about the tools that people really
> use, which includes both setuptools and distribute.
with PEP 345 & 386, and is also the main reason I've postponed
accepting PEP 426 (PJE and others pointed out a couple of major gaps
that either made it difficult to update pkg_resources to read the new
format, or else made it difficult to generate metadata that existing
releases of pkg_resources would understand).