Packaging and the dependency on distribute/setuptools
Kelsey Hightower
trying to uninstall projects based on distribute/setuptools. The
current install method used in packaging is to "shell" out and call:
# projects based on distutils
python setup.py install --record=/path/to/record_file
# projects based on setuptools
python setup.py install --record=/path/to/record_file --single-
version-externally-managed"
This works and gets the project installed. However, this does not give
us PEP 376 compatibility and adds a dependency on distribute/
setuptools. Another side effect from the methods above, the RECORD
file does not end up in the right place. I am sure we can fix this,
but do we really want to?
To get everything working it seems we will have to do a custom
distribute/setuptools install similar to how pip does things.
https://github.com/pypa/pip/blob/master/pip/req.py#L540
Note, we do not have the above issue with packaging (setup.cfg) based
projects.
At this point it seems we need to pick one of three options:
1. Only support packaging based projects (setup.cfg)
2. Only support packaging and distutils based projects, punt on
projects that require distribute/setuptools.
3. Support all three, packaging, distutils, and distribute/setuptool
based projects.
Pros and Cons
=============
Option 1
--------
Only support packaging based projects(setup.cfg).
Pros
* enforce the new standard, setup.cfg and static metadata.
* clear project scope; one way forward
* no dependencies on 3rd party modules (distribute)
* chance to make a clean break; distutils is still in the stdlib and
pip supports Python 3
Cons
* unable to install most projects day one; will have to defer to 3rd
party tools such as pip and distribute
Option 2
--------
Only support installing packaging and distutils based projects and
punt on projects that require distribute/setuptools.
Pros
* can install more projects than option 1.
* no dependencies on 3rd party modules (distribute)
* promote the new standard with a transition for distutils based
projects
Cons
* still supports setup.py
* causes confusion; can install some projects with a setup.py and not
others
* may prolong the move to setup.cfg and static metadata
* miss our chance for a clean break
Option 3
--------
Support all three, packaging, distutils, and distribute/setuptools
based projects.
Pros
* can install more projects than option 1 and option 2
Cons
* WILL prolong the move to setup.cfg and static metadata. Most people
will continue to use what they know best.
* miss our chance for a clean break
I am leaning toward Option 1 myself, as option 2 will only cause
confusion. Option 3 seems like going backward by requiring distribute
and will prolong the move to setup.cfg and static metadata. Going with
option 1 will require a lot of effort to get the community on board
and start converting to a valid setup.cfg, or providing support for
both, a valid setup.cfg and setup.py.
With packaging and pysetup being new tools we should keep them clean
and focused on the future. This is especially true when good
alternatives exist to provide a transition strategy for most projects.
Thoughts?
RonnyPfannschmidt
instead distribute should be extended to support the new standard and
pip then can install all old packages with the right metadata
as far as i understood this mostly requires writing the dist info dir
in addition to the egg-info dir (RECORDing the content of egg-info as
installed files, so uninstall works)
On Apr 10, 5:06 pm, Kelsey Hightower <kelsey.highto...@gmail.com>
wrote:
Elson Rodriguez
setup.py. There will be pain as projects are ported to use setup.cfg,
however hopefully these user stories can be used to refine Packaging
functionality.
On Apr 10, 11:06 am, Kelsey Hightower <kelsey.highto...@gmail.com>
wrote:
Tarek Ziadé
projects. If we do so, no one will use pysetup because most packages
will not have what is required in setup.cfg. This is very important
for dependencies installations.
Developers will eventually move to distutils2 but that'll take years
and in some case even will never occur: some dependencies we use are
not maintained anymore but still part of frameworks eco-systems, and
their setup.cfg will never be changed.
Another important thing is that we don't enforce a dependency on
setuptools or distribute, it's the project that uses it in its
setup.py that enforces this.
If setuptools is encountered, it's up to the end-user to install
setuptools or distribute.
So I think we need to make sure we're able to install *all* projects,
even if the uninstall story is not fully provided for non-distutils2
bits. otherwise pysetup will stay that labs/prototype installer, that
cannot be used in real life.
Cheers
Tarek
Kelsey Hightower
It seems you are in favor of option 3. If so, I propose we improve on
the distutils1/setuptools/distribute installation process and use
distribute if available. We can borrow the installation method used by
pip in regards to distutils1/setuptools/distribute installs. IMHO,
this is a requirement for option 3 to work, and make pysetup useful in
real life ;)
The docs should note that distribute is not a requirement, but highly
recommended for the reasons listed in your response. We can add
additional checks and feedback when a project requires setuptools/
distribute and setuptools/distribute are not installed.
Alexis Métaireau
Tarek, you made a point: we can't propose a solution which is unable to
install setuptools based distributions. But we can think about a
strategy to support this out of packaging (after all, the goal of
packaging/distutils2 is to provides tools able to deal with the new
metadata formats, defined in PEP 376).
Beside, we are talking about something pip already does, and does well
(supporting setuptools based prokects).
Having a clean implementation of PEP 376, with *no* support of
setuptools can force the projects to rethink their packaging the
PEP-376-ish way. We could start by keeping the compatibility with
setuptools projects via 3rd party tools such as pip (which would use the
distutils2 API to write PEP-376 compliant metadata files, and support
the RECORD file for uninstalling).
This can also have the opposite effect: people will not port their
packaging to the new version. We probably can help them here, and even
do a major part of the work automatically using the setup.py → setup.cfg
convertor on the packages hosted on PyPI.
On the other hand, I fear that providing a tool to install without any
problem setuptools based projects would let the possibility to have
setuptools based projects in the future. If we want to avoid having to
support setuptools in the stdlib for ages, we need to make a break.
I've already made some work to support setuptools based projects in
packaging, but I'm affraid this work is not yet finished. Having a
wrapper around packaging, able to ease the transition to it is probably
a good thing, and having it *not* in the stdlib is also probably a good
idea, as it can evolve easily in the future, once packaging will be
freezed in the stdlib.
I see also this as an opportunity to join efforts between pip and
packaging. It is time to think about the future packaging ecosystem, and
the role of each tool.
Cheers,
Alex
On 11/04/2011 09:11, Tarek Ziadé wrote:
> It's unrealistic not to install distutils1/setuptools/distribute based
> projects. If we do so, no one will use pysetup because most packages
> will not have what is required in setup.cfg. This is very important
> for dependencies installations.
>
> Developers will eventually move to distutils2 but that'll take years
> and in some case even will never occur: some dependencies we use are
> not maintained anymore but still part of frameworks eco-systems, and
> their setup.cfg will never be changed.
>
> Another important thing is that we don't enforce a dependency on
> setuptools or distribute, it's the project that uses it in its
> setup.py that enforces this.
> If setuptools is encountered, it's up to the end-user to install
> setuptools or distribute.
>
> So I think we need to make sure we're able to install*all* projects,
> even if the uninstall story is not fully provided for non-distutils2
> bits. otherwise pysetup will stay that labs/prototype installer, that
> cannot be used in real life.
>
--
Alexis — http://notmyidea.org
Tarek Ziadé
> Hi all, thanks Kelsey for bringing this subject here :-)
>
> Tarek, you made a point: we can't propose a solution which is unable to
> install setuptools based distributions. But we can think about a strategy to
> support this out of packaging (after all, the goal of packaging/distutils2
> is to provides tools able to deal with the new metadata formats, defined in
> PEP 376).
>
> Beside, we are talking about something pip already does, and does well
> (supporting setuptools based prokects).
>
> Having a clean implementation of PEP 376, with *no* support of setuptools
> can force the projects to rethink their packaging the PEP-376-ish way. We
> could start by keeping the compatibility with setuptools projects via 3rd
> party tools such as pip (which would use the distutils2 API to write PEP-376
> compliant metadata files, and support the RECORD file for uninstalling).
That does not work. we provide "pysetup", a tool that installs a
project and its dependencies.
If pysetup encounters a setuptools-based project it has two choices:
1/ install it with a backward-compat installer included
2/ stop the whole process and tell the user that he or she should use
pip or another tool instead
and well, this is going to happen 99% of the time at first, so 2/
means : "don't use pysetup" :D
Now in 1/, we could also just run setup.py to read the metadata and
options, and install the project using the new standards
>
> This can also have the opposite effect: people will not port their packaging
> to the new version. We probably can help them here, and even do a major part
> of the work automatically using the setup.py → setup.cfg convertor on the
> packages hosted on PyPI.
you cannot bet on this. some projects that are intensively used will
never migrate. this is a fact.
>
> On the other hand, I fear that providing a tool to install without any
> problem setuptools based projects would let the possibility to have
> setuptools based projects in the future. If we want to avoid having to
> support setuptools in the stdlib for ages, we need to make a break.
I disagree here. It's always possible to deprecate the setuptools
compat in 3.4 and just issue warnings in 3.3
>
> I've already made some work to support setuptools based projects in
> packaging, but I'm affraid this work is not yet finished. Having a wrapper
> around packaging, able to ease the transition to it is probably a good
> thing, and having it *not* in the stdlib is also probably a good idea, as it
> can evolve easily in the future, once packaging will be freezed in the
> stdlib.
-1. we have 17 months to finish that work, which seems enough to me,
at the stage we're currently are at.
> I see also this as an opportunity to join efforts between pip and packaging.
> It is time to think about the future packaging ecosystem, and the role of
> each tool.
Again, there will be an installer in 3.3, and it needs to install
existing projects. The pip people can help us with this installer and
provide extra features on their own, but I am opposed to push the
installer work effort from packaging outside the stdlib.
> Cheers,
> Alex
>
> On 11/04/2011 09:11, Tarek Ziadé wrote:
>>
>> It's unrealistic not to install distutils1/setuptools/distribute based
>> projects. If we do so, no one will use pysetup because most packages
>> will not have what is required in setup.cfg. This is very important
>> for dependencies installations.
>>
>> Developers will eventually move to distutils2 but that'll take years
>> and in some case even will never occur: some dependencies we use are
>> not maintained anymore but still part of frameworks eco-systems, and
>> their setup.cfg will never be changed.
>>
>> Another important thing is that we don't enforce a dependency on
>> setuptools or distribute, it's the project that uses it in its
>> setup.py that enforces this.
>> If setuptools is encountered, it's up to the end-user to install
>> setuptools or distribute.
>>
>> So I think we need to make sure we're able to install*all* projects,
>> even if the uninstall story is not fully provided for non-distutils2
>> bits. otherwise pysetup will stay that labs/prototype installer, that
>> cannot be used in real life.
>>
>
>
> --
> Alexis — http://notmyidea.org
>
--
Tarek Ziadé | http://ziade.org
Kelsey Hightower
for some projects, add would also break similar to /2, so 1/ means:
"dont use pysetup" to some people. I think we have to go all the way
and use setuptools to install projects that rely on setuptools. This
should not be much work, but we need to agree that it's ok to use
setuptools in the stdlib. We can recommend that people install
distribute to be sure they can install all projects. Make distribute a
soft dependency and warn people that it's required but not installed
when we run into a project that requires setuptools.
I guess we really need a decision.
Do we support setuptools in packaging or not?
Do we only attempt to read the metadata from setup.py, try the
install, and if that fails recommend using pip or some other warning
to that effect?
Since I respect Tarek as the project lead, I will support his
"decision" and get back to work :)
Alexis Métaireau
>> can force the projects to rethink their packaging the PEP-376-ish way. We
>> could start by keeping the compatibility with setuptools projects via 3rd
>> party tools such as pip (which would use the distutils2 API to write PEP-376
>> compliant metadata files, and support the RECORD file for uninstalling).
>
> That does not work. we provide "pysetup", a tool that installs a
> project and its dependencies.
>
> If pysetup encounters a setuptools-based project it has two choices:
>
> 1/ install it with a backward-compat installer included
> 2/ stop the whole process and tell the user that he or she should use
> pip or another tool instead
>
> and well, this is going to happen 99% of the time at first, so 2/
> means : "don't use pysetup" :D
>
> Now in 1/, we could also just run setup.py to read the metadata and
> options, and install the project using the new standards
Yes, that's how it is working by now.
>> This can also have the opposite effect: people will not port their packaging
>> to the new version. We probably can help them here, and even do a major part
>> of the work automatically using the setup.py → setup.cfg convertor on the
>> packages hosted on PyPI.
>
> you cannot bet on this. some projects that are intensively used will
> never migrate. this is a fact.
Friends, we'll get rid of setup.py and setuptools one day! Maybe this
day is not tomorrow, but still let's do what's needed to make it happen
once :-)
>> On the other hand, I fear that providing a tool to install without any
>> problem setuptools based projects would let the possibility to have
>> setuptools based projects in the future. If we want to avoid having to
>> support setuptools in the stdlib for ages, we need to make a break.
>
> I disagree here. It's always possible to deprecate the setuptools
> compat in 3.4 and just issue warnings in 3.3
Okay for warnings + deprecation in 3.4 then. Seems a reasonable plan to
push setuptools and exotic formats out. We need that or we'll deal
forever with setup.py files and setuptools compliance...
>> I've already made some work to support setuptools based projects in
>> packaging, but I'm affraid this work is not yet finished. Having a wrapper
>> around packaging, able to ease the transition to it is probably a good
>> thing, and having it *not* in the stdlib is also probably a good idea, as it
>> can evolve easily in the future, once packaging will be freezed in the
>> stdlib.
>
> -1. we have 17 months to finish that work, which seems enough to me,
> at the stage we're currently are at.
That's true. I'm wondering for the future of packaging, but we'll
probably have something strong enough at this stage, and 17month will be
plenty.
>> I see also this as an opportunity to join efforts between pip and packaging.
>> It is time to think about the future packaging ecosystem, and the role of
>> each tool.
>
> Again, there will be an installer in 3.3, and it needs to install
> existing projects. The pip people can help us with this installer and
> provide extra features on their own, but I am opposed to push the
> installer work effort from packaging outside the stdlib.
Anyway, we should have a support for setuptools somewhere. Having it in
the stdlib + deprecation process seems a good tradeoff, let's go for it.
Kelsey Hightower
Sent from my iPhone
Carl Meyer
Sorry for jumping in a bit late here - I'm in the midst of moving across
the country and haven't had much online time recently.
I agree with Tarek that pysetup has to be able to install legacy
projects or it won't be used.
More specifically:
On 04/10/2011 11:06 AM, Kelsey Hightower wrote:
> While working on the new pysetup commands I ran into some issues when
> trying to uninstall projects based on distribute/setuptools. The
> current install method used in packaging is to "shell" out and call:
>
> # projects based on distutils
>
> python setup.py install --record=/path/to/record_file
>
> # projects based on setuptools
>
> python setup.py install --record=/path/to/record_file --single-
> version-externally-managed"
I think this is basically the right approach for handling legacy
projects in pysetup, and should be kept with tweaks as needed.
One tweak that I think would be worth it is to add a .egg-info directory
for distutils1 installs (and yes I think it should be called .egg-info,
like a setuptools --single-version-externally-managed install, to keep
legacy metadata clearly distinguished from dist-info-style metadata),
simply so that we can place a record file there and allow uninstalls to
work.
It is tempting to use the setuptools-import hack like pip does so that
even d1 projects get installed setuptools-style by default, and remove
the need for doing this manually ourselves. I reluctantly have to agree
with Tarek, though, that this is not a good idea. There is an important
distinction between pysetup itself relying on setuptools/distribute, and
someone's setup.py relying on it; the latter is unavoidable, the former
should be avoided.
> This works and gets the project installed. However, this does not give
> us PEP 376 compatibility
I've toyed with the idea of trying to achieve PEP 376 compatible
installs from legacy projects by installing to a temporary sandbox and
then copying the install over with "fixes" to make it PEP 376
compatible. But to be honest, I think that would be a lot of work, would
probably break with some projects, and isn't worth the hassle. I think
we should just accept that legacy projects will get installed in a
legacy way, and try to enhance that just as much as needed for the
features we want (i.e. uninstall).
> and adds a dependency on distribute/setuptools.
Not directly. Only if the project's setup.py depends on it.
> Another side effect from the methods above, the RECORD
> file does not end up in the right place. I am sure we can fix this,
> but do we really want to?
Note that the --record option of distutils generates just a plain list
of files, not a proper PEP 376 RECORD file with checksums. So I think
this file should not be misleadingly named RECORD; it should be named
something else instead. I propose that this legacy version of the record
file be named installed-files.txt and placed in the egg-info directory,
because that's what pip does currently and there's no reason to
gratuitously break backwards-compatibility of install format for legacy
projects.
This of course requires adding an egg-info dir for distutils installs
which would be installed without any metadata at all.
> To get everything working it seems we will have to do a custom
> distribute/setuptools install similar to how pip does things.
>
> https://github.com/pypa/pip/blob/master/pip/req.py#L540
Like I said, I think this is probably not a good idea for pysetup as it
means pysetup itself would depend on setuptools, even for installing
pure distutils projects.
When you say "to get everything working" what all are you thinking of? I
think pysetup can manually "fake" just enough of a .egg-info directory
for distutils1 installations to allow uninstall to work, without needing
the setuptools-import hack - what else are you thinking might be
problematic?
Carl
Kelsey Hightower
> Hi Kelsey et al,
>
> Sorry for jumping in a bit late here - I'm in the midst of moving across
> the country and haven't had much online time recently.
>
> I agree with Tarek that pysetup has to be able to install legacy
> projects or it won't be used.
> I think this is basically the right approach for handling legacy
> projects in pysetup, and should be kept with tweaks as needed.
really easy to maintain.
> One tweak that I think would be worth it is to add a .egg-info directory
> for distutils1 installs (and yes I think it should be called .egg-info,
> like a setuptools --single-version-externally-managed install, to keep
> legacy metadata clearly distinguished from dist-info-style metadata),
> simply so that we can place a record file there and allow uninstalls to
> work.
>
> It is tempting to use the setuptools-import hack like pip does so that
> even d1 projects get installed setuptools-style by default, and remove
> the need for doing this manually ourselves. I reluctantly have to agree
> with Tarek, though, that this is not a good idea. There is an important
> distinction between pysetup itself relying on setuptools/distribute, and
> someone's setup.py relying on it; the latter is unavoidable, the former
> should be avoided.
>
> installs from legacy projects by installing to a temporary sandbox and
> then copying the install over with "fixes" to make it PEP 376
> compatible. But to be honest, I think that would be a lot of work, would
> probably break with some projects, and isn't worth the hassle. I think
> we should just accept that legacy projects will get installed in a
> legacy way, and try to enhance that just as much as needed for the
> features we want (i.e. uninstall).
>
> of files, not a proper PEP 376 RECORD file with checksums. So I think
> this file should not be misleadingly named RECORD; it should be named
> something else instead. I propose that this legacy version of the record
> file be named installed-files.txt and placed in the egg-info directory,
> because that's what pip does currently and there's no reason to
> gratuitously break backwards-compatibility of install format for legacy
> projects.
>
> This of course requires adding an egg-info dir for distutils installs
> which would be installed without any metadata at all.
>
> means pysetup itself would depend on setuptools, even for installing
> pure distutils projects.
>
> When you say "to get everything working" what all are you thinking of? I
> think pysetup can manually "fake" just enough of a .egg-info directory
> for distutils1 installations to allow uninstall to work, without needing
> the setuptools-import hack - what else are you thinking might be
> problematic?
info to create the .egg-info dir and installed-files.txt, then we
might have enough
info for PEP 376.
Also, since packaging is an updated version of distutils, we could add
support for PEP 376 in packaging. All we need to do next is update
distribute to be PEP 376 compliant, then we are all set.
Carl Meyer
On 04/13/2011 06:16 AM, Kelsey Hightower wrote:
> I agree, we can skip the setuptools-import hack, but If we have enough
> info to create the .egg-info dir and installed-files.txt, then we
> might have enough
> info for PEP 376.
Hmm, that's probably true. Generating a proper PEP 376 RECORD file based
on the file generated by --record isn't hard; just iterate over the
listed installed files and get their md5sum. And it looks like metadata
version 1.2 (PEP 345) is pretty much backwards-compatible with previous
metadata versions (it just adds new fields), so creating METADATA
shouldn't be an issue either.
If we do this for distutils1 projects, it really seems like we ought to
do it for distribute-based projects as well, but I'm not sure how that
ought to interact with the legacy .egg-info dir that distribute will
create with --single-version-externally-managed. Do we leave the
.egg-info in place and also create a .dist-info with PEP-376-style
metadata in it? Or do we copy setuptools' extra metadata files
(top_level.txt, namespace_packages.txt, etc) into the .dist-info dir and
remove .egg-info?
> Also, since packaging is an updated version of distutils, we could add
> support for PEP 376 in packaging.
Did you mean "add support for PEP 376 in distutils"? I don't think this
is an option, AFAIK distutils is frozen.
> All we need to do next is update
> distribute to be PEP 376 compliant, then we are all set.
Not really, users might have older versions of distribute installed that
don't have this support.
I think we're better off not modifying distutils or distribute at all,
and implementing whatever compatibility layer is needed in d2. Making
big changes in distutils or distribute really just adds to the
legacy-support headaches in d2, it doesn't help.
Carl
Kelsey Hightower
On Apr 13, 12:48 pm, Carl Meyer <c...@oddbird.net> wrote:
> If we do this for distutils1 projects, it really seems like we ought to
> do it for distribute-based projects as well, but I'm not sure how that
> ought to interact with the legacy .egg-info dir that distribute will
> create with --single-version-externally-managed. Do we leave the
> .egg-info in place and also create a .dist-info with PEP-376-style
> metadata in it? Or do we copy setuptools' extra metadata files
> (top_level.txt, namespace_packages.txt, etc) into the .dist-info dir and
> remove .egg-info?
I like the idea of removing .egg-info and copying the extra files into
.dist-info, but do other tools/libs rely on .egg-info being there?
If not, then this sounds like a good idea.
> Not really, users might have older versions of distribute installed that
> don't have this support.
>
> I think we're better off not modifying distutils or distribute at all,
> and implementing whatever compatibility layer is needed in d2. Making
> big changes in distutils or distribute really just adds to the
> legacy-support headaches in d2, it doesn't help.
We can make things complaint during the installation process.
Plus this has the added benefit of being able to use the same
uninstall process to remove all packages no matter how their installed.
Carl Meyer
> I like the idea of removing .egg-info and copying the extra files into
> .dist-info, but do other tools/libs rely on .egg-info being there?
>
> If not, then this sounds like a good idea.
Well, any tool that isn't updated to support PEP 376 won't know about
anything pysetup installs, but that's not a problem we need to be
concerned about, IMO -- it'll be the case anyway for any new project
that uses setup.cfg.
Carl
Kelsey Hightower
On Apr 13, 2:54 pm, Carl Meyer <c...@oddbird.net> wrote:
> Well, any tool that isn't updated to support PEP 376 won't know about
> anything pysetup installs, but that's not a problem we need to be
> concerned about, IMO -- it'll be the case anyway for any new project
> that uses setup.cfg.
list
that Tarek can review. This will be a big part of the upcoming sprint.
Thanks for your input by the way.
Carl Meyer
On 04/13/2011 03:55 PM, Kelsey Hightower wrote:
> Thanks for your input by the way.
Of course! Thanks more for all your work on pysetup.
Carl
Kelsey Hightower
=========================================================
Projects based on distutils/setuptools/distribute will be installed by
"shelling" out and using the sys.executable.
python setup.py --single-version-externally-managed --record /path/
to/record
During the installation pysetup will print a deprecation warning.
PEP 376
=======
The resulting install will be "fixed up" to support PEP 376.
* The record file will be updated with absolute paths, md5sums, and
file
sizes
* egg-info directories will be replaced by dist-info directories
* PKG-INFO, SOURCES.txt, dependency_links.txt, installed-files.txt,
and top_level.txt files will be removed in favor of METADATA, RECORD,
INSTALLER, and REQUESTED files
Uninstalling distutils/setuptools/distribute based projects
===========================================================
All distributions will be removed the same way, using the new
uninstall mechanisms providing by packaging.
Carl Meyer
On 04/16/2011 12:45 PM, Kelsey Hightower wrote:
> Installing distutils/setuptools/distribute based projects
> =========================================================
>
> Projects based on distutils/setuptools/distribute will be installed by
> "shelling" out and using the sys.executable.
>
> python setup.py --single-version-externally-managed --record /path/
> to/record
>
> During the installation pysetup will print a deprecation warning.
Looks good, with the proviso that --single-version-externally-managed is
only used for distribute packages, not distutils.
> PEP 376
> =======
>
> The resulting install will be "fixed up" to support PEP 376.
>
> * The record file will be updated with absolute paths, md5sums, and
> file
> sizes
> * egg-info directories will be replaced by dist-info directories
> * PKG-INFO, SOURCES.txt, dependency_links.txt, installed-files.txt,
> and top_level.txt files will be removed in favor of METADATA, RECORD,
> INSTALLER, and REQUESTED files
I don't think it's advisable to just remove setuptools' extra metadata
files (dependency_links.txt, top_level.txt, namespace_packages.txt). I
believe PEP 376 permits arbitrary additional metadata files in the
dist-info directory; I think setuptools' non-standard metadata files
should be copied over into dist-info as-is.
installed-files.txt is not relevant - that's pip-specific and will never
be present when pysetup does the installation.
>
>
> Uninstalling distutils/setuptools/distribute based projects
> ===========================================================
>
> All distributions will be removed the same way, using the new
> uninstall mechanisms providing by packaging.
Looks good to me!
Carl
Alexis Métaireau
> So is everyone +1 on setuptools?
Let's go for it !
--
Alexis — http://notmyidea.org
Alexis Métaireau
> Looks good, with the proviso that --single-version-externally-managed is
> only used for distribute packages, not distutils.
We then need a realistic way to detect setuptools dependency. The way
we've done it so far is not good enough (we just checked if the setup.py
contained "setuptools").
We can, after trying all other possibilities (normal setup.cfg,
setup.py) try to launch the setup.py, catch any ImportError and check if
it is setuptools.
> I don't think it's advisable to just remove setuptools' extra metadata
> files (dependency_links.txt, top_level.txt, namespace_packages.txt). I
> believe PEP 376 permits arbitrary additional metadata files in the
> dist-info directory; I think setuptools' non-standard metadata files
> should be copied over into dist-info as-is.
+1. It can provide some useful information in case the PEP 376 standard
metadata files contain errors, so we could (manually) fix errors if any.
We can probably keep those files in the .dist-info directory as well,
probably under a "setuptools" folder?
Alexis � http://notmyidea.org
Kelsey Hightower
In my last couple of commits I have added initial support for
converting egginfo to distinfo.
How does it work?
* Parse the RECORD file produced by python setup.py --record
* create a new .dist-info directory next to the existing .egg-info dir
(setuptools) or egg-info file (distutils)
* copy over any extra metadata files from .egg-info into .dist-info
* copy PKG-INFO (setuptools) or egg-info (distutils) to .dist-info/
METADATA
* generate the required metadata files per PEP 376
* remove existing egg-info if requested.
Please Review:
https://bitbucket.org/tarek/cpython/changeset/e2909cbf3247
https://bitbucket.org/tarek/cpython/changeset/21497603de73
https://bitbucket.org/tarek/cpython/changeset/aae5c4704377
Alexis Métaireau
> Team,
>
> In my last couple of commits I have added initial support for
> converting egginfo to distinfo.
Great !
I started something similar before the port/merge with python 3.3
(https://bitbucket.org/ametaireau/distutils2/changeset/37f92dc5acd4)
I remember having done some refactoring in commands/install_distinfo.py
in order to not duplicate the code in charge of writing the RECORD file.
Kelsey, feel free to have a look to what I did last time and see if it
could be of any use. Anyway, we need to communicate more, what you are
doing by keeping updated on the ML is a great idea, I should do so as
well in order to avoid such duplication of work :-)
> How does it work?
>
> * Parse the RECORD file produced by python setup.py --record
I'm not sure this is still an issue on python 3, but I remember having
to distinguish between windows/mac/linux EOL symbols ("\n, \r etc."),
and I cannot see it in you code. Is it not needed on python 3?
> * create a new .dist-info directory next to the existing .egg-info dir
> (setuptools) or egg-info file (distutils)
> * copy over any extra metadata files from .egg-info into .dist-info
> * copy PKG-INFO (setuptools) or egg-info (distutils) to .dist-info/
> METADATA
> * generate the required metadata files per PEP 376
> * remove existing egg-info if requested.
nitpicking: here we specify the installer name directly in the code,
maybe should
we provide a constant and use it instead? (L 1247 of utils.py)
Also, having some documentation about all that could really be useful.
We probably can add that for the next sprint (if nobody else want to do
it, I can).
--
Alexis
Éric Araujo
> having
> to distinguish between windows/mac/linux EOL symbols ("\n, \r etc."),
> and I cannot see it in you code. Is it not needed on python 3?
Python 3’s open function always uses universal newline mode, whereas
you
have to open with 'rU' mode to get it in 2.x. See the docs.
Cheers
Kelsey Hightower
> could be of any use. Anyway, we need to communicate more, what you are
> doing by keeping updated on the ML is a great idea, I should do so as
> well in order to avoid such duplication of work :-)
something a little more light weight for 3rd party tools to make use
of, including pysetup. I will review your work to prevent further
duplication :)
> I'm not sure this is still an issue on python 3, but I remember having
> to distinguish between windows/mac/linux EOL symbols ("\n, \r etc."),
> and I cannot see it in you code. Is it not needed on python 3?
testing 3rd party packages on Python 3.3.
> nitpicking: here we specify the installer name directly in the code,
> maybe should
> we provide a constant and use it instead? (L 1247 of utils.py)
> Also, having some documentation about all that could really be useful.
> We probably can add that for the next sprint (if nobody else want to do
> it, I can).
before I finalize them.