build dependencies
Carl Meyer
We regularly get bug reports in pip about pip failing with packages
that have an import dependency on some other package in their
setup.py, and/or a build dependency that isn't made explicit using
setuptools and setup_requires. See for instance https://github.com/pypa/pip/issues/25
I remember some discussion here about expressing dependencies other
than runtime ones in distutils2 (build deps, test deps, etc), but I
don't remember the conclusion.
Does distutils2 have a way for packages to express their build-time
dependencies?
Carl
Éric Araujo
> We regularly get bug reports in pip about pip failing with packages
> that have an import dependency on some other package in their
> setup.py, and/or a build dependency that isn't made explicit using
> setuptools and setup_requires. See for instance https://github.com/pypa/pip/issues/25
“Python Packaging Authority”? Is that a putsch? <wink>
> I remember some discussion here about expressing dependencies other
> than runtime ones in distutils2 (build deps, test deps, etc), but I
> don't remember the conclusion.
No agreement was reached. There were some arguments in favor of not
merging test dependencies and build dependencies (as I proposed), and
not much feedback about the extra runtime dependencies (apart from a
general feeling that they were used and useful). One sort of conclusion
was that a PEP 345 update discussion was needed.
> Does distutils2 have a way for packages to express their build-time
> dependencies?
Right now: nothing easy and built-in. A pre-build hook could be
written, but I’d much prefer we agree on adding more kinds of dependencies.
Cheers
Carl Meyer
On 04/29/2011 10:15 AM, Éric Araujo wrote:
>> We regularly get bug reports in pip about pip failing with packages
>> that have an import dependency on some other package in their
>> setup.py, and/or a build dependency that isn't made explicit using
>> setuptools and setup_requires. See for instance https://github.com/pypa/pip/issues/25
> “Python Packaging Authority”? Is that a putsch? <wink>
Uh oh - looks like I blew our cover ;-)
>> I remember some discussion here about expressing dependencies other
>> than runtime ones in distutils2 (build deps, test deps, etc), but I
>> don't remember the conclusion.
> No agreement was reached. There were some arguments in favor of not
> merging test dependencies and build dependencies (as I proposed), and
> not much feedback about the extra runtime dependencies (apart from a
> general feeling that they were used and useful). One sort of conclusion
> was that a PEP 345 update discussion was needed.
It seems that if we agree on the need, it's just a matter of identifying
the particular dependencies that should be identified. Personally I've
never seen a case for anything other than "build" and "test" in addition
to regular runtime dependencies, but I don't think there's a great harm
in adding one or two more if there are convincing needs for them.
Beyond identifying which ones are needed, I don't think there's much
controversy here. The naming should just follow existing patterns, e.g.
"Build-Requires-Dist" and "Test-Requires-Dist". Tarek, what process is
needed for this? Do we need a new PEP, or can we still revise PEP 345?
>> Does distutils2 have a way for packages to express their build-time
>> dependencies?
> Right now: nothing easy and built-in. A pre-build hook could be
> written, but I’d much prefer we agree on adding more kinds of dependencies.
Agreed.
Carl
Éric Araujo
> the particular dependencies that should be identified. Personally I've
> never seen a case for anything other than "build" and "test" in addition
> to regular runtime dependencies, but I don't think there's a great harm
> in adding one or two more if there are convincing needs for them.
syntax checks for the long description”.
> Beyond identifying which ones are needed, I don't think there's much
> controversy here. The naming should just follow existing patterns, e.g.
> "Build-Requires-Dist" and "Test-Requires-Dist".
Are you following distutils-sig and catalog-sig? Alexis raised a big
issue last year: the fields are named *-Dist but the values are releases
(name + version), not distributions (name + version + type). This is
still an outstanding item.
> Tarek, what process is needed for this? Do we need a new PEP, or can we
> still revise PEP 345?
Last time we discussed the new kinds of dependencies, Tarek talked about
revising PEP 345.
Regards
Carl Meyer
>> It seems that if we agree on the need, it's just a matter of identifying
>> the particular dependencies that should be identified. Personally I've
>> never seen a case for anything other than "build" and "test" in addition
>> to regular runtime dependencies, but I don't think there's a great harm
>> in adding one or two more if there are convincing needs for them.
> You know, stuff like “optional dep on docutils for distutils2 to enable
> syntax checks for the long description”.
That sort of ad-hoc use case seems better handled by something like
setuptools' extras than by proliferating explicit requirements types for
all possibilities. I think that should be considered separately from the
question of introducing the core types of dependencies that are commonly
needed.
>> Beyond identifying which ones are needed, I don't think there's much
>> controversy here. The naming should just follow existing patterns, e.g.
>> "Build-Requires-Dist" and "Test-Requires-Dist".
> Are you following distutils-sig and catalog-sig? Alexis raised a big
> issue last year: the fields are named *-Dist but the values are releases
> (name + version), not distributions (name + version + type). This is
> still an outstanding item.
That may be an outstanding question, but it's orthogonal. What I said
remains accurate: the new ones should simply follow the pattern,
whatever it is. If Requires-Dist becomes Requires-Release, then
obviously Build-Requires-Dist should become Build-Requires-Release.
These are separate questions that don't need to be harnessed together.
>> Tarek, what process is needed for this? Do we need a new PEP, or can we
>> still revise PEP 345?
> Last time we discussed the new kinds of dependencies, Tarek talked about
> revising PEP 345.
Great. Would it be helpful if I submitted a patch to PEP 345 that added
Build-Requires-Dist and Test-Requires-Dist?
Carl
Tarek Ziadé
..
> Great. Would it be helpful if I submitted a patch to PEP 345 that added
> Build-Requires-Dist and Test-Requires-Dist?
you should present it here first, so we can iterate, then we can think
of a patch.
Some questions I have for Build-Requires-Dist, which opens a can of
worms I think
how do you determine the target platforms ? are you going to use the
environment markers ?
we're talking about pure Python deps right ? what are some examples of
build reqs ?
is this metadata field can be used by other existing building tools ?
if so, how do they do right now ?
what about non-python build deps ?
Cheers
>
> Carl
>
--
Tarek Ziadé | http://ziade.org
Éric Araujo
> setuptools' extras than by proliferating explicit requirements types for
> all possibilities. I think that should be considered separately from the
> question of introducing the core types of dependencies that are commonly
> needed.
adding one time of option runtime dependencies, like seputools extras or
APT recommends/suggests.
> That may be an outstanding question, but it's orthogonal. What I said
> remains accurate: the new ones should simply follow the pattern,
> whatever it is.
True.
> Great. Would it be helpful if I submitted a patch to PEP 345 that added
> Build-Requires-Dist and Test-Requires-Dist?
I think an email to distutils-sig with proposed new paragraphs would be
better than a patch.
Cheers
Alexis Métaireau
> Are you following distutils-sig and catalog-sig? Alexis raised a big
> issue last year: the fields are named *-Dist but the values are releases
> (name + version), not distributions (name + version + type). This is
> still an outstanding item.
The problem is we didn't agreed on a vocabulary to use (eg. packages vs
projects/releases/dists). I'm not sure what to do with this, I still do
have a version of the PEP with modifications to reflect that here:
https://bitbucket.org/ametaireau/python-peps/src/tip/pep-0345.txt)
What am I supposed to do with that ? If the fields have to be changed,
they have to be before the release of tools that are using this PEP, so
it's probably the best time to work on that.
--
Alexis
Tarek Ziadé
Please let's keep it here for now -- this list has enough active
members and we can open the discussion later on distutils-sig once we
have something solid.
>
> Cheers
Carl Meyer
On 04/29/2011 11:25 AM, Tarek Ziad� wrote:
> On Fri, Apr 29, 2011 at 6:18 PM, Carl Meyer <ca...@oddbird.net> wrote:
> ..
>> Great. Would it be helpful if I submitted a patch to PEP 345 that added
>> Build-Requires-Dist and Test-Requires-Dist?
>
> you should present it here first, so we can iterate, then we can think
> of a patch.
Ok.
> Some questions I have for Build-Requires-Dist, which opens a can of
> worms I think
I'll try to stuff the worms back in the can as best I'm able ;-)
> how do you determine the target platforms ? are you going to use the
> environment markers ?
I don't see why environment markers wouldn't be allowed here, same as
the current Requires-Dist.
> we're talking about pure Python deps right ? what are some examples of
> build reqs ?
This isn't really a use case I have personally; I'm seeing it in pip bug
reports. The only example I have in mind right now is numpy/scipy, the
bug I linked at the start of this thread. So no, not pure-Python only,
but yes, Python distributions only (as in, I would prefer to leave
purely system dependencies out of scope for this).
Having numpy installed is a build requirement for scipy, not only a
runtime requirement.
> is this metadata field can be used by other existing building tools ?
> if so, how do they do right now ?
Well, right now scipy enforces this build dependency in part by having
its setup.py import from numpy (in order to use its fork of distutils).
So you can't get anything at all out of scipy's setup.py, even metadata,
unless you already have numpy available. This is obviously unfortunate,
and I would rather that in future they could specify numpy as a build
dependency, so pip (and pysetup) would know how it has to order its
installs.
> what about non-python build deps ?
If you mean things like OS-level -dev packages, I have no desire to
address that issue at all.
Carl
Tarek Ziadé
so you want to install it locally, or remove it after ?
or do you plan to have it installed before by checking if it's also a
run dependency ?
do we have other uses cases were the package is only required at build time ?
>
>> what about non-python build deps ?
>
> If you mean things like OS-level -dev packages, I have no desire to
> address that issue at all.
if we limit ourselves to the pure python packages and only the
environment marker, we're putting back most worms in the can :)
also, I am wondering if it's a good name because what this is really
is an install dependency we don't know if the dependency is used to
build anything, just that it's a prerequisite for installation.
For example, if a project depends on an extension of packaging to get
installed, it might be also the case for binary distributions.
s/Build-Requires-Dist/Install-Requires-Dist/ ?
Carl Meyer
>> Well, right now scipy enforces this build dependency in part by having
>> its setup.py import from numpy (in order to use its fork of distutils).
>> So you can't get anything at all out of scipy's setup.py, even metadata,
>> unless you already have numpy available. This is obviously unfortunate,
>> and I would rather that in future they could specify numpy as a build
>> dependency, so pip (and pysetup) would know how it has to order its
>> installs.
>
> so you want to install it locally, or remove it after ?
> or do you plan to have it installed before by checking if it's also a
> run dependency ?
Personally, I don't think the use case of a build time dependency that
isn't a runtime dependency is worth the extra hassle of a temporary
install. So I would probably just install everything and leave it,
though I'd be open to arguments against that.
In any case, separating build requirements is useful because those are
the only ones that need to affect installation order. If everything is
lumped into Requires-Dist, then we have to sort packages for
installation to avoid Requires-Dist forward-references, and that really
increases the potential for cycles.
> do we have other uses cases were the package is only required at build time ?
The only one I've run across is setuptools vcs plugins, but that's not
really relevant (unless distutils2 had plugins, which gets into the
territory �ric was bringing up).
>>> what about non-python build deps ?
>>
>> If you mean things like OS-level -dev packages, I have no desire to
>> address that issue at all.
>
> if we limit ourselves to the pure python packages and only the
> environment marker, we're putting back most worms in the can :)
Yay!
> also, I am wondering if it's a good name because what this is really
> is an install dependency we don't know if the dependency is used to
> build anything, just that it's a prerequisite for installation.
>
> For example, if a project depends on an extension of packaging to get
> installed, it might be also the case for binary distributions.
>
> s/Build-Requires-Dist/Install-Requires-Dist/ ?
True - It might confuse people because setuptools' install_requires is
generally used for runtime dependencies, but I'm fine with that name change.
Carl
Erik Bray
> > Does distutils2 have a way for packages to express their build-time
> > dependencies?
>
> Right now: nothing easy and built-in. A pre-build hook could be
> written, but I’d much prefer we agree on adding more kinds of dependencies.
hooks, that does raise a question for me: If a Build-Requires-Dist is
added, would setup_hooks be run before build requirements are
installed or after? I would argue for after: That way a setup_hook
from a build requirement can be used (an idea I'm exploring right
now).
Either that, there could also be setup requirements, which were
discussed as a possibility in the previous thread. The idea of a
setup requirement is that it would be required for any setup command
to run, be in building, testing, etc. This would also be required for
adding custom commands, I think. Though Build-Requires would be fine
for this so long as setup_hooks and custom commands are processed
after the build requirements are installed.
Erik
Éric Araujo
>> also, I am wondering if it's a good name because what this is really
>> is an install dependency we don't know if the dependency is used to
>> build anything, just that it's a prerequisite for installation.
>>
>> For example, if a project depends on an extension of packaging to
>> get
>> installed, it might be also the case for binary distributions.
>>
>> s/Build-Requires-Dist/Install-Requires-Dist/ ?
> True - It might confuse people because setuptools' install_requires
> is
> generally used for runtime dependencies, but I'm fine with that name
> change.
“Build dependencies” is a widely used term to name “things you need in
order to do anything with this source tarball”. If you really think you
need to change it, “install” is no better than “build”; “setup
dependencies“ would work best for distutils, I think.
Regards
Tarek Ziadé
> Hi,
>
>>> also, I am wondering if it's a good name because what this is really
>>> is an install dependency we don't know if the dependency is used to
>>> build anything, just that it's a prerequisite for installation.
>>>
>>> For example, if a project depends on an extension of packaging to get
>>> installed, it might be also the case for binary distributions.
>>>
>>> s/Build-Requires-Dist/Install-Requires-Dist/ ?
>>
>> True - It might confuse people because setuptools' install_requires is
>> generally used for runtime dependencies, but I'm fine with that name
>> change.
We need to use the best name possible and not decide on the name based
on the legacy names used by older tools
> “Build dependencies” is a widely used term to name “things you need in order
> to do anything with this source tarball”.
Yes exactly: source tarball. But here, we've described a
post-installation dependency that could applied to binary
distribution, so it's not the right name.
> If you really think you need to
> change it, “install” is no better than “build”; “setup dependencies“ would
> work best for distutils, I think.
please define "setup" as opposed to "install"
>
> Regards
Tarek Ziadé
> On Sat, Apr 30, 2011 at 6:42 PM, Éric Araujo <win...@netwok.org> wrote:
>> Hi,
>>
>>>> also, I am wondering if it's a good name because what this is really
>>>> is an install dependency we don't know if the dependency is used to
>>>> build anything, just that it's a prerequisite for installation.
>>>>
>>>> For example, if a project depends on an extension of packaging to get
>>>> installed, it might be also the case for binary distributions.
>>>>
>>>> s/Build-Requires-Dist/Install-Requires-Dist/ ?
>>>
>>> True - It might confuse people because setuptools' install_requires is
>>> generally used for runtime dependencies, but I'm fine with that name
>>> change.
>
> We need to use the best name possible and not decide on the name based
> on the legacy names used by older tools
one good example that comes in mind: description vs long_descri[ption
vs summary.
Distutils2 cleans up this naming by having setup.cfg options that are
the same names that the metadata fields
That will be confusing for setuptools/distutils1 users but that's one
thing we've fixed in d2
cheers
Éric Araujo
> based
> on the legacy names used by older tools
We know that we disagree on this :)
> please define "setup" as opposed to "install"
Any action (command) run with pysetup or other tools using setup.cfg.
Regards
Tarek Ziadé
>> We need to use the best name possible and not decide on the name based
>> on the legacy names used by older tools
>
> We know that we disagree on this :)
Not sure why you say this, so I'll just make an assumption : If you
are implying that I am against the -Dist suffix, this is not the same
level.
I am against limiting ourselves choosing a name that might be confused
with an option name of setuptools or distutils1 (like install_requires
or long_description) because those are not even standards, just option
names.
I want us to define the best standard possible, with the previous
version of this standard in mind. And to an extent, choose options
which names are explicit.
>
>> please define "setup" as opposed to "install"
>
> Any action (command) run with pysetup or other tools using setup.cfg.
That sounds better yes,
Éric Araujo
> On Sat, Apr 30, 2011 at 7:04 PM, Éric Araujo <win...@netwok.org> wrote:
>>> We need to use the best name possible and not decide on the name based
>>> on the legacy names used by older tools
>> We know that we disagree on this :)
> Not sure why you say this,
>>> please define "setup" as opposed to "install"
>> Any action (command) run with pysetup or other tools using setup.cfg.
> That sounds better yes,
Cool, let’s add Setup-Requires-Dist and then rename s/-Dist/-Release/ :)
Regards