pip and dependency links

[Donald Stufft]

Setuptools has a “feature” called dependency links which allows one project to provide places to look for another project. I think this is dead wrong. Not only should project A not be able to tell you where to get project B, but in general control of *where* something is installed from should be up to the person doing the installing, not the projects they are installed from.

I’d like to remove this in 1.5 as I believe it a security issue. At the very least we need a flag to disable dependency links as this is causing the ensurepip module to reach out to the internet even with —no-index passed in.

Thoughts?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Holger Krekel]

Do you happen to have statistics how much this is used in actual packages?

It sure is an obscure feature ...

holger

[Donald Stufft]

On Oct 26, 2013, at 8:47 AM, Holger Krekel <holger...@gmail.com> wrote:

Do you happen to have statistics how much this is used in actual packages? 

It sure is an obscure feature ...

I don’t have any stats, I can try to generate them but it will require manually reaching into every file on PyPI.

[Holger Krekel]

On Sat, Oct 26, 2013 at 2:48 PM, Donald Stufft <don...@stufft.io> wrote:

On Oct 26, 2013, at 8:47 AM, Holger Krekel <holger...@gmail.com> wrote:

Do you happen to have statistics how much this is used in actual packages? 

It sure is an obscure feature ...

I don’t have any stats, I can try to generate them but it will require manually reaching into every file on PyPI.

I am not asking to cause you work but rather because i suspect it's hardly used which makes it easier to phase it out or remove it.

Holger
 

[Sebastien Douche]

On Sat, Oct 26, 2013 at 2:47 PM, Holger Krekel <holger...@gmail.com> wrote:
> Do you happen to have statistics how much this is used in actual packages?
> It sure is an obscure feature ...

I believe it was used when all packages was not on PyPI. Now, it's a
bad practice imho (and a security issue).




--
Sebastien Douche <sdo...@gmail.com>
Twitter: @sdouche / G+: +sdouche

[Sebastien Douche]

On Sat, Oct 26, 2013 at 2:55 PM, Sebastien Douche <sdo...@gmail.com> wrote:
> On Sat, Oct 26, 2013 at 2:47 PM, Holger Krekel <holger...@gmail.com> wrote:
>> Do you happen to have statistics how much this is used in actual packages?
>> It sure is an obscure feature ...

From what I remember; Marcus asked the same question;

[Jim Fulton]

On Sat, Oct 26, 2013 at 8:44 AM, Donald Stufft <don...@stufft.io> wrote:
> Setuptools has a “feature” called dependency links which allows one project to provide places to look for another project. I think this is dead wrong. Not only should project A not be able to tell you where to get project B, but in general control of *where* something is installed from should be up to the person doing the installing, not the projects they are installed from.
>
> I’d like to remove this in 1.5 as I believe it a security issue. At the very least we need a flag to disable dependency links as this is causing the ensurepip module to reach out to the internet even with —no-index passed in.
>
> Thoughts?

FWIW, buildout has had an option to disable these from it's beginning.
I made the default to honor dependency links out of conservative
deference to setuptools. If I knew then what I know now, I'd have
made the default to not honor them.

I agree with you that they're a bad idea, at least now.

I wonder of any buildout users would object to changing the default.

Jim

--
Jim Fulton
http://www.linkedin.com/in/jimfulton

[Donald Stufft]

If it helps any I’m working on generating some statistics for how many packages actually use them.

>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton

[Jason R. Coombs]

I remember the Turbogears project used this feature as part of their standard
installation procedure. They would host their dependencies in a particular
location to explicitly manage the versions available for the larger suite (so
that even if a dependency's dependency allowed a version incompatible or
untested with the suite, it wouldn't be included in the install).

I've used the command-line parameter for the same property (--find-links) to
install a package against private, patched, or otherwise one-off builds of
dependency packages in a self-hosted directory.

What is the proposed replacement for those use-cases if dependency-links is
deprecated?

> -----Original Message-----
> From: pypa...@googlegroups.com [mailto:pypa...@googlegroups.com]
> On Behalf Of Donald Stufft
> Sent: Saturday, 26 October, 2013 08:44
> To: pypa-dev; Nick Coghlan
> Subject: pip and dependency links
>
> Setuptools has a "feature" called dependency links which allows one project
> to provide places to look for another project. I think this is dead wrong.
> Not
> only should project A not be able to tell you where to get project B, but in
> general control of *where* something is installed from should be up to the
> person doing the installing, not the projects they are installed from.
>
> I'd like to remove this in 1.5 as I believe it a security issue. At the very
> least
> we need a flag to disable dependency links as this is causing the ensurepip

> module to reach out to the internet even with -no-index passed in.

[Donald Stufft]

On Oct 26, 2013, at 11:09 AM, Jason R. Coombs <jar...@jaraco.com> wrote:

> I remember the Turbogears project used this feature as part of their standard
> installation procedure. They would host their dependencies in a particular
> location to explicitly manage the versions available for the larger suite (so
> that even if a dependency's dependency allowed a version incompatible or
> untested with the suite, it wouldn't be included in the install).
>
> I've used the command-line parameter for the same property (--find-links) to
> install a package against private, patched, or otherwise one-off builds of
> dependency packages in a self-hosted directory.

—find-links wouldn’t be going away because that’s in the control of the person
invoking pip.

>
> What is the proposed replacement for those use-cases if dependency-links is
> deprecated?

For the first one? Nothing in particular. You could pin the exact versions in your
own install_requires of course, or distribute requirements.txt’s if you only care
about pip installation. Or even host your own index server that has those exact
versions and tell people to -i https://your/index.server.

The idea being that Turbogears *shouldn’t* be able to force people to install
from their location unless that person explicitly asks them to.

[Marcus Smith]

I’d like to remove this in 1.5 as I believe it a security issue. At the very least we need a flag to disable dependency links

The responsible thing it seems to me is to *only* deprecate it now.  (i.e. setuptools deprecates the keyword; pip deprecates using it)

But realize setuptools itself uses dependency links:  https://bitbucket.org/pypa/setuptools/src/27df3c725f9696ba730456f3f444cc2fb5271d4b/setup.py?at=default#cl-191

It would need to stop using them obviously before it could deprecate it.

 

as this is causing the ensurepip module to reach out to the internet even with —no-index passed in.

why? due to setuptools having 6 dependency links in it's setup.py?  

 

[Marcus Smith]

for the 6 dependency links, they are about fulfilling extras

3 are straight from pypi, so don't seem needed.

the other 3 could be provided via --find-links (assuming users had instructions)

**********

    dependency_links = [

        'https://pypi.python.org/packages/source/c/certifi/certifi-0.0.8.tar.gz#md5=dc5f5e7f0b5fc08d27654b17daa6ecec',

        'https://pypi.python.org/packages/source/s/ssl/ssl-1.16.tar.gz#md5=fb12d335d56f3c8c7c1fefc1c06c4bfb',

        'https://pypi.python.org/packages/source/w/wincertstore/wincertstore-0.1.zip#md5=2f9accbebe8f7b4c06ac7aa83879b81c',

        'https://bitbucket.org/pypa/setuptools/downloads/ctypes-1.0.2.win32-py2.4.exe#md5=9092a0ad5a3d79fa2d980f1ddc5e9dbc',

        'https://bitbucket.org/pypa/setuptools/downloads/ssl-1.16-py2.4-win32.egg#md5=3cfa2c526dc66e318e8520b6f1aadce5',

        'https://bitbucket.org/pypa/setuptools/downloads/ssl-1.16-py2.5-win32.egg#md5=85ad1cda806d639743121c0bbcb5f39b',

    ],

------------------------

from requires.txt

-------------------------

[ssl:sys_platform=='win32' and python_version=='2.4']

ctypes==1.0.2

[ssl:python_version in '2.4, 2.5']

ssl==1.16

[certs]

certifi==0.0.8

[ssl:sys_platform=='win32']

wincertstore==0.1

[Marcus Smith]

the commit that added these links (for the easy_install ssl support I think)

https://bitbucket.org/pypa/setuptools/commits/f40b810acc5f6494735c912a625d647dc2a3c582

[Donald Stufft]

Yes.

[Donald Stufft]

Here are the numbers:

Counter({'total': 167796, 'no dependency links': 163791, 'has dependency links': 4005})

The script uses to get those numbers: https://gist.github.com/dstufft/7173539

This does not state wether those dependency links are required for those projects, or how

old those projects that use them are. All this says is that for the source distributions turning

them off would in some way affect 2% of the total number of source packages uploaded to

PyPI.

At the very least deprecating this seems sane, and at the most I don’t think outright removing

it is going to be noticed.

[Marcus Smith]

At the very least deprecating this seems sane, and at the most I don’t think outright removing

it is going to be noticed.

I think deprecate first, after a distutils-sig discussion (or maybe this has already happened),  and do it in conjunction with setuptools.

but, as for the practical issue right now for ensurepip, I think a PR to setuptools to drop setuptools' own use of dependency links makes the most sense.

[Donald Stufft]

setuptools uses them to get those things securely installed before it has SSL available (the hashes).

[Jason R. Coombs]

I'm strongly considering dropping support for Python 2.5, which would obviate the need for those links but would also lock in the need for those links with Python 2.4+ on setuptools 1.x.

Sent from my comm

[Donald Stufft]

pip doesn’t support Python 2.5 so those links are useless from the pip side of things FWIW.

[Choongmin Lee]

I'm using dependency_links to pull a package from GitHub since the version on PyPI breaks with Python 3 while the one on GitHub is more recent and works. What's my alternative? I know the best option would be uploading the new version to PyPI, but that probably won't happen, or at least not soon, considering that the last time the package author made a change to the project was 6 months ago.

[Marcus Smith]

On Thu, Feb 27, 2014 at 3:18 PM, Charles Dimino <dim...@gmail.com> wrote:

Correction, the -i command offers a way to use an alternate index.

the point is for the user to be fully in control of where requirements are fulfilled from, and not have it "quietly" altered due to dependency links defined in a project's metadata.

user's have control with --index-url, --find-links, and can also use vcs urls in requirements files.

[Charles Dimino]

I found the alternatives, and had deleted my comment, but since you "resurrected" it (no worries!), I feel I should mention an article I came across which helped me get my head around where I eventually ended up: both a requirements.txt file and a setup.py file: https://caremad.io/blog/setup-vs-requirement/

[Marcus Smith]

yes, that's donald's article.  also, here's a page explaining the transition, that will be in the 1.6 docs when it's released.

http://www.pip-installer.org/en/develop/dependency_links.html

[Donald Stufft]

Sounds like you got it al worked out, glad to hear that!

Just to iterate, this is nothing to do with preventing people from installing from other locations and everything to do with moving the control of “where” you install from to the end user and not the package author.

[stm...@gmail.com]

Hello,

We are currently using dependency_links in my company for the following use case:

We have developed a number of libraries that for one reason or another aren't released as open source. The way we install them is by hosting them on bitbucket and having "git+ssh://g...@bitbucket.org/..." dependencies on each project's requirements.txt. This works fine for simple cases, but some of our private libraries depend on other private libraries.

This is what we use dependency_links for. If library_one depends on library_two, it will have in its setup.py the following:

setup(
    ....
    install_requires=[
        ...,
        'library_two==1.2.3',
    ],
    dependency_links=[
        'git+ssh://g...@bitbucket.org/mycompany/library...@1.2.3#egg=library_two-1.2.3',
    ]
)

I don't know if this is a common use case, but if you search for "dependency links" in google, the first couple of hits are people struggling with the above. It's also a use case that wouldn't be found by scanning public packages on PyPI.

I don't mind having to pass "--process-dependency-links" to pip and I'm not particularly attached to the current implementation, but I think it is a valid use case that should be somehow supported by pip (even if dependency_links is deprecated) without having to maintain a private pypi index.

Regards,
Stratos.

Τη Σάββατο, 26 Οκτωβρίου 2013 3:44:24 μ.μ. UTC+3, ο χρήστης Donald Stufft έγραψε:

Setuptools has a “feature” called dependency links which allows one project to provide places to look for another project. I think this is dead wrong. Not only should project A not be able to tell you where to get project B, but in general control of *where* something is installed from should be up to the person doing the installing, not the projects they are installed from.

I’d like to remove this in 1.5 as I believe it a security issue. At the very least we need a flag to disable dependency links as this is causing the ensurepip module to reach out to the internet even with —no-index passed in.

Thoughts?

[Gilles Dartiguelongue]

Le dimanche 20 avril 2014 15:37:15 UTC+2, stm...@gmail.com a écrit :

Hello,

We are currently using dependency_links in my company for the following use case:

We do the same at my company. The use case is mostly the same. Development branch of the code may depend on other private libraries which are not yet released. This is used by our developers to setup a virtualenv quickly with every dependency up-to-date and by our jenkins jobs to run unittests.

Then there is the master branch that only depends on released packages.

We do have a private pypi mirror and private packages index but we cannot use it to push unfinished releases or various branch builds every hour just to have clean venvs quickly up and running.
The current approach of removing the dependency_links support is highly annoying to us, that resulted in https://github.com/pypa/pip/issues/1428,  and seems to be a regression compared to newly popular languages and softwares such as go, node.js, docker, etc.

I agree that the current dependency_links code might cover too much use cases, deploying a package index isn't much work, but pulling code from repositories seems worth it.

[Tai Lee]

Same here, too. We have many internal libraries that are hosted on GitHub, BitBucket and Assembla. These will never be released and are developed by different people.

Ideally we need to create new projects which might use one or more of these libraries, which have their own dependencies on other internal libraries.

`dependency_links` makes it easy for us to do this, and I haven't heard a good alternative being suggested, yet (for our needs). Running an internal package index seems like it will add a whole new layer of unecessary complexity to our process.

Your stats mentioned that 4,000 packages are using dependency links? Maybe a small percentage over all, but that's still 4,000 packages.

I understand the idea that `setup.py` should not define *where* to get packages and force those locations to be used. However I still think it is a legitimate and useful feature for managing dependencies between private libraries.

What about making the `dependency_links` only used as a fallback, if the required packages are not found publicly or at any of the locations provided on the command line by the person doing deployment?

Or prompting "are you SURE you want to install foo from git+ssh://..." for each requirement being satisfied by `dependency_links`, giving people a chance to be really sure they are getting the package they want?

Otherwise, it will be a nightmare trying to manage all of these inter-dependencies with `requirements.txt` alone, or additional maintenance burden in setting up and maintaining a private package index, or even having to provide extra deployment instructions telling people where to find the private packages and have them tell pip explicitly.

Cheers.

Tai.

[Charl Botha]

In our scenario, we have a top-level package that depends on a number of internal packages via its requirements.txt. However, each of those packages also depends on internal repos.

It used to work really well: You would pip install the top-level package, which would pull in the internal packages, each of which would then pull in their dependencies from internal repos via their setup.py files.

This means that we now have to maintain an internal package index, which is unfortunately decoupled from the separate package configurations.

I understand the motivations behind this change. However, retaining the opt-in --process-dependency-links would have been a good compromise between those motivations and the real-world applications of many users. Are there any other work-arounds (besides the separately maintained package index above) that are also closely coupled to and localized in the package configuration? (package-local requirements.txt is not processed when that package is itself pulled in as a dependency, unless I am mistaken.)

[Donald Stufft]

After further thought, I’ve just submitted a PR that will revert the removal
of ``--process-dependency-links``. While I believe that dependency links on
a *public* index are bad, for private packages they are fine. In the future
Metadata 2.0 will provide a better means for this, but in the mean time keeping
this opt-in at least mostly removed them from default usage on a public index.

https://github.com/pypa/pip/pull/1955