pip install mpi4py no longer installs latest version

[Nathan Goldbaum]

Hi all,

Due to a recent change in pip/pypi,

pip install mpi4py

will now install an old version, mpi4py 0.6. This is the last version for which there is a source package hosted on pypi. Unfortunately, in recent releases of pypi externally hosted files have been deprecated for security reasons.

We've had two users run into this today on our project's IRC channel.

Would it be possible to upload an mpi4py package directly to pypi?

Cheers,

Nathan

[Nathan Goldbaum]

Sorry for the double-post...

[Lisandro Dalcin]

On 5 March 2014 19:58, Nathan Goldbaum <natha...@gmail.com> wrote:
> Hi all,
>
> Due to a recent change in pip/pypi,
>
> pip install mpi4py
>
> will now install an old version, mpi4py 0.6. This is the last version for which there is a source package hosted on pypi. Unfortunately, in recent releases of pypi externally hosted files have been deprecated for security reasons.
>

Security reason? I don't want to start a rant, but it really bothers me that

$ pip install mpi4py

does not work, but

$ pip install http://bitbucket.org/mpi4py/mpi4py/downloads/mpi4py-1.3.1.tar.gz

success.

These new features make it much harder to use PyPI to install from
development repositories, and I really do not see the reason of being
so picky about externally hosted files. Protecting the users from bad
downloads through checking the MD5 hash is OK, but disabling by
default externally hosted files that can be accesses via https (like
bitbucket or github downloads), well, that seems a little too much.

Anyway, the external hosting issue relatively easy to fix with a pip
config file, see below.

> We've had two users run into this today on our project's IRC channel.
>
> Would it be possible to upload an mpi4py package directly to pypi?
>

I would really prefer to continue hosting my downloads exclusively in
bitbucket.

I've just added a download URL to PyPI that contains the md5 hash of
the file hosted in bitbucket. Now, with a little configuration, pip
seems happy again. Do you think this would be enough?

$ echo "[install]" >> ~/.pip/pip.conf
$ echo "allow_all_external=1" >> ~/.pip/pip.conf
$ cat ~/.pip/pip.conf
[install]
allow_all_external=1

$ pip install mpi4py --user
Downloading/unpacking mpi4py
mpi4py an externally hosted file and may be unreliable
Downloading mpi4py-1.3.1.tar.gz (1.0MB): 1.0MB downloaded
Running setup.py (path:/tmp/pip_build_dalcinl/mpi4py/setup.py)
egg_info for package mpi4py
Installing collected packages: mpi4py
....

Do you think this could make everyone happy? I guess mpi4py is not the
only project that will suffer this issue.

Please note that this is not too much different that adding a new
software repository to yum or apt-get. Well, it is. Now that pip is so
picky about external hosts, they should provide a way to designate
some external hosts as "trusted", eg:

[install]
trusted_hosts = bitbucket.org github.com

Feel free to discuss the issue and make this suggestion to pip
developers if you use freenode for IRC.

PS: If you feel I'm wrong and I'm putting too much burden on
end-user's shoulders, then reply back, I might surrender and upload a
tarball to PyPI.


--
Lisandro Dalcin
---------------
CIMEC (UNL/CONICET)
Predio CONICET-Santa Fe
Colectora RN 168 Km 472, Paraje El Pozo
3000 Santa Fe, Argentina
Tel: +54-342-4511594 (ext 1016)
Tel/Fax: +54-342-4511169

[Yury V. Zaytsev]

On Thu, 2014-03-06 at 11:59 +0300, Lisandro Dalcin wrote:
>
> PS: If you feel I'm wrong and I'm putting too much burden on
> end-user's shoulders, then reply back, I might surrender and upload a
> tarball to PyPI.

I've also bumped into this a couple of times recently while rebuilding a
suitable Python environments on my clusters, and I have to say that it's
quite annoying.

I didn't have time to figure out exactly what's going on, so I just used
your earlier suggestion of supplying the URL directly, but I don't see
how manually configuring pip to add an exception is any different: the
point is, it doesn't work out of the box.

I'm not sure what disabling external downloads if the host is accessible
through https and has been explicitly designated as an authoritative
source by the package author adds in terms of security. PyPI still can
be hacked just as any external hosts, and package authors are still free
to upload whatever they like to PyPI, including trojaned code.

I'd really prefer relaxing this restriction on the PyPI side, but
barring this option, maybe it's actually not too much effort to host
releases on PyPI as well?

--
Sincerely yours,
Yury V. Zaytsev

[Lisandro Dalcin]

On 6 March 2014 12:32, Yury V. Zaytsev <yu...@shurup.com> wrote:
> I'm not sure what disabling external downloads if the host is accessible
> through https and has been explicitly designated as an authoritative
> source by the package author adds in terms of security. PyPI still can
> be hacked just as any external hosts, and package authors are still free
> to upload whatever they like to PyPI, including trojaned code.
>

I totally agree with you.

> I'd really prefer relaxing this restriction on the PyPI side, but
> barring this option, maybe it's actually not too much effort to host
> releases on PyPI as well?

Done, please test.

[Nathan Goldbaum]

Thanks for uploading the latest version. No need to upload 1.3.0, that was a typo on my part.

I agree that making a backward incompatible change in a package manager like this is extremely annoying for package maintainers. Let's just say that I'm not terribly surprised that python packaging ecosystem is still sprouting new warts...

[Yury V. Zaytsev]

On Thu, 2014-03-06 at 13:36 +0300, Lisandro Dalcin wrote:
>
> > I'd really prefer relaxing this restriction on the PyPI side, but
> > barring this option, maybe it's actually not too much effort to host
> > releases on PyPI as well?
>
> Done, please test.

FYI, I've been re-building the whole Python stack on yet another cluster
today and it works beautifully now, many thanks!