Bug#1035546: python3-pip: pip install ignores explicit --prefix=/usr (even with --root)

[Martin Pitt]

Package: python3-pip
Version: 23.0.1+dfsg-1

My upstream project (cockpit) is soon going to deliver a new Python component,
where it installs a locally built wheel (i.e. some Python packages and
auto-generated executable wrappers) with

python3 -m pip install --no-index --force-reinstall --root='$(DESTDIR)/' --prefix='$(prefix)' tmp/pybuild/dist/*.whl

This works fine on Fedora, but not on Debian:

rm -rf /tmp/i
python3 -m pip install --no-index --force-reinstall --root=/tmp/i --prefix=/usr tmp/pybuild/dist/*.whl
find /tmp/i

This shows that everything gets installed into /tmp/i/usr/local , i.e. /bin,
lib/python3.11, etc.

I'm quite sure this is somehow related to the Debian magic of not installing
into /usr [1][2], but this is happening during *building* a Debian package,
with --root.

It seems to me that (1) explicitly specifying --prefix=/usr should still be
respected, and not silently be changed to /usr/local (as a *default*,
/usr/local/ is fine of course). And (2) perhaps this change shouldn't be
applied with --root?

Do you know a workaround for this? I can't simply move debian/tmp/usr/local/*
to debian/tmp/usr/, as the upstream build system already assumes that a wheel
binary lands in DESTDIR/usr/bin/, i.e. `make install` is just broken.

Thanks,

Martin

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=771794
[2] https://salsa.debian.org/python-team/packages/python-pip/-/blob/master/debian/patches/hands-off-system-packages.patch

[Martin Pitt]

Hello all,

Is that patch even necessary still? In Fedora with python3-pip 2.22.3,
`sudo pip install python-dbusmock` installs into
/usr/local/lib/python3.11/site-packages/dbusmock/ , i.e. does not conflict
with /usr any more. They do have a more targetted protection of /usr [1], but
no general path mangling in their other patches [2].

Thanks,

Martin

[1] https://src.fedoraproject.org/rpms/python-pip/blob/rawhide/f/remove-existing-dist-only-if-path-conflicts.patch
[2] https://src.fedoraproject.org/rpms/python-pip/tree/rawhide

[Martin Pitt]

Hello again,

I just locally reverted [1], and it's still the same, so that Debian patch
actually isn't to blame.

I also tried --prefix=/opt which then gets me /opt/local/{bin,lib}. So this
behaviour directly contradicts the --help documentation:

> --prefix <dir> Installation prefix where lib, bin and other
> top-level folders are placed

It's also not about my local wheel --

rm -rf /tmp/i; python3 -m pip install --root=/tmp/i --prefix=/opt small-timer
find /tmp/i

again gives /tmp/i/opt/local/lib/python3.11/dist-packages/small_timer

So this "/local" thing is both sticky and hard to avoid..

Martin

[1] https://salsa.debian.org/python-team/packages/python-pip/-/blob/master/debian/patches/hands-off-system-packages.patch

[Stefano Rivera]

Control: forwarded -1 https://github.com/pypa/pip/issues/10978

Hi Martin (2023.05.05_13:03:43_+0200)

> I also tried --prefix=/opt which then gets me /opt/local/{bin,lib}. So this
> behaviour directly contradicts the --help documentation:

...

> So this "/local" thing is both sticky and hard to avoid..

Yeah, the cause of this is our posix_local sysconfig scheme.

Ideally pip should have a --scheme argument. At least in Debian, where
we have multiple schemes interacting with each other, that could make
sense.

Or pip could use some knowledge of Debian's schemes to select
posix_prefix / posix_local as appropriate...

This all gets messy, I'm afraid.

Stefano

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

[J.J. Green]

A note on the effects of this, I'm managing a largeish
commercial scientific software package designed to run
on Debian and derivatives, it has a bit of Python in
it. We install the package, then the users have full
range on the machine. To fix deprecation warnings
about

python setup.py install

I tried the documented replacement

pip3 install . --prefix=/usr/local

and was baffled to find no files in /usr/local/bin etc,
a "find / -name ..." found them in /usr/local/local as
per discussion above. To work around this, I guess I
could use --prefix=/usr, but that seems particularly
dangerous, all it needs is someone on the machine to run
pip3 --update pip and that above command installs into
/usr. Some kind of autoconf detection of whether the
pip3 we've got is "debian-extra-local"-ised would seem
the only option, gnarly.

This becomes pressing as the deprecation of setup.py
installations progresses of course.

Best wishes

Jim
--
J.J. Green, Seaview Sensing Ltd.,
http://seaviewsensing.com