Bug#1019293: python3-pip: installs setup_requires into /tmp/.../overlay/local/bin but then adds /tmp/.../overlay/bin to PATH

[Simon McVittie]

Package: python3-pip
Version: 22.2+dfsg-1
Severity: important
X-Debbugs-Cc: pyt...@packages.debian.org

This might be the same issue as https://bugs.debian.org/1008783, but I'm
reporting it here in case it isn't. If the bug is reassigned to python3
or python3.10, please set it as affecting python3-pip.

To reproduce:

$ podman run -it --rm debian:sid-slim
# apt update
# apt install python3-pip
# pip install -vv --user dbus-python==1.3.2

Expected result:

pip downloads some setup_requires (meson, ninja, patchelf, etc.),
installs them into /tmp/pip-build-env-XXXXXXXX/overlay, and uses them to
try to build dbus-python. In this small reproducer the build will fail,
because dbus-python needs libdbus-1-dev and libglib2.0-dev installed on
the host system, but the thing I'm interested in here is that it should
get as far as successfully running meson from pip's overlay. In this
case I'm intentionally using Meson as packaged on PyPI, not Meson as
packaged by Debian.

Actual result:

pip installs the dependencies into /tmp/pip-build-env-XXXXXXXX/overlay,
with the Python code in
/tmp/pip-build-env-XXXXXXXX/overlay/local/lib/python3.10/dist-packages
and the executables in /tmp/pip-build-env-XXXXXXXX/overlay/local/bin/:

> Installing collected packages: patchelf, ninja, wheel, tomli, setuptools, pyparsing, meson, packaging, pyproject-metadata, meson-python
> changing mode of /tmp/pip-build-env-5u2k8uno/overlay/local/bin/ninja to 755
> changing mode of /tmp/pip-build-env-5u2k8uno/overlay/local/bin/wheel to 755
> changing mode of /tmp/pip-build-env-5u2k8uno/overlay/local/bin/meson to 755
> Successfully installed meson-0.63.2 meson-python-0.8.1 ninja-1.10.2.3 packaging-21.3 patchelf-0.15.0.0 pyparsing-3.0.9 pyproject-metadata-0.6.1 setuptools-65.3.0 tomli-2.0.1 wheel-0.37.1

Then it runs the actual build system, which successfully invokes mesonpy,
aka meson-python, which is one of the setup_requires, and is a PEP 517
hook to build Python packages with Meson. It gets as far as trying to
run this command:

> Running command Getting requirements to build wheel
> + meson setup --native-file=/tmp/pip-install-t6dj_vtk/dbus-python_b0dad2d1d4204e928c3741e1194fdaf0/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 --prefix=/usr /tmp/pip-install-t6dj_vtk/dbus-python_b0dad2d1d4204e928c3741e1194fdaf0 /tmp/pip-install-t6dj_vtk/dbus-python_b0dad2d1d4204e928c3741e1194fdaf0/.mesonpy-v6ryteth/build

But that fails:

> File "/tmp/pip-build-env-5u2k8uno/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 477, in _meson
> return self._proc('meson', *args)
> File "/tmp/pip-build-env-5u2k8uno/overlay/local/lib/python3.10/dist-packages/mesonpy/__init__.py", line 472, in _proc
> subprocess.check_call(list(args))
...
> FileNotFoundError: [Errno 2] No such file or directory: 'meson'

I tried hacking some extra debugging into python3-pip, and it looks like
it's adding /tmp/pip-build-env-XXXXXXXX/overlay/bin/ to the PATH. You'll
notice that isn't the same directory
/tmp/pip-build-env-XXXXXXXX/overlay/local/bin/ where pip actually installed
the executables. The difference is the presence or absence of /local.

I suspect this is a side-effect of
debian/patches/distutils-install-layout.diff in python3.10, which has
special cases for when VIRTUAL_ENV is set, but apparently doesn't detect
pip's temporary pseudo-virtualenv as being equivalent.

Either python3.10 should not be applying the Debian-specific unix_local
installation scheme when installing into non-/usr, non-/usr/local
locations, or python3-pip should have a corresponding Debian patch
to force a suitable installation scheme in its temporary overlay, or
python3-pip should have a Debian to patch compensate for the unix_local
installation scheme and set its PATH accordingly.

Workaround for my reproducer:

# apt install meson ninja-build patchelf
# pip install -vv --user dbus-python==1.3.2

which gets as far as successfully running Meson (Debian's Meson rather
than PyPI's Meson). It errors out with

> ../../subprojects/dbus-gmain/meson.build:107:0: ERROR: Pkg-config binary for machine 1 not found. Giving up.

but that's fine (not a python3-pip bug, having pkg-config available is
out-of-scope for pip).

smcv

-- System Information:
Debian Release: bookworm/sid
APT prefers unstable
APT policy: (500, 'unstable')
merged-usr: no
Architecture: amd64 (x86_64)

Kernel: Linux 5.18.0-4-amd64 (SMP w/2 CPU threads; PREEMPT)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8) (ignored: LC_ALL set to C.UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages python3-pip depends on:
ii ca-certificates 20211016
ii python3 3.10.6-1
ii python3-distutils 3.10.6-1
ii python3-setuptools 59.6.0-1.2
ii python3-wheel 0.37.1-2

Versions of packages python3-pip recommends:
ii build-essential 12.9
ii python3-dev 3.10.6-1

python3-pip suggests no packages.

-- no debconf information

[Stefano Rivera]

Hi Simon (2022.09.07_00:26:25_+0200)

> pip installs the dependencies into /tmp/pip-build-env-XXXXXXXX/overlay,
> with the Python code in
> /tmp/pip-build-env-XXXXXXXX/overlay/local/lib/python3.10/dist-packages
> and the executables in /tmp/pip-build-env-XXXXXXXX/overlay/local/bin/:

My initial triage of this is that pip is calling a sub-pip with --prefix in
src/pip/_vendor/pep517/envbuild.py

That will use the posix_local scheme by default on Debian, which won't do
what pip expects, when it assumes the posix_prefix scheme to be used.
Probably selecting having pip explicitly select posix_prefix when
--prefix is supplied would make sense. I don't know what the pip
upstream will think about having to do that...

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

[Simon McVittie]

Control: tags -1 + patch
Control: forwarded -1 https://github.com/pypa/pip/pull/11623

On Mon, 17 Oct 2022 at 21:41:42 +0200, Stefano Rivera wrote:
> That will use the posix_local scheme by default on Debian, which won't do
> what pip expects, when it assumes the posix_prefix scheme to be used.
> Probably selecting having pip explicitly select posix_prefix when
> --prefix is supplied would make sense. I don't know what the pip
> upstream will think about having to do that...

I think https://github.com/pypa/pip/pull/11623 might be the upstream
solution for this?

smcv

[stef...@debian.org]

Hi Simon (2023.01.20_13:54:00_+0000)

> > That will use the posix_local scheme by default on Debian, which won't do
> > what pip expects, when it assumes the posix_prefix scheme to be used.
> > Probably selecting having pip explicitly select posix_prefix when
> > --prefix is supplied would make sense. I don't know what the pip
> > upstream will think about having to do that...
>
> I think https://github.com/pypa/pip/pull/11623 might be the upstream
> solution for this?

Thanks for that, it looks like a step in the right direction, but I'm
afraid it doesn't solve your bug, yet, for a system-wide pip install.

[stef...@debian.org]

Hi Simon (2023.01.20_13:54:00_+0000)

Actually, with a tweak, it can work.