Bug#1025485: pybuild-plugin-pyproject: pyproject plugin installs data_files in wrong place

[Marcin Owsiany]

Package: pybuild-plugin-pyproject
Version: 5.20221122
Severity: important

TL;DR: plugin_pyproject.py installs files listed under
[options.data_files] (such as manpages) in an unexpected place,
different than pip would have.

Dear Maintainer,

First, a disclaimer: I have almost zero experience and knowledge about
Python project installation tools, and setup.cfg in particular.
So please let me know if I'm talking nonsense here.

A bit of background:

Upstream of ledgerhelpers has recently switched from setup.py to
pyproject.toml + setup.cfg. The latter file contains a bunch of data
files (desktop files, documentation, and manpages) which are supposed to
be installed in FHS-compliant places under /usr/share:
https://github.com/Rudd-O/ledgerhelpers/blob/48ba347b3a81b8759850e5b45ffff4bb91c03bce/setup.cfg#L38-L58

This is indeed more or less when these files land when the application
is installed using pip or the RPM package prepared by the upstream
maintainer. However when I tried to adapt my debian package to the
current upstream version, the pyproject plugin for pybuild installs them
together with the python modules. So for example the manpages end up in
somewhat weird location like /usr/lib/python3.10/dist-packages/share/man/man1

I had a look at the code, and I think this FIXME is exactly about this
issue I'm having:

https://salsa.debian.org/python-team/tools/dh-python/-/blob/master/dhpython/build/plugin_pyproject.py#L131-132

If I correctly understand what is going on in build_step2() and
install() then I think data_files should be treated similarly to
scripts: unpacked into a separate directory and then copied into something like:
args['destdir'] + paths['data']

While the data_files concept seems to be deprecated according to
https://setuptools.pypa.io/en/latest/userguide/declarative_config.html#opt-4I
and does not have a replacement judging by
https://github.com/pypa/packaging-problems/issues/72
I think it would still make sense to try and imitate what pip is doing at
the moment.

-- System Information:
Debian Release: bookworm/sid
APT prefers testing-debug
APT policy: (500, 'testing-debug'), (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.19.0-2-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=pl_PL.UTF-8, LC_CTYPE=pl_PL.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages pybuild-plugin-pyproject depends on:
ii dh-python 5.20221122
ii python3-build 0.7.0-4
ii python3-installer 0.5.1+dfsg1-2
ii python3-tomli 2.0.1-1

pybuild-plugin-pyproject recommends no packages.

pybuild-plugin-pyproject suggests no packages.

-- no debconf information

[Scott Kitterman]

I ran into a similar issue packaging the new spf-engine, which now uses flit and it's new external-data section [1]. I ended up using .install and .manpages files to get things in the right place.

The challenge is that none of this is standardized, so there's no way to come up with a general solution (pip is even somewhat inconsistent depending on how you do the installation). At least arguably worse is that it appears that the upstream Python community largely regards this lack of standards as a feature.

Scott K

[1] https://flit.pypa.io/en/latest/pyproject_toml.html?highlight=data#external-data-section

[Stefano Rivera]

Hi Marcin (2022.12.05_12:40:17_-0400)

> TL;DR: plugin_pyproject.py installs files listed under
> [options.data_files] (such as manpages) in an unexpected place,
> different than pip would have.

Yeah, that's because we never got around to making a test case to see if
we were doing the right thing or not.

I think I've fixed that in https://salsa.debian.org/python-team/tools/dh-python/-/commit/afbc167a10c024ce4890a9d520e6a3ba513494f1

I'll need to do some verification that that doesn't break other
packages. And ideally add some test cases for other build systems.

Note that setuptools deprecated [options.data_files] because it doesn't
work correctly with wheels. But we can do our best to make it work as
much as possible.

SR

--
Stefano Rivera
http://tumbleweed.org.za/
+1 415 683 3272

[rhatto]

Thanks for reporting.

I haven't had time to check this, but I wonder if there's a suggested
way to fix.