Bug#984824: dh-python: pybuild needs to support toml (PEP517/PEP518) builds with no setup.py

[Drew Parsons]

Package: dh-python
Version: 4.20201102+nmu1
Severity: important

Increasingly upstream packages have been moving to the PEP518 build
format, for which a pyproject.toml file is provided, and no setup.py

pybuild fails to handle such packages. In some cases a trivial
setup.py can be patched in, or an old one from before upstream updated
to PEP518. But this does not work for all packages (fails badly with
opendrop, for instance, where scons is invoked).

pybuild needs to start supporting the PEP518 build method.

Too late for bullseye, but I'd like to consider this a serious (FTBFS)
bug post-bullseye.

Drew


-- System Information:
Debian Release: bullseye/sid
APT prefers unstable
APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-4-amd64 (SMP w/8 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE
Locale: LANG=en_AU.UTF-8, LC_CTYPE=en_AU.UTF-8 (charmap=UTF-8), LANGUAGE=en_AU:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages dh-python depends on:
ii python3 3.9.2-2
ii python3-distutils 3.9.2-1

dh-python recommends no packages.

Versions of packages dh-python suggests:
ii dpkg-dev 1.20.7.1
ii libdpkg-perl 1.20.7.1

-- no debconf information

[Trent W. Buck]

Drew,

FYI dh-python=4.20201102+nmu1 can do this (pyproject.toml without setup.py), but
you must opt-in to "flit" specifically.

Documentation is here:

https://manpages.debian.org/unstable/dh-python/pybuild.1.en.html#flit_plugin

A minimal complete example is on #987296:

https://bugs.debian.org/cgi-bin/bugreport.cgi?att=2;bug=987296;filename=xx_0~1.tar.xz;msg=5

A real-world package in Debian is jeepney=0.6.0-1.
It has a ./setup.py, but if you rm it, the package still builds OK.

https://salsa.debian.org/python-team/packages/jeepney.git

NOTE: (I think) this only works if upstream picked flit, i.e.
pyproject.toml has [tool.flit.metadata].

Some random examples NOT using flit:
https://github.com/python/mypy/blob/master/pyproject.toml
https://github.com/numpy/numpy/blob/main/pyproject.toml
https://github.com/pandas-dev/pandas/blob/master/pyproject.toml

In your case, https://github.com/jdber1/opendrop/blob/master/pyproject.toml
I guess pybuild needs a plugin /usr/share/dh-python/dhpython/build/plugin_scons.py
similar to the existing plugin_flit.py.

It looks like "./pyproject.toml exists" is far from sufficient to know
how to actually build something :-(

[Osamu Aoki]

Hi,

Here is my thought ....

PEP-518 package may be without setup.py and may still be based on setuptools. For
that kind of package, work around using flit plugin doesn't work.

Packaging Python Projects tutorial has such example without setup.py by presenting
only setup.conf case as the first choice case.
https://packaging.python.org/tutorials/packaging-projects/

It even states:
> Dynamic metadata (setup.py) should be used only as an escape hatch when absolutely
> necessary. setup.py used to be required, but can be omitted with newer versions of
> setuptools and pip.

Of course, if I add the following escape hatch setup.py to the example using
setup.cnf only, such test package can build Debian package OK using the old setup.py
mechanism.

> #!/usr/bin/python3
> import setuptools
> setuptools.setup()

I think what we need future oriented solution, here. We should create another plugin
for "build" which uses python3-build package as the PEP518 build backend.

https://pypi.org/project/build/
https://github.com/pypa/build
https://packages.debian.org/unstable/python3-build

For Debian package building, we need to use --no-isolation to make sure build without
network access and with proper version.

> python3 -m build --no-isolation

Then install from generated wheel package file or from build directory, I guess.

This is more future oriented generic solution than the manual workaround using a
plugin or patching upstream source for each build tools.

If this "build" plugin matures, it should be selected automatically for package with
projects.toml.

Regards,

Osamu

[Osamu Aoki]

Oops, typos:

s/what we need future/what we need is future/
s/projects.toml./pyproject.toml./

[Drew Parsons]

On 18 Jun 2021, Osamu Aoki wrote:
> PEP-518 package may be without setup.py and may still be based on
> setuptools. For
> that kind of package, work around using flit plugin doesn't work.

> ...
> I think what we need is future oriented solution, here. We should

> create another plugin
> for "build" which uses python3-build package as the PEP518 build
> backend.

Osamu, is sounds like you've got a good idea of what needs to be done to
get pybuild and dh-python working with PEP518 (pyproject.toml).

Are you able to develop the dh-python patch (the PEP518 plugin) we need
to make it work?


Drew

[Stuart Prescott]

I won't go as far as to tag this bug with "patch"... but a draft of a pybuild
plugin that can work with the PEP517 interfaces is available for discussion
and improvement at:

https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/20

--
Stuart Prescott http://www.nanonanonano.net/ stu...@nanonanonano.net
Debian Developer http://www.debian.org/ stu...@debian.org
GPG fingerprint 90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7