[gentoo-dev] Python 3.11 is ready to enter PYTHON_COMPAT (and other Python news)

[Michał Górny]

Hi, everyone.

TL;DR:

1. Gentoo eclasses now support Python 3.11 in ~arch. Please start
testing your packages. (It's going to be stabilized once it leaves
beta, i.e. sometime Nov-Dec.)

2. We're switching to Python 3.10 in the beginning of June, so if you
haven't ported your packages yet, please hurry.

3. While updating your packages, please switch to DISTUTILS_USE_PEP517
build.

For help, please consult our docs:
https://projects.gentoo.org/python/guide/


===================================
Python 3.11 now supported in Gentoo
===================================
Python 3.11.0b1 has been released recently. This means upstream feature
freeze (i.e. technically no breaking changes, unless they do some last-
minute reverts like with 3.10.0 final). This in turn we can start
testing packages and marking them as 3.11 happy.

We've already have ported some initial packages (which nowadays means
tons of packages). My experience so far is that Python 3.11 is rather
painless, compared to previous releases. What usually breaks are gross
hacks or deprecations ignored for years (both common in Python ecosystem
but not that common).

Bad news is that for one useful package, you may end up having to deal
with half a dozen unmaintained, broken NIH dependencies. Good news is
that we're not alone with this, and neither we're the first ones to
fight Python 3.11. So if something's broken, then there's a good chance
that either it's already fixed in the upstream VCS repository or someone
has made a pull request/patch for it.

My usual recommendation is to prefer fixing problems over skipping
tests. However, if something is important enough and the test failures
are minor compared to the package's functionality, you can use
EPYTEST_IGNORE and EPYTEST_DESELECT to skip broken tests conditionally
to Python version. Sometimes it also makes sense to skip some
dependency (python_gen_cond_dep) and possibly tests depending on it
(using has_version). You can find some examples in the Guide, as well
as by grepping the existing ebuilds.

One package we haven't ported yet (and that we would really prefer not
to port) is dev-python/nose. It's unmaintained for years, we're already
carrying a ton of patches to keep it working and with Python 3.11, it's
once again a mess. Plus, the code itself is a mess full of hacks. Good
news is: 3.11 broke it completely, so upstreams can't just ignore
the problem ("it works in my virtualenv"), so there's some hope they
will be for switching to another test runner. That is, unless you're
dealing with yet another unmaintained NIH dependency of an unmaintained
NIH dependency. But there's some chance that you should be able to get
it running with pytest easily or with a reasonably small patch.

I'd also like to ask you to update your packages to EAPI 8 (if possible)
and DISTUTILS_USE_PEP517 while touching them. Setuptools has deprecated
`setup.py install` that the legacy builds are using long time ago. It
doesn't look like it's going to stop working anytime soon but it's
better to avoid running into last minute porting.

The new flags are currently stable-masked on all profiles. Python
3.11.0 final release is planned for October 2022. We're probably going
to stabilize it sometime after, then work on unmasking the flags.


=============================================
Python 3.10 coming as the default interpreter
=============================================
As announced before, we're going to be switching to Python 3.10
in the beginning of June 2022. There shouldn't be any unplanned delay,
and all the most important packages are ready.

If you haven't ported your packages, please do so ASAP.

If something isn't suitable for Python 3.10, it's probably a good idea
to give the users an early warning that it's likely to be lastrited
at some point.

If you need help, please don't hesitate to ask on #gentoo-python.
We can help, point you at the right docs or good examples of ebuilds.


TIA for all you help and have fun with new Pythons!

--
Best regards,
Michał Górny