Follow NEP 29: Recommended Python version

[Tobias Diez]

Hi all,

I propose to follow the NumPy enhancement proposal 29: "Recommend Python and Numpy version support as a community policy standard" available at: https://numpy.org/neps/nep-0029-deprecation_policy.html

In essence it specifies when it's okay to drop support for old Python version. Namely, a release should support "all minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions. ". In particular, this means:

- On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)

- On Apr 05, 2024 drop support for Python 3.9 (initially released on Oct 05, 2020)

The main idea of NEP 29 is to have a community-wide standard to prevent issues with e.g. dependencies dropping support for older Python versions to early. It is followed by many scientific packages such as Scipy, Matplotlib, and IPython, among others. Sage itself is also roughly following it (based on the past drops, e.g. Python 3.7 support was dropped 9 months ago and NEP recommends Dec 26, 2021). But as far as I'm aware there is not yet a clear policy for when to drop support for older versions.

Of course, the dates in the NEP 29 are only a guideline and not set in stone, e.g. you don't want to drop support at the end of the release cycle just because the NEP says so. But for example, Sage 10 will be most likely released after Apr 14 2023, so we can now already drop support for Python 3.8.

Relevant issue: https://github.com/sagemath/sage/issues/30384

[Matthias Koeppe]

For a bit of necessary context: The NEP 29 drop schedule is much faster than the EOL schedule of Python itself. Python 3.8 is supported until 2024-10, but NEP 29 already drops it 2023-04.

NEP 29 is intended to improve interoperability of fast-moving scientific software. It is quite relevant where in the dependency streams a project is. NumPy, SciPy are *upstream* of many projects and *downstream* of nothing. In contrast, Sage is *downstream* of lots and lots of projects, and currently *upstream* of nothing except for some user packages.

Effectively I don't think we need to make any commitment to a particular schedule at a time, as the upgrades of major dependencies (mostly NumPy, SciPy, NetworkX, SymPy) just force us to drop support for a particular Python version.

For example, a SciPy upgrade to the 1.10 series is due for 10.0 (and already positively reviewed in https://github.com/sagemath/sage/pull/35082); it still supports Python 3.8. The following SciPy release 1.11 (which will drop Python 3.8) is not coming out before 2023-07. So we would gain nothing by dropping Python 3.8 already in a Sage 10.0 release some this spring.

Of course, another argument for dropping support for a Python version (such as Python 3.8) would be that it is missing critical features that hold us back in the development of the Sage library. Issue https://github.com/sagemath/sage/issues/29756 has a bit of this discussion.

[Dima Pasechnik]

I think it is a great idea. Even if the system python is older than one needed, one can still use e.g. pyenv to get an up to date python, without the need to build your own.

And pyenv does allow us to drop python3 spkg,

something we should do, too.

Dima 

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/a232c964-b216-4e4f-86f0-b987c1455fc6n%40googlegroups.com.

[Matthias Koeppe]

For easy reference: Tobias has proposed a vote on this question in the sage-devel thread https://groups.google.com/g/sage-devel/c/3Zoq0CNE1hE

On Friday, February 24, 2023 at 7:10:07 PM UTC-8 Tobias Diez wrote:

[Volker Braun]

NEP 29 just defines the minimum supported Python version, having a longer support window is equally NEP 29 compatible. 

Since we depend on numpy, sage-the-library inherits NEP 29. There isn't really anything to vote on here.

As far as sage-the-distribution goes, we should tie the support window to popular distros. Ideally by liberally calling "conda install python" whenever the distro python is too old.

Its unclear to me if this or the vote thread is about the library code or the distribution...

[Tobias Diez]

> Since we depend on numpy, sage-the-library inherits NEP 29.

Correct. In practice, we waited for the numpy release that drops a certain python version and then dropped that version in tandem with the upgrade of numpy. Thus, there was about one release cycle between the drop schedule of NEP 29 and the actual drop happening in sage. Formally adopting NEP29 would allow us to drop older version a little bit faster (i.e. without needing for the numpy release).

> library code or the distribution

At the moment they have the same minimum Python version, right? Sage-the-distro installs a newer version if the system version doesn't fit the minimum version of sage-the-library. It is not the plan of this proposal to change this behavior (but +1 for using conda in this context). Did I understand your question correctly?

[Matthias Koeppe]

On Saturday, May 27, 2023 at 4:02:27 AM UTC-7 Volker Braun wrote:

NEP 29 just defines the minimum supported Python version, having a longer support window is equally NEP 29 compatible. 

That was my initial interpretation too when I became interested in NEP 29 in 2020 and opened the Trac ticket https://github.com/sagemath/sage/issues/30384.

For example, when we had achieved support for Python 3.6–3.9 in Sage 9.2, I wrote in the Release Tour "Hence, Sage 9.2 conforms to (and exceeds) NumPy Enhancement Proposal 29 regarding Python version support policies." (https://wiki.sagemath.org/ReleaseTours/sage-9.2#Support_for_Python_3.6.2C_3.8.2C_and_3.9_added)

However, in https://github.com/sagemath/sage/issues/30384#issuecomment-1418097192 (2020-12), Tobias claimed that adopting NEP 29 means that we should (or must?) drop support for Python versions when the NEP 29 schedule says so.

And this strict implementation is what Tobias's branch https://github.com/sagemath/sagetrac-mirror/compare/develop...public/python3/nep29 proposed to add into our developer's guide. (Now it's PR https://github.com/sagemath/sage/pull/35403/files)  

I strongly disagree with adopting this strict policy. 

I have little interest in discussing what the "correct interpretation" of NEP 29 is; such discussions are usually not very productive.

My point is that this strict interpretation would be a poor fit for our project, as it would lead to much worse decisions on version support than what we have been doing.

[Matthias Koeppe]

On Saturday, May 27, 2023 at 7:24:11 AM UTC-7 Tobias Diez wrote:

> library code or the distribution

At the moment they have the same minimum Python version, right? Sage-the-distro installs a newer version if the system version doesn't fit the minimum version of sage-the-library. It is not the plan of this proposal to change this behavior (but +1 for using conda in this context).

I agree with Tobias that it is probably not useful to make a distinction between library code and the distribution when considering this question.

I view Sage-the-distribution as the "reference distribution" for developing and testing the Sage library. 

- Because we are able to take python3 from the system (default if available, configure --with-system-python3), we can test the support for a range of Python versions found in distributions

- Because the Sage distribution also brings our own Python (configure --without-system-python), we can also conveniently test experimental or patched versions of Python (e.g. for Python 3.12 - https://github.com/sagemath/sage/issues/34788).

If the Sage library were to support a larger version range than Sage-the-distribution were to support, we would have to test this version support by another mechanism, separate from Sage-the-distribution. Of course, we are already doing this, for example using the experimental version of the conda workflow; and our downstream packages also do this. But nevertheless it would create unnecessary friction for developers.

(The other direction, having the Sage distribution support a larger version range than the Sage library, does have some plausible use cases. For example, using the Sage distribution people can install selected software in a convenient way without actually installing Sage. And I have provided portability CI scripts to some upstream projects (https://github.com/sagemath/sage/issues/33338). which reuse Sage's CI and the Sage distribution; at some point I had to patch the distribution so it accepts a wider system Python range.)

[Dima Pasechnik]

On Sun, 28 May 2023, 16:18 Matthias Koeppe, <matthia...@gmail.com> wrote:

On Saturday, May 27, 2023 at 7:24:11 AM UTC-7 Tobias Diez wrote:

> library code or the distribution

At the moment they have the same minimum Python version, right? Sage-the-distro installs a newer version if the system version doesn't fit the minimum version of sage-the-library. It is not the plan of this proposal to change this behavior (but +1 for using conda in this context).

I agree with Tobias that it is probably not useful to make a distinction between library code and the distribution when considering this question.

I view Sage-the-distribution as the "reference distribution" for developing and testing the Sage library. 

- Because we are able to take python3 from the system (default if available, configure --with-system-python3), we can test the support for a range of Python versions found in distributions

- Because the Sage distribution also brings our own Python (configure --without-system-python), we can also conveniently test experimental or patched versions of Python (e.g. for Python 3.12 - https://github.com/sagemath/sage/issues/34788).

If the Sage library were to support a larger version range than Sage-the-distribution were to support, we would have to test this version support by another mechanism, separate from Sage-the-distribution.

I don't understand - Sage the distribution is already on a newer than 3.8 Python, whereas Sage library does support 3.8.

Of course, we are already doing this, for example using the experimental version of the conda workflow; and our downstream packages also do this. But nevertheless it would create unnecessary friction for developers.

It would? It is already creating problems for distributions which want to package Sage, as Sage is often far behind, version-wise, in what Python packages it is using.

And the reluctance to drop old Python versions

only contributes to this friction.

Dima

(The other direction, having the Sage distribution support a larger version range than the Sage library, does have some plausible use cases. For example, using the Sage distribution people can install selected software in a convenient way without actually installing Sage. And I have provided portability CI scripts to some upstream projects (https://github.com/sagemath/sage/issues/33338). which reuse Sage's CI and the Sage distribution; at some point I had to patch the distribution so it accepts a wider system Python range.)

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/d78591c5-e6f9-4e2b-bb46-9b9b2903a1d7n%40googlegroups.com.

[Matthias Koeppe]

On Sunday, May 28, 2023 at 9:18:50 AM UTC-7 Dima Pasechnik wrote:

On Sun, 28 May 2023, 16:18 Matthias Koeppe, <matthia...@gmail.com> wrote:

If the Sage library were to support a larger version range than Sage-the-distribution were to support, we would have to test this version support by another mechanism, separate from Sage-the-distribution.

I don't understand - Sage the distribution is already on a newer than 3.8 Python, whereas Sage library does support 3.8.

"Support", not "ship". As you are aware, through our spkg-configure mechanism, we accept system Python 3.8–3.11. (https://github.com/sagemath/sage/blob/develop/build/pkgs/python3/spkg-configure.m4)

spkg-configure is a mechanism of the Sage distribution, not of the Sage library. The version check is not just for ensuring that the Sage library works with the system Python, but also for all the Python packages that the Sage distribution ships.

The Python version check of the Sage library, on the other hand, is done by "pip" on the basis of "python_requires" defined in src/setup.cfg[.m4] (https://github.com/sagemath/sage/blob/develop/src/setup.cfg.m4).

[Dima Pasechnik]

On Sun, May 28, 2023 at 7:38 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
>
> On Sunday, May 28, 2023 at 9:18:50 AM UTC-7 Dima Pasechnik wrote:
>
> On Sun, 28 May 2023, 16:18 Matthias Koeppe, <matthia...@gmail.com> wrote:
>
> If the Sage library were to support a larger version range than Sage-the-distribution were to support, we would have to test this version support by another mechanism, separate from Sage-the-distribution.
>
>
> I don't understand - Sage the distribution is already on a newer than 3.8 Python, whereas Sage library does support 3.8.
>
>
> "Support", not "ship". As you are aware, through our spkg-configure mechanism, we accept system Python 3.8–3.11. (https://github.com/sagemath/sage/blob/develop/build/pkgs/python3/spkg-configure.m4)
>
> spkg-configure is a mechanism of the Sage distribution, not of the Sage library.

This is misleading, and not correct for most of the spkg-configure.m4 files.
Python's spkg-configure.m4 is responsible for
testing whether it's possible to find Python3 good enough to be used with Sage.
The difference with "pure" library building is that you'd need to
invoke ./configure --with-system-python3=force
to get ./configure to bail out on not finding a good Python3, rather
than attempting to roll their own.

Similarly, e.g. eclib's spkg-configure.m4 is responsible for
testing whether libec.so is present and is good enough to build
various Python extensions using it, e.g.
homspace.cpython-39-x86_64-linux-gnu.so

> The version check is not just for ensuring that the Sage library works with the system Python, but also for all the Python packages that the Sage distribution ships.
>
> The Python version check of the Sage library, on the other hand, is done by "pip" on the basis of "python_requires" defined in src/setup.cfg[.m4] (https://github.com/sagemath/sage/blob/develop/src/setup.cfg.m4).

This check is just a formality. Why would it ever fail if the
corresponding check in ./configure succeeds?

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/b88ae2a0-8d12-4e68-b261-aa2a23efa40fn%40googlegroups.com.