VOTE: Follow NEP 29: Recommended Python version

[Tobias Diez]

Dear Sage developers,

the NumPy enhancement proposal 29: "Recommend Python and Numpy version support as a community policy standard" (available at https://numpy.org/neps/nep-0029-deprecation_policy.html) specifies when it's okay to drop support for old Python version. 

Namely, a release should support "all minor versions of Python released 42 months prior to the project, and at minimum the two latest minor versions. ". In particular, this means:

- Currently, Sage should support > 3.8.

- On Apr 05, 2024 we should drop support for Python 3.9 (initially released on Oct 05, 2020)

Based on previous discussions on this topic (https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ, https://github.com/sagemath/sage/issues/30384, https://github.com/sagemath/sage/pull/35403), I'm calling for a vote on adapting the Python version support of NEP 29 in Sage. Voting ends at the 7th June,  AoE. Please use this thread only for sending votes, to make it easier to count them afterward; and use the thread https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ for discussion.

Summary of the points brought forward in the discussions linked above

1. The current practice in Sage is to evaluate whether to increase the minimum version of Python supported at the beginning of each release cycle. With regard to such a practice, the NEP 29 documents remarks "As there is no objective threshold to when the minimum version should be dropped, it is easy for these version support discussions to devolve into bike shedding and acrimony." Sadly, an example of this can be found in the current discussion of dropping Python 3.8 support in https://github.com/sagemath/sage/pull/35404 with emotions running so high that sage-abuse had to step in. Adopting a version policy would prevent such discussions. On the other hand, by following a given policy, we would loose some flexibility.

2. The main idea of NEP 29 is to have a community-wide standard. It is followed by many scientific packages such as Scipy, Matplotlib, IPython, Jupyter, Pandas, scikit, astropy, cuda, cirq, jax, pytorch among others. The adoption of NEP 29 will harmonize Sage's deprecation policy with these other major libraries.

3. The NEP 29 drop schedule is much faster than the EOL schedule of Python itself. Python 3.8 is supported until 2024-10, but NEP 29 already drops it 2023-04. However, adhering to the EOL schedule would prevent us to updating these packages that follow NEP 29.

4. The NEP 29 schedule is about one release cycle faster than the previous drops (e.g. Python 3.7 support was dropped in Sage 9.7 while according to NEP 29 it would have been Sage 9.6).

5. The faster drop schedule will free developer resources (less systems to test) and potentially increase developer productivity as it allows us to use newer language features.

6. The faster drop schedule might be inconvenient for users who rely on older Python versions. To some extend this is remedied by our python install package, and relatively straightforward upgrade paths on most system. One should also note that users relying on other scientific python packages are likely forced to upgrade anyway, since these other packages likely follow NEP 29.

7. The faster drop schedule would force users to upgrade to newer Python versions and thereby profit from fewer bugs and security issues. It is however questionable if Sage should play this educator role.

8. One of the main goals of NEP 29 is to improve downstream project planning by having a community-wide standard. This is currently not very relevant for us as Sage is currently upstream of nothing except for some user packages. With the modularization effort, this may change in the future.

9. There are not many other documented policies. As said above, most scientific python projects follow NEP 29. Most projects in web development (e.g flask) seem to drop a version once it reaches EOL. Machine learning projects follow a similar EOL policy (e.g. tensorflow) or roughly follow NEP 29 (scikit-learn). Some end-user applications have even stricter version constraints then NEP 29 (e.g. home-assistant only supports the two latest minor releases).

[Dima Pasechnik]

I am for adapting NEP 29 for Sage (and thus dropping Python 3.8 now).

Few comments on the Summary:

2) it's very important to be in sync with various Sage's upstream
parts, as much as possible. We don't want to lag behind, preventing us
from
being able to use latest versions.

3) Saying that Python 3.8 is still supported, by python.org, is not
100% correct. (it's supported as being on life support :-))
Python 3.8 is still getting security updates - it does not get any
other bugs fixed.
The only Python version getting the bugfixes is now 3.11, the older
Python versions are legacy versions. See
https://devguide.python.org/versions/

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/d321fa7e-47d0-40ed-9468-e090b11cb86fn%40googlegroups.com.

[Matthias Koeppe]

Thanks, Tobias, for opening this vote thread. Here on sage-devel, this is a much better setting than what you attempted in https://github.com/sagemath/sage/pull/35404#issuecomment-1504474945

I am voting NO.

There's a simple rationale: 

I. This proposed policy change does not solve any problem. There are no problems whatsoever with how we have managed the support of Python versions since 2020 (when it became possible to use system Python instead of only the Python from our SPKG.) 

II. The proposed policy change creates new problems. Following this policy would force us to drop support for a particular Python version at times when it would be harmful for our project. Specifically, right now it would *force* us to drop support for Python 3.8 and hence for using the default Python on Ubuntu Linux 20.04 (an LTS release, with "End of Standard Support" April 2025 and "End Of Life" April 2030. It is the very point of LTS releases to provide a stable software environment; so, yes, Python 3.8 is fully supported, and if Python 3.8.x had bugs relevant for Sage, we would know about it because we are testing. 

III. Our existing practice is to carefully consider and weigh various factors that are relevant for the Sage project, rather than following a fixed schedule that is set by an external, largely separate community. I briefly explained what we do in https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/x3qOdPB5BQAJ; but I'll expand here on some important factors:

a) Sage has a dual role as a library ("project") and as a distribution. NEP 29 was designed for projects, and not for software distributions.

b) In Sage, we only have one line of releases. Hence users who want any bug fixes need to use our latest version. In contrast, just like Python itself, many other projects have at least two separate branches: A branch on which the cutting edge development takes place (new features etc.), and a branch from which maintenance updates are made. For example, NumPy removed support of Python 3.8 in their development branch earlier this year; but this in preparation for the 1.25 release expected this summer. NumPy continued to make maintenance releases on the 1.24 branch (https://github.com/numpy/numpy/releases), and by policy, these maintenance upgrades never drop the support of a previously supported version.

c) NEP29 was designed for and is in use by a part of the scientific Python community, to address the need to be able to use features of new Python versions and features of NumPy/SciPy faster. This is important for many projects that have NumPy/SciPy as their dependencies. 

d) In contrast, our uses of NumPy/SciPy in the Sage library are very basic and dating back by about a decade; with the exception of the optional use of a recent SciPy feature (the high-performance optimization solver HiGHS, see https://github.com/sagemath/sage/wiki/Sage-10.0-Release-Tour#linear-programming-and-extensions), which motivated our quick upgrade to the current SciPy version in Sage. And as another example, also our use of matplotlib in the library dates back by a decade or more; we regularly have to update when new matplotlib versions come out that make API changes, but we haven't picked up any new features in a very long time.

e) Yes, synchronization between projects matters for maintainability. But Sage is downstream of lots of Python packages; before we can offer support for a new version of Python, we often have to wait until all or most of our dependencies provide support for that new version. For example, some projects are actively working on support for the Python 3.12 release expected this early Fall; but for us, this is not actionable because we have to wait for critical dependencies; see https://github.com/sagemath/sage/issues/34788 . Likewise, it is not useful for us to drop support for an old version before there is a clear benefit for us, brought for example by important upgrades that have dropped support already.

(As suggested by Tobias, any discussion of my explanations above is best sent to the https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ thread.)

[dim...@gmail.com]

On Fri, May 26, 2023 at 06:34:25AM -0700, Matthias Koeppe wrote:
> Thanks, Tobias, for opening this vote thread. Here on sage-devel, this is a
> much better setting than what you attempted
> in https://github.com/sagemath/sage/pull/35404#issuecomment-1504474945
>
> I am voting NO.
>
> There's a simple rationale:
>
> I. This proposed policy change does not solve any problem. There are no
> problems whatsoever with how we have managed the support of Python versions
> since 2020 (when it became possible to use system Python instead of only
> the Python from our SPKG.)

this is not true. It solves a range of problems, e.g., in no particular
order:
1) lack of hands to support all that obsolete versions,
2) blocking new Python features in 3.9 from being used in Sage
3) falling behind w.r.t. versions of various Python packages used in Sage,
packages which are already merging 3.8-incompatible code, e.g. networkx,
scipy, etc.
4) keeping special backported to Python 3.8 packages (at least one such
package exists)

>
> II. The proposed policy change creates new problems. Following this policy
> would force us to drop support for a particular Python version at times
> when it would be harmful for our project. Specifically, right now it would
> *force* us to drop support for Python 3.8 and hence for using the default
> Python on Ubuntu Linux 20.04 (an LTS release, with "End of Standard
> Support" April 2025 and "End Of Life" April 2030. It is the very point of
> LTS releases to provide a stable software environment;

No, this is not a problem, as Ubuntu Linux 20.04 provides a Python 3.9
system package, which can be installed and used if desired.
(And there are other options, e.g. pyenv, conda, etc)
And we have a long story of supporting Linux distros with system Python
being much older than the one needed by Sage.
It's also not clear what you propose in this respect - support Python
3.8 until Ubuntu 20.04 reaches EOL (which is much longer than Python's
3.8. EOL).

> so, yes, Python 3.8
> is fully supported, and if Python 3.8.x had bugs relevant for Sage, we
> would know about it because we are testing.

Python 3.8 is not "fully supported". It's on life support, only
receiving security-related fixes, but no other fixes.
The only fully supported Python is 3.11, which does get bug fixes
for all the bugs found.

>
> III. Our existing practice is to carefully consider and weigh various
> factors that are relevant for the Sage project, rather than following a
> fixed schedule that is set by an external, largely separate community. I
> briefly explained what we do in
> https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/x3qOdPB5BQAJ; but
> I'll expand here on some important factors:
>
> a) Sage has a dual role as a library ("project") and as a distribution. NEP
> 29 was designed for projects, and not for software distributions.

No, Sage is just a project, with lots of dependencies (way too many).
It's not a software distribution in any way, it does not include
essential tools to build it (e.g. no C/C++ compiler on macos).

>
> b) In Sage, we only have one line of releases. Hence users who want any bug
> fixes need to use our latest version. In contrast, just like Python itself,

no, Sage is more like Python - you don't get the bug fixed in Python 3.10 or
older!

> many other projects have at least two separate branches: A branch on which
> the cutting edge development takes place (new features etc.), and a branch
> from which maintenance updates are made. For example, NumPy removed support
> of Python 3.8 in their development branch earlier this year; but this in
> preparation for the 1.25 release expected this summer.

Do you propose to fall behind Numpy?
Dropping Python 3.8 now will let us to be ready to upgrade our Numpy.

> NumPy continued to
> make maintenance releases on the 1.24 branch
> (https://github.com/numpy/numpy/releases), and by policy, these maintenance
> upgrades never drop the support of a previously supported version.
>
> c) NEP29 was designed for and is in use by a part of the scientific Python
> community, to address the need to be able to use features of new Python
> versions and features of NumPy/SciPy faster. This is important for many
> projects that have NumPy/SciPy as their dependencies.
>
> d) In contrast, our uses of NumPy/SciPy in the Sage library are very basic
> and dating back by about a decade;

No, not true. E.g.

schemes/riemann_surfaces/riemann_surface.py is neither "basic" nor
dating back that much - it's also under relatively active development, see its header:

- Alexandre Zotine, Nils Bruin (2017-06-10): initial version
- Nils Bruin, Jeroen Sijsling (2018-01-05): algebraization, isomorphisms
- Linden Disney-Hogg, Nils Bruin (2021-06-23): efficient integration
- Linden Disney-Hogg, Nils Bruin (2022-09-07): Abel-Jacobi map

It's using Voronoi diagrams from scipy.

> with the exception of the optional use
> of a recent SciPy feature (the high-performance optimization solver HiGHS,
> see
> https://github.com/sagemath/sage/wiki/Sage-10.0-Release-Tour#linear-programming-and-extensions),
> which motivated our quick upgrade to the current SciPy version in Sage. And
> as another example, also our use of matplotlib in the library dates back by
> a decade or more; we regularly have to update when new matplotlib versions
> come out that make API changes, but we haven't picked up any new features
> in a very long time.
>
> e) Yes, synchronization between projects matters for maintainability. But
> Sage is downstream of lots of Python packages; before we can offer support
> for a new version of Python, we often have to wait until all or most of our
> dependencies provide support for that new version. For example, some
> projects are actively working on support for the Python 3.12 release
> expected this early Fall; but for us, this is not actionable because we
> have to wait for critical dependencies;

I don't see how this is relevant - it's actually beneficial to limit the
old versions of Python supported in Sage, as we don't want to get
blocked by any package which stops supporting Python 3.8. As everything
we use supports Python 3.9+, we don't need Python 3.8 already now.
Once again, it's not about picking up new features, it's about dropping
old ones, which is a completely different story.

> see https://github.com/sagemath/sage/issues/34788 . Likewise, it is not
> useful for us to drop support for an old version before there is a clear
> benefit for us, brought for example by important upgrades that have dropped
> support already.

So far you failed to list any benefits.

>
>
> (As suggested by Tobias, any discussion of my explanations above is best
> sent to
> the https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ
> thread.)

sorry, it will get lost there, so I'm replying here instead.

Dima

>
>
> On Friday, May 26, 2023 at 3:12:16 AM UTC-7 Tobias Diez wrote:
>
> > Dear Sage developers,
> >
> > the NumPy enhancement proposal 29: "Recommend Python and Numpy version
> > support as a community policy standard" (available at
> > https://numpy.org/neps/nep-0029-deprecation_policy.html) specifies when
> > it's okay to drop support for old Python version.
> >
> > Namely, a release should support "all minor versions of Python released 42
> > months prior to the project, and at minimum the two latest minor versions.
> > ". In particular, this means:
> > - Currently, Sage should support > 3.8.
> > - On Apr 05, 2024 we should drop support for Python 3.9 (initially
> > released on Oct 05, 2020)
> >
> > Based on previous discussions on this topic (
> > https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ,
> > https://github.com/sagemath/sage/issues/30384,
> > https://github.com/sagemath/sage/pull/35403), I'm calling for a vote on
> > adapting the Python version support of NEP 29 in Sage. Voting ends at the
> > 7th June, AoE. Please use this thread only for sending votes, to make it
> > easier to count them afterward; and use the thread
> > https://groups.google.com/g/sage-devel/c/j1cwbTU8aOU/m/2sTiwdKPBQAJ for
> > discussion.
> >

> > *Summary *of the points brought forward in the discussions linked above

> > 1. The current practice in Sage is to evaluate whether to increase the
> > minimum version of Python supported at the beginning of each release cycle.
> > With regard to such a practice, the NEP 29 documents remarks "As there is
> > no objective threshold to when the minimum version should be dropped, it is
> > easy for these version support discussions to devolve into bike shedding

> > <https://en.wikipedia.org/wiki/Wikipedia:Avoid_Parkinson%27s_bicycle-shed_effect>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f6bf61cd-373f-4829-bfba-392ce32daeban%40googlegroups.com.

[William Stein]

Hi,

To help with people who want to make an informed decision, is there
any public discussion of the original NEP 29 proposal?

The only thing I could find was this post from Sebastian Berg, where he says at

https://mail.python.org/pipermail/numpy-discussion/2019-October/080128.html

"We propose formally accepting the NumPy enhancement proposal 29... If
there are no objections within a week it may be accepted.".
There's no public voting or anything, and there's exactly one quick
random response of "yes" in the thread.

In the actual NEP 29 proposal
https://numpy.org/neps/nep-0029-deprecation_policy.html there is a
section labeled "Discussion" and it is empty.
I would love to read about the discussions for/against NEP 29 from
when they decided on it in the first place!

There are follow up discussions, just like this one, by other
projects, e.g., for sympy here:
https://github.com/sympy/sympy/issues/21884, which
is still not decided, and has been under regular discussion for nearly
2 years. There are many other similar conversations. But they are
all about whether to align with numpy or not, and the core question of
why it is best to ignore what the official upstream python supports
isn't as relevant.

The original NEP 29 says "The Python release cadence increased in PEP
0602, [...] Thus, we do not expect our users to upgrade Python faster,
and our 42 month support window will cover the same portion of the
upstream support of any given Python release." I don't really know
what that means, but I have the impression that NEP 29 tried to very
rigidly define end of life by a specific timeline with no flexibility
for the potential that the official Python release timelines are not
rigid and fixed in stone forever, while simultaneously acknowledging
this conflict. I would love to see the arguments for doing that,
which could be compelling. I fully realize that this is something
that came from maintainers of open source software, and they were
probably feeling annoyed and burned out, and this NEP 29 may have
helped them keep their sanity. But if that's the case, it's decided
in secret as far as I can tell.

-- William

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f6bf61cd-373f-4829-bfba-392ce32daeban%40googlegroups.com.

--
William (http://wstein.org)

[William Stein]

On Fri, May 26, 2023 at 7:57 AM <dim...@gmail.com> wrote:
> > a) Sage has a dual role as a library ("project") and as a distribution. NEP
> > 29 was designed for projects, and not for software distributions.
>
> No, Sage is just a project, with lots of dependencies (way too many).
> It's not a software distribution in any way, it does not include
> essential tools to build it (e.g. no C/C++ compiler on macos).

Since I started the project in 2005 and until now Sage is definitely
both a big python library *and* a distribution.
I've given a million talks with a slide about how Sage is both a new
library of code and a distribution. Being a
distribution was the whole reason people would consider using open
source software for math, instead of
sticking with Maple or Magma -- at least it was possible that they
could get it running on their computers. Also,
it helped to make other open source math software beyond just sage
more accessible.

In the nearly two decades since starting Sage, software distribution
and the Python
ecosystem have improved enough that there is hope that Sage could
transition to just
being a bunch of libraries, and all the distribution gets handled by some
third party distribution such as conda (etc.). That's been discussed
with great optimism
recently on this list.

I hope soon Sage isn't a distribution, but right now it still is.

--
William (http://wstein.org)

[Matthias Koeppe]

On Friday, May 26, 2023 at 7:57:53 AM UTC-7 dim...@gmail.com wrote:

On Fri, May 26, 2023 at 06:34:25AM -0700, Matthias Koeppe wrote:

> I am voting NO.

>
> I. This proposed policy change does not solve any problem. There are no
> problems whatsoever with how we have managed the support of Python versions
> since 2020 (when it became possible to use system Python instead of only
> the Python from our SPKG.)
this is not true. It solves a range of problems, e.g., in no particular
order:

I'll be replying to one these at a time...

> d) In contrast, our uses of NumPy/SciPy in the Sage library are very basic

> and dating back by about a decade;
No, not true. E.g.

schemes/riemann_surfaces/riemann_surface.py is neither "basic" nor
dating back that much - it's also under relatively active development, see its header:

- Alexandre Zotine, Nils Bruin (2017-06-10): initial version
- Nils Bruin, Jeroen Sijsling (2018-01-05): algebraization, isomorphisms
- Linden Disney-Hogg, Nils Bruin (2021-06-23): efficient integration
- Linden Disney-Hogg, Nils Bruin (2022-09-07): Abel-Jacobi map

It's using Voronoi diagrams from scipy.

Well, scipy.spatial.Voronoi was added in SciPy 0.12.0 (https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.Voronoi.html), released pretty much exactly a decade ago -- https://pypi.org/project/scipy/#history

Next?

[Dima Pasechnik]

1) You wrote: "OUR USES of NumPy/SciPy are ... dating back by about a decade".
2) I replied that it's not true, Voronoi's USE in Sage in much more recent.
3) You are now trying to invalidate what I wrote by telling us about
the history of Voronoi in SciPy. How is it relevant to 1) and 2) ?
See, what you wrote is FACTUALLY INCORRECT. Please admit it, otherwise
I don't see a way to continue a discussion with you.



>
> Next?

>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/baead8c6-65d2-4799-92c9-3e6c14ad5a62n%40googlegroups.com.

[William Stein]

On Fri, May 26, 2023 at 9:19 AM Dima Pasechnik <dim...@gmail.com> wrote:
> Please admit it, otherwise. I don't see a way to continue a discussion with you.

Can you please continue to engage, but view this as a public debate
for the benefit of all sage developers, rather than a discussion with
Matthias? It's a debate. It's the sort of heated discussion about
NEP 29 that I wish I could read, but (maybe) I can't because some
"powers that be" just decided on NEP 29 in private (which is their
right of course as volunteers and open source maintainers).


-- William

--
William (http://wstein.org)

[Tobias Diez]

Here is the PR that introduced NEP 29: https://github.com/numpy/numpy/pull/14086. The main discussion happened in person at scipy 2019 and in webcalls. But a lot of the points raised here are answered in this PR.

For example, the point about Linux LTS / Python EOL is addressed by one of the writers of NEP 29 as follows (https://github.com/numpy/numpy/pull/14086#issuecomment-516661140)

"

Conversely, I am not sure of a real world example where a user must have:

• 4 year old version of Python
• just-released version of numpy / Matplotilb / scikit-learn [add sage here]
  

If you are making the choice to stay on an old version of Python (which there are good reasons to do!) then why would you not also make the conservative choice to stay on the contemporaneous version of the scipy stack?

"

The point that "NEP is about libraries and not downstream applications" is also not valid since this clearly was part of the discussion back then. For example, one of the core maintainers of jupyter and ipython writes https://github.com/numpy/numpy/pull/14086#issuecomment-516904953

" There is also the same catch 22 than dropping python 2; downstream was not dropping because upstream was still supporting Python 2, and upstream was still supporting Python 2 because downstream was relying on it."

In other words, if all downstream applications would have a more conservative policy, then naturally also numpy etc would need to adapt. NEP 29 is about homogenizing the supported Python version in the scientific python community. This is also very clearly written in the first line of the NEP: "recommends that all projects across the Scientific Python ecosystem adopt a common “time window-based” policy for support of Python and NumPy versions" (emphasize mine). Its "all projects", not "all upstream libraries". So unless someone wants to argue that sage is not part of the scientific python ecosystem, we are directly addressed.

By the way, what happened to the "please let the discussion go somewhere else"? Should I restart the voting in a new thread?

[Tobias Diez]

> I have the impression that NEP 29 tried to very
rigidly define end of life by a specific timeline with no flexibility
for the potential that the official Python release timelines are not

rigid and fixed in stone forever

The PR linked above mentions quite often that they do take variations/changes of the Python release schedule into account. For example, NEP 29 writes " support at least all minor versions of Python introduced and released in the prior 42 months from the anticipated release date with a minimum of 2 minor versions of Python". With a rigid schedule, this is a unnecessary duplication - but the "at least two minor versions" is there to prevent the worst case scenario where python releases are delayed by a few years and no python version would be supported otherwise. But yes, naturally there are some included expectations about the Python release schedule included in NEP 29 and it would probably be revised if Python changes its schedule. Conversely, NEP 29 is adapted by so many libraries that Python itself includes it in discussions about its release schedule (e.g. https://peps.python.org/pep-0605/#implications-for-the-proposed-scientific-python-ecosystem-support-period)

[Oscar Benjamin]

On Fri, 26 May 2023 at 16:19, William Stein <wst...@gmail.com> wrote:
>
> On Fri, May 26, 2023 at 7:57 AM <dim...@gmail.com> wrote:
> > > a) Sage has a dual role as a library ("project") and as a distribution. NEP
> > > 29 was designed for projects, and not for software distributions.
> >
> > No, Sage is just a project, with lots of dependencies (way too many).
> > It's not a software distribution in any way, it does not include
> > essential tools to build it (e.g. no C/C++ compiler on macos).

...

>
> In the nearly two decades since starting Sage, software distribution
> and the Python
> ecosystem have improved enough that there is hope that Sage could
> transition to just
> being a bunch of libraries, and all the distribution gets handled by some
> third party distribution such as conda (etc.). That's been discussed
> with great optimism
> recently on this list.
>
> I hope soon Sage isn't a distribution, but right now it still is.

Usually a distribution tries to include everything so that it can
constrain the versions. That would imply that if Sage is a
distribution then it should also distribute its own Python or the Sage
versions should be constrained by the Python versions. Then it would
not be necessary for any single version of Sage to have compatibility
with multiple versions of Python. A Linux distro does not try to
support many different Python versions using the same versions of
Python packages. Each version of the distro will update the Python
version and also the version of NumPy, SymPy etc simultaneously.

The intention of NEP 29 is for various libraries such as NumPy to
coordinate on ensuring that there exists some mutually consistent set
of versions that can be used together either right now or some time in
the future. It isn't necessary for newer NumPy versions to support
significantly older Python versions because if you want to use an
older version of Python then you can just use an older version of
NumPy (which is precisely what a distro would do in any case). There
are then two ways that those consistent versions get used:

- Some people will use Python packaging tools to install the latest
versions of things from PyPI and the most recent versions of packages
are designed to be consistent with the most recent versions of Python.
- Distributions like conda or Linux distros will use some consistent
set of older versions i.e. an older version of Python *and* older
versions of the packages.

It sounds to me like Sage is trying to live in between these two
things and potentially ends up trying to mix older and newer versions
together which will always be painful because no one else is trying to
make that work.

What is wrong with Sage just saying that an older version of an
operating system only works with an older version of Sage?

Maybe if you want to use the latest version of Sage on some old
version of Debian then you should just install a newer version of
Python first?

Sage 10 might be better than Sage 9 but Sage 9 was still good before
Sage 10 came along. Is it so bad that someone might need to wait until
they are ready to upgrade other things before getting the latest
version of Sage?

There are many different levels at which you can ask these questions
about exactly which new things should be compatible with which older
things. The purpose of NEP 29 is that there should exist some rolling
set of consistent package versions that aligns with the releases of
Python itself. It is then up to downstream to decide whether they want
the latest stuff fresh off the press today or if they would rather
wait and use 5 years old versions of things.

> There are follow up discussions, just like this one, by other projects, e.g., for sympy here:
> https://github.com/sympy/sympy/issues/21884,

The situation for SymPy is very different from NumPy because it is
really not a big burden for SymPy to support Python 3.8 being only
pure Python. SymPy's release is just a single wheel that works on any
OS, all versions of Python, both CPython and PyPy etc. NumPy has to
provide and test many more binaries and also uses CPython's C API
which has bigger cost in compatibility problems across Python
versions.

That being said the cost to SymPy is not zero and in some ways
discussions like this about which version to support can just be time
wasted that could be spent on better things. In the SymPy issue
Matthias suggested that it is useful for Sage if SymPy continues to
have wider version support. Since it is not a big cost it is fine for
SymPy to do that. Otherwise though I would probably push for SymPy
adopting NEP 29 just because it is a documented policy and it is good
if everyone both in the project and downstream can see a clear policy
and knows what to expect.

--
Oscar

[Matthias Koeppe]

On Friday, May 26, 2023 at 9:19:52 AM UTC-7 Dima Pasechnik wrote:

On Fri, May 26, 2023 at 5:01 PM Matthias Koeppe
<matthia...@gmail.com> wrote:
> > d) In contrast, our uses of NumPy/SciPy in the Sage library are very basic
> > and dating back by about a decade;
> No, not true. E.g.
> schemes/riemann_surfaces/riemann_surface.py is neither "basic" nor
> dating back that much - it's also under relatively active development, see its header:

> - Alexandre Zotine, Nils Bruin (2017-06-10) [...]  (2022-09-07): Abel-Jacobi map

> It's using Voronoi diagrams from scipy.
>
> Well, scipy.spatial.Voronoi was added in SciPy 0.12.0 (https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.Voronoi.html), released pretty much exactly a decade ago -- https://pypi.org/project/scipy/#history

1) You wrote: "OUR USES of NumPy/SciPy are ... dating back by about a decade".
2) I replied that it's not true, Voronoi's USE in Sage in much more recent.
3) You are now trying to invalidate what I wrote by telling us about
the history of Voronoi in SciPy. How is it relevant to 1) and 2) ?
See, what you wrote is FACTUALLY INCORRECT. Please admit it

Yes, I confess that "dating back" was not worded clearly enough. I'm sorry that I offended you with that.

I should have said "our uses of NumPy/SciPy are of NumPy/SciPy features that date back by about a decade".

Which is exactly the point I was making: When we update NumPy/SciPy, it is typically NOT because of the needs of the Sage library for new features but because we want to stay on supported versions, for picking up portability fixes, etc., and for the convenience of the maintainers of downstream packaging of Sage in Linux distributions.

The only exception that I'm aware of being the high-performance optimization solver HiGHS, see https://github.com/sagemath/sage/wiki/Sage-10.0-Release-Tour#linear-programming-and-extensions). 

Do you know others?

[Michael Orlitzky]

On Fri, 2023-05-26 at 18:15 +0100, Oscar Benjamin wrote:
>
> What is wrong with Sage just saying that an older version of an
> operating system only works with an older version of Sage?

Matthias alluded to this when he mentioned that we only have one
release branch of sage. Our version numbers are not really meaningful,
and bug fixes and features are released simultaneously. So the only way
to get fixes for critical bugs is to upgrade, but the upgrades come
with new features, and those new features can have new requirements.

That said, I still share your sentiment when there is a good reason (a
term best left undefined) to break compatibility.

[Emmanuel Charpentier]

I'd like to stress one important point already made : dropping support for old/antique/paleontologic versions of Python lightens the maintainance workload of the *small* team of Sage developpers.

Given the size and the state of this team, this point should *not* be taken lightly...

I'd also like to hear on this point from our release manager : would this proposed policy change have an influence on his workload ?

HTH,

[Tobias Diez]

> a) Sage has a dual role as a library ("project") and as a distribution. NEP 29 was designed for projects, and not for software distributions.

What problems do you see that come from sage having aspects of a software distribution?

"Sage-the-distribution" allows us to reduce the inconvenience for end-users when their (system) python version is too old. In this case, sage automatically installs a newer Python. So I would argue that NEP29 is actually a great fit for a hybrid application like sage since the distribution aspect reduces some of the disadvantages that come with the quicker drop schedule of NEP29.

[Dima Pasechnik]

One big plus in favour of NEP 29 for "Sage the project" is that it
makes the number of
Python versions to support smaller. As we are trying to move to using
unvendored Python packages,
i.e. these that come with "system Python", where the latter refers to
the unvendored Python used by Sage
instead of its own Python3 (which we can already drop, as it is an
unneeded burden, just as well as gcc), it
reduces the complexity of the upstream packages to support.

And NEP 29 has very little influence on "Sage the distribution", zero, in fact.

>
> It sounds to me like Sage is trying to live in between these two
> things and potentially ends up trying to mix older and newer versions
> together which will always be painful because no one else is trying to
> make that work.
>
> What is wrong with Sage just saying that an older version of an
> operating system only works with an older version of Sage?

Not much, indeed.

>
> Maybe if you want to use the latest version of Sage on some old
> version of Debian then you should just install a newer version of
> Python first?

yes, sure. E.g. pyenv makes it easy.

Dima

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxSj%2BdK0nRVW49VL5zLMxOw68xwSWhdDQrN%3DDqynAV%3Dvvw%40mail.gmail.com.

[Matthias Koeppe]

On Sunday, May 28, 2023 at 3:31:07 AM UTC-7 Dima Pasechnik wrote:

[...] we are trying to move to using

unvendored Python packages,
i.e. these that come with "system Python", where the latter refers to
the unvendored Python used by Sage
instead of its own Python3

Would you mind elaborating what you mean by this?

Maybe you are talking about using the system's Jupyter/JupyterLab installation (https://github.com/sagemath/sage/issues/30306)?

Other than that, to my knowledge there is no current effort to make use of system Python packages in the Sage distribution.

(see https://github.com/sagemath/sage/issues/29023, https://github.com/sagemath/sage/issues/29665)

[Dima Pasechnik]

On Sun, 28 May 2023, 15:43 Matthias Koeppe, <matthia...@gmail.com> wrote:

On Sunday, May 28, 2023 at 3:31:07 AM UTC-7 Dima Pasechnik wrote:

[...] we are trying to move to using
unvendored Python packages,
i.e. these that come with "system Python", where the latter refers to
the unvendored Python used by Sage
instead of its own Python3

Would you mind elaborating what you mean by this?

indeed, https://github.com/sagemath/sage/issues/29023 is the meta-ticket to oversee this process. Needless to say this needs more effort.

I can also point at the already existing possibility  to use Conda's Python packages on a Conda-based install.

Maybe you are talking about using the system's Jupyter/JupyterLab installation (https://github.com/sagemath/sage/issues/30306)?

Other than that, to my knowledge there is no current effort to make use of system Python packages in the Sage distribution.

(see https://github.com/sagemath/sage/issues/29023, https://github.com/sagemath/sage/issues/29665)

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/af2cae2a-d11f-46a6-80cb-265a70a9af33n%40googlegroups.com.

[Matthias Koeppe]

On Sunday, May 28, 2023 at 8:20:18 AM UTC-7 Dima Pasechnik wrote:

On Sun, 28 May 2023, 15:43 Matthias Koeppe, <matthia...@gmail.com> wrote:

On Sunday, May 28, 2023 at 3:31:07 AM UTC-7 Dima Pasechnik wrote:

[...] we are trying to move to using
unvendored Python packages,
i.e. these that come with "system Python", where the latter refers to
the unvendored Python used by Sage
instead of its own Python3

Would you mind elaborating what you mean by this?

indeed, https://github.com/sagemath/sage/issues/29023 is the meta-ticket to oversee this process. Needless to say this needs more effort.

Speaking as someone who has worked on this, I'll just add that I do not see this as a promising or important direction for the Sage project. Mixing system Python packages with pip-installed packages is very tricky, and I consider it unlikely that we can create a stable solution based on this for the Sage distribution.

I can also point at the already existing possibility  to use Conda's Python packages on a Conda-based install.

This mode of installation (https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental) bypasses the Sage distribution entirely and installs the Sage library separately using pip.

[Tobias Diez]

I can also point at the already existing possibility  to use Conda's Python packages on a Conda-based install.

This mode of installation (https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental) bypasses the Sage distribution entirely and installs the Sage library separately using pip.

And there were problems with this mode of installation for Python 3.8 that no one had the time to fix. On the other hand, it works perfectly well and is tested via github actions for all NEP29-supported Python versions. So this serves as a prime example of how supporting multiple Python versions can be challenging.

 

[Matthias Koeppe]

What you are describing is a downstream distribution problem, not a Sage problem. It is not under control of our project what conda packages are available and working for what Python version.

It does point to an important issue: That if we abandon the Sage distribution in favor of using conda-forge as our reference environment --- as recently discussed as a possibility --- then we are losing control of the necessary software environment for running tests. So if we go this way, we need to find a way to motivate Sage developers to engage in matters of conda-forge packaging.

[Dima Pasechnik]

On Sun, May 28, 2023 at 4:31 PM Matthias Koeppe

<matthia...@gmail.com> wrote:
>
> On Sunday, May 28, 2023 at 8:20:18 AM UTC-7 Dima Pasechnik wrote:
>
> On Sun, 28 May 2023, 15:43 Matthias Koeppe, <matthia...@gmail.com> wrote:
>
> On Sunday, May 28, 2023 at 3:31:07 AM UTC-7 Dima Pasechnik wrote:
>
> [...] we are trying to move to using
> unvendored Python packages,
> i.e. these that come with "system Python", where the latter refers to
> the unvendored Python used by Sage
> instead of its own Python3
>
>
> Would you mind elaborating what you mean by this?
>
>
> indeed, https://github.com/sagemath/sage/issues/29023 is the meta-ticket to oversee this process. Needless to say this needs more effort.
>
>
> Speaking as someone who has worked on this, I'll just add that I do not see this as a promising or important direction for the Sage project. Mixing system Python packages with pip-installed packages is very tricky, and I consider it unlikely that we can create a stable solution based on this for the Sage distribution.

You misunderstood. By "system" I don't mean the OS, I mean an
environment which does not need Python packages vendored by Sage;
Conda is one example of this.

I certainly agree that insisting on OS Python is not a good idea.

>
> I can also point at the already existing possibility to use Conda's Python packages on a Conda-based install.
>
>
> This mode of installation (https://doc.sagemath.org/html/en/installation/conda.html#using-conda-to-provide-all-dependencies-for-the-sage-library-experimental) bypasses the Sage distribution entirely and installs the Sage library separately using pip.

So what? How is this relevant to the question at hand?

>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/9d902562-7219-46c0-a273-8c7b657b0b19n%40googlegroups.com.

[Michael Orlitzky]

On 2023-05-28 16:20:02, Dima Pasechnik wrote:
>
> indeed, https://github.com/sagemath/sage/issues/29023 is the meta-ticket to
> oversee this process. Needless to say this needs more effort.

I'm sure it's out of date for boring reasons, but the branch from
29665 worked great:

https://github.com/sagemath/sagetrac-mirror/compare/develop...u/mjo/ticket/29665

There were the usual packaging snafus but I don't recall any looming
conceptual problems.

[G. M.-S.]

My vote is empty, for the following reasons:

—I think the question asked is not clear enough, as per reactions.

—My "dream" is having an easy to install recent version of SageMath for everybody wishing to do mathematics with it.  Currently this is only the case for macOS and perhaps some flavours of Linux, AFAICT (my experience with my students and colleagues is not very good, especially under Windows).

Guillermo