Releasing SymPy 1.13

[Oscar Benjamin]

Hi all,

It is past time that a new release of SymPy is needed. Let's get SymPy
1.13 out ASAP.

I had been meaning to put out a bugfix release of SymPy 1.12.1 because
of a few things that have been fixed since SymPy 1.12 but it didn't
happen. In particular compatibility with Python 12's ast module for
parsing string expressions was fixed and a few other small things.
Also some changes were recently made for compatibility with NumPy 2.0
which is probably going to be released in a few months but we should
get out a SymPy release first.

There are of course also many other improvements since SymPy 1.12. You
can see the release notes here:
https://github.com/sympy/sympy/wiki/Release-Notes-for-1.13

I've created a release issue here to track releasing 1.13:
https://github.com/sympy/sympy/issues/26169

Please say if there are any release blocker issues or other things
that should be resolved before the release.

Here is one from me:

After the 0.6.0 release of python-flint I would like to get this PR
merged to use Flint for polynomials and matrices over GF(p) as well as
ZZ and QQ:
https://github.com/sympy/sympy/pull/25940

A larger issue to be resolved is how exactly SymPy should use
python-flint. Currently with the SymPy master branch it is necessary
to set an environment variable SYMPY_GROUND_TYPES=flint to get SymPy
to use python-flint. That is not how it works with gmpy2 though, which
will just be picked up automatically when it is installed without the
need to set an environment variable.

In principle the use of python-flint should not change the behaviour
of SymPy at all except that some particular things are made a lot
faster. Currently those things are:

- Univariate polynomials over ZZ, QQ (and GF(p) if the PR above is
merged). This means various functions like `factor`, `gcd` etc are
faster.
- Algebraic extension fields like QQ.algebraic_field(sqrt(2)) since
their ANP type uses Flint's univariate polynomials as well now.
- The dense format of DomainMatrix when the domain is ZZ, QQ (or GF(p)
if the PR above is merged).

These changes are very particular about what exactly is or is not made
faster but some particular operations are made much faster and for
others it makes no difference. In future Flint's multivariate
polynomials will be used and that will end up having a more noticeable
impact on the speed of higher-level operations for most end users.

The question is whether to be conservative about making use of
python-flint by requiring an opt-in environment variable like this.
There are two possible downsides for a given user if SymPy starts
using python-flint automatically:

1. It is not yet that widely tested by many different people on
different computers etc so it is possible that there are unknown bugs.
2 I expect that many things will change in python-flint over the next
few years. It is possible that a SymPy release of 1.13 now will not be
fully compatible with say a future release of python-flint in a few
years time.

Note that python-flint is not as widely used or installed as SymPy by
a long way. I doubt that there are many people who just happen to have
python-flint and SymPy installed and would then be surprised that
SymPy started using python-flint. Probably most of the time the fact
that someone has installed both packages together is because they want
SymPy to use python-flint.

I think that probably the right way to do this is just for SymPy to
use python-flint by default if it is installed but with an almost
exact version check like `python-flint == 0.6.x` because `0.6.0` is
the minimum version needed for the PR above but also the maximum
version that has been tested before release. It would be good to allow
the flexibility to have `0.6.1` though in case there is a small bugfix
release of python-flint. Over time if things seem more stable then we
could relax that version constraint. The environment variable could
still be used to override the version check but the default if no
environment variable is set should just be to use python-flint but
only the expected 0.6.x versions.

Does anyone have any thoughts on how exactly to do this?

--
Oscar

[Aaron Meurer]

I think it should just be enabled by default. Almost no one will have
flint installed, so only people who go out of their way to install it
will get the benefits, so requiring an opt-in is just an extra step
for those people. The only way this wouldn't be the case is if we add
flint as a dependency in the conda package like we do for gmpy, but it
sounds like it's a bit early to do that.

Although I'm wondering about Sage. Does Sage always install both SymPy
and python-flint?

As for the verison checks, it depends on how likely you expect there
to be breaking changes in python-flint. If that package is going to
have a lot of breaking changes, it will make using it in SymPy a lot
harder.

Aaron Meurer

>
> --
> Oscar
>
> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxTb9K-2uLsCCzfZGHK9HmFekM1LKHQJYVbys60aPr25pg%40mail.gmail.com.

[Oscar Benjamin]

On Fri, 2 Feb 2024 at 21:52, Aaron Meurer <asme...@gmail.com> wrote:
>
> On Fri, Feb 2, 2024 at 9:41 AM Oscar Benjamin
> <oscar.j....@gmail.com> wrote:
> >
> > I think that probably the right way to do this is just for SymPy to
> > use python-flint by default if it is installed but with an almost
> > exact version check like `python-flint == 0.6.x` because `0.6.0` is
> > the minimum version needed for the PR above but also the maximum
> > version that has been tested before release. It would be good to allow
> > the flexibility to have `0.6.1` though in case there is a small bugfix
> > release of python-flint. Over time if things seem more stable then we
> > could relax that version constraint. The environment variable could
> > still be used to override the version check but the default if no
> > environment variable is set should just be to use python-flint but
> > only the expected 0.6.x versions.
> >
> > Does anyone have any thoughts on how exactly to do this?
>
> I think it should just be enabled by default. Almost no one will have
> flint installed, so only people who go out of their way to install it
> will get the benefits, so requiring an opt-in is just an extra step
> for those people. The only way this wouldn't be the case is if we add
> flint as a dependency in the conda package like we do for gmpy, but it
> sounds like it's a bit early to do that.
>
> Although I'm wondering about Sage. Does Sage always install both SymPy
> and python-flint?

Sage always includes SymPy but not python-flint but it might do now
that SymPy adds the optional dependency:
https://github.com/sagemath/sage/pull/37224#issuecomment-1924466879

Sage already includes and uses Flint separately with its own wrapper
but python-flint would be an additional package whose purpose for now
would just be to be used by SymPy inside of Sage. I hope that in
future there will be some convergence between python-flint and Sage's
Flint wrapper. It isn't possible for SymPy to use Sage's Flint wrapper
(outside of Sage) but it could be possible in future for both to use
python-flint or for the Sage wrapper to become something separate from
Sage.

> As for the verison checks, it depends on how likely you expect there
> to be breaking changes in python-flint. If that package is going to
> have a lot of breaking changes, it will make using it in SymPy a lot
> harder.

I don't anticipate many breaking changes but python-flint is in a more
active development phase than SymPy and I doubt that we will want to
test all version combinations (just like we don't with mpmath or numpy
etc).

Actually that reminds me that some of the changes in SymPy 1.13 are
needed for the next release of mpmath (whenever that happens). Now
that mpmath is also in a more active development phase than it has
been for a long time we should review the version checking there as
well. Currently SymPy checks for a 10 year old version of mpmath as
the minimum version:
https://github.com/sympy/sympy/blob/0b25b279a5f089885cfd1b5e7685515338695f58/setup.py#L37

Without making python-flint a hard dependency there is no way to
specify what versions should be acceptable in an install so I think
that the safest thing for now is to keep an upper bound that is
checked at runtime. We also definitely need a lower bound because so
far SymPy (master branch) uses the latest python-flint features as
they roll out.

I'll add a python-flint == 0.6.x check for now. In future I don't
think it will be necessary and in the meantime someone can use the
environment variable to force the use of python-flint if they want to
but have a different version.

--
Oscar

[Aaron Meurer]

Well if Sage is going to do that intentionally to enable Flint support
in SymPy then we shouldn't worry about it.

I don't know about pip, but conda does let you specify version pins
for optional dependencies (it won't install the dependency but if it
is installed it will make sure that version is satisfied). But I would
be careful with this because it would make it impossible for people to
upgrade python-flint in their environment. We used to pin the
antlr-python-runtime like this, but then it made it difficult for
other packages using antlr, so we removed it (see
https://github.com/conda-forge/sympy-feedstock/issues/46).

Generally speaking, aggressive version pins can end up biting you.
There's a long blog post about this here if you are interested
https://iscinumpy.dev/post/bound-version-constraints/. My suggestion
at this point would be to not pin in the package metadata. If you're
really worried about it, you can add soft checks in the code, which
are either just warnings or can be overridden somehow like you
suggested.

Aaron Meurer

>
> I'll add a python-flint == 0.6.x check for now. In future I don't
> think it will be necessary and in the meantime someone can use the
> environment variable to force the use of python-flint if they want to
> but have a different version.
>
> --
> Oscar
>

> --
> You received this message because you are subscribed to the Google Groups "sympy" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxQUfFTsOyQVdVSL3a33auXZT%3D6u0zCXi8Z33SvHJXk8Fw%40mail.gmail.com.

[Oscar Benjamin]

On Fri, 2 Feb 2024 at 22:36, Aaron Meurer <asme...@gmail.com> wrote:
>
> On Fri, Feb 2, 2024 at 3:28 PM Oscar Benjamin
> <oscar.j....@gmail.com> wrote:
> >
> > Without making python-flint a hard dependency there is no way to
> > specify what versions should be acceptable in an install so I think
> > that the safest thing for now is to keep an upper bound that is
> > checked at runtime. We also definitely need a lower bound because so
> > far SymPy (master branch) uses the latest python-flint features as
> > they roll out.
>
> I don't know about pip, but conda does let you specify version pins
> for optional dependencies (it won't install the dependency but if it
> is installed it will make sure that version is satisfied). But I would
> be careful with this because it would make it impossible for people to
> upgrade python-flint in their environment. We used to pin the
> antlr-python-runtime like this, but then it made it difficult for
> other packages using antlr, so we removed it (see
> https://github.com/conda-forge/sympy-feedstock/issues/46).

It is not possible to do this in the PyPI package metadata. Either
python-flint is a hard dependency or there is no version constraint.

> Generally speaking, aggressive version pins can end up biting you.
> There's a long blog post about this here if you are interested
> https://iscinumpy.dev/post/bound-version-constraints/. My suggestion
> at this point would be to not pin in the package metadata. If you're
> really worried about it, you can add soft checks in the code, which
> are either just warnings or can be overridden somehow like you
> suggested.

To be clear I am just talking about a runtime check for the version here:
https://github.com/sympy/sympy/blob/0b25b279a5f089885cfd1b5e7685515338695f58/sympy/external/gmpy.py#L90-L147

Basically if the environment variable is not set then check if
python-flint is installed with version 0.6.x and if so then use it. In
future the same logic can apply but perhaps with a looser version
check.

--
Oscar

[Matthias Köppe]

On Friday, February 2, 2024 at 2:28:43 PM UTC-8 Oscar wrote:

> Although I'm wondering about Sage. Does Sage always install both SymPy
> and python-flint?

Sage always includes SymPy but not python-flint but it might do now
that SymPy adds the optional dependency:
https://github.com/sagemath/sage/pull/37224#issuecomment-1924466879

Sage already includes and uses Flint separately with its own wrapper
but python-flint would be an additional package whose purpose for now
would just be to be used by SymPy inside of Sage.

I'll note that if python-flint is only declared as an optional dependency of SymPy, it is likely that we wouldn't make python-flint a standard package in Sage immediately.

 

I hope that in
future there will be some convergence between python-flint and Sage's
Flint wrapper. It isn't possible for SymPy to use Sage's Flint wrapper
(outside of Sage) but it could be possible in future for both to use
python-flint or for the Sage wrapper to become something separate from
Sage.

FWIW, in https://github.com/sagemath/sage/pull/35095, I have been preparing a distribution package sagemath-flint, which has relatively modest requirements.

See https://github.com/mkoeppe/sage/tree/t/32432/modularization_of_sagelib__break_out_a_separate_package_sagemath_polyhedra/pkgs/sagemath-flint

I hope that these modularized distribution packages will become available in Q2 2024.

Matthias