Help needed with adding user packages as optional/experimental packages

[Matthias Koeppe]

Meta-ticket https://trac.sagemath.org/ticket/31164 proposes to add user packages, such as those listed at ​https://wiki.sagemath.org/SageMathExternalPackages, to the Sage distribution as "pip" packages. This has the following benefits:

• They will be automatically included in our reference manual and Sage website (https://trac.sagemath.org/ticket/29655).
• We have GitHub Actions workflows in place (tox-optional.yml, tox-experimental.yml), which will automatically try to build the packages on each beta release. This provides continuous integration that will allow us to catch unintended breaking changes during the Sage development cycle.

Help is needed with this task. 

- If a package listed at https://wiki.sagemath.org/SageMathExternalPackages is outdated (does not work with a current version of Sage), it will be helpful to add a note to the wiki page and to notify the package authors.

- If a package seems to work with current Sage, help by creating a ticket for the inclusion in Sage and list the ticket in meta-ticket https://trac.sagemath.org/ticket/31164

- If a significant package is neither listed in build/pkgs/ nor in the wiki page, please add it. 

[Matthias Koeppe]

The first few user packages have been added in 9.3.beta7:

ore_algebra, sage_flatsurf, admcycles, slabbe, surface_dynamics.

[Vincent Delecroix]

Great!

Should they be removed from the wiki?
(https://wiki.sagemath.org/SageMathExternalPackages)
The list of "official" user packages are now part of the sage
documentation. We could keep the wiki for the not yet integrated
user packages.

Le 08/02/2021 à 01:49, Matthias Koeppe a écrit :
> The first few user packages have been added in 9.3.beta7:
> ore_algebra, sage_flatsurf, admcycles, slabbe, surface_dynamics.
>
>
> On Wednesday, January 13, 2021 at 7:41:48 PM UTC-8 Matthias Koeppe wrote:
>
>> Meta-ticket https://trac.sagemath.org/ticket/31164 proposes to add user
>> packages, such as those listed at
>> https://wiki.sagemath.org/SageMathExternalPackages, to the Sage
>> distribution as "pip" packages. This has the following benefits:
>>

>> - They will be automatically included in our reference manual and Sage
>> website (https://trac.sagemath.org/ticket/29655).
>> - We have GitHub Actions workflows in place

[Thierry]

Hi,

On Sun, Feb 07, 2021 at 04:49:23PM -0800, Matthias Koeppe wrote:
> The first few user packages have been added in 9.3.beta7:
> ore_algebra, sage_flatsurf, admcycles, slabbe, surface_dynamics.

We should notice that, contrary to other packages, those packages should
be considered as downstream (not upstream), and this should be reflected
in our release process.

Indeed, their code get adapted after Sage changes, see e.g. (randomly
chosen examples):

https://github.com/mkauers/ore_algebra/commit/2d71b50ebad81e62432482facfe3f78cc4961c4f
https://github.com/mkauers/ore_algebra/commit/8c678f9c70aee8df211c04211dd2f7e230a4770f

For upstream packages (especially important ones), they should usually
be merged in the beginning of a new release (on beta[n] with small n),
so that incompatibilities could be reported and fixed before the
official release.

In the current case, it should be the opposite, after downstream adapted
their code to our changes, so that those packages are not broken for the
official release.

Hence, there should be a way to update the versions of such packages
during beta[-1] (the one before rc0), at least those packages should be
tagged "downstream" somewhere (e.g. whithin SPKG.rst file or whatever),
so that someone could have a look before the rc.

While our naming of development releases is pretty poor (beta or rc), we
should perhaps indicate somewhere which intervals are reserved for
backward-incompatible changes, and so on.

Another question: what happens if at the end of a Sage release cycle,
the downstream package is still broken ? Shall we remove the package
from build/pkgs ? Shall we add a temporary patch within the spkg
directory ? Or add a "broken" keyword for the "type" file ?

Ciao,
Thierry

> On Wednesday, January 13, 2021 at 7:41:48 PM UTC-8 Matthias Koeppe wrote:
>
> > Meta-ticket https://trac.sagemath.org/ticket/31164 proposes to add user
> > packages, such as those listed at
> > https://wiki.sagemath.org/SageMathExternalPackages, to the Sage
> > distribution as "pip" packages. This has the following benefits:
> >

> > - They will be automatically included in our reference manual and Sage
> > website (https://trac.sagemath.org/ticket/29655).
> > - We have GitHub Actions workflows in place

> > (tox-optional.yml, tox-experimental.yml), which will automatically try to
> > build the packages on each beta release. This provides continuous
> > integration that will allow us to catch unintended breaking changes during
> > the Sage development cycle.
> >
> > Help is needed with this task.
> > - If a package listed at
> > https://wiki.sagemath.org/SageMathExternalPackages is outdated (does not
> > work with a current version of Sage), it will be helpful to add a note to
> > the wiki page and to notify the package authors.
> > - If a package seems to work with current Sage, help by creating a ticket
> > for the inclusion in Sage and list the ticket in meta-ticket
> > https://trac.sagemath.org/ticket/31164
> > - If a significant package is neither listed in build/pkgs/ nor in the
> > wiki page, please add it.
> >
> >
> >
> >
> >
> >
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/88ba0229-2e6f-4a04-a955-157248c5f90bn%40googlegroups.com.

[Vincent Delecroix]

For most of the packages, there is no explicit version tracking. On
the current rather small list only ore_algebra has an explicit
reference to a commit in requirements.txt

$ cat admcycles/requirements.txt
admcycles

$ cat surface_dynamics/requirements.txt
surface_dynamics

$ cat ore_algebra/requirements.txt
git+https://github.com/mkauers/ore_algebra@6826ac49b4cdf66a563449aced21a2fd1fd085c9#egg=ore_algebra

If instead ore_algebra would properly release on PyPI, the
sage side of things would be simpler.

Best
Vincent

PS: still there is the issue of "old" SageMath versions that might
not be compatible with "recent" package versions. The only mechanism
is to tight a single version of a package to a SageMath version as it
was done with ore_algebra. The latter is quite restrictive given that
packges might evolve between two sage releases.

[Matthias Koeppe]

On Sunday, February 7, 2021 at 11:55:22 PM UTC-8 vdelecroix wrote:

Should they be removed from the wiki?
(https://wiki.sagemath.org/SageMathExternalPackages)
The list of "official" user packages are now part of the sage
documentation. We could keep the wiki for the not yet integrated
user packages.

I would suggest that we remove them from that page when 9.3 is released.

 

[Matthias Koeppe]

On Monday, February 8, 2021 at 4:06:38 AM UTC-8 Thierry (sage-googlesucks@xxx) wrote:

On Sun, Feb 07, 2021 at 04:49:23PM -0800, Matthias Koeppe wrote:
> The first few user packages have been added in 9.3.beta7:
> ore_algebra, sage_flatsurf, admcycles, slabbe, surface_dynamics.

We should notice that, contrary to other packages, those packages should
be considered as downstream (not upstream), and this should be reflected
in our release process.

That's right, this is an important distinction.

In the current case, it should be the opposite, after downstream adapted
their code to our changes, so that those packages are not broken for the
official release.

Hence, there should be a way to update the versions of such packages
during beta[-1] (the one before rc0), at least those packages should be
tagged "downstream" somewhere (e.g. whithin SPKG.rst file or whatever),
so that someone could have a look before the rc.

As Vincent has pointed out, we currently do not use version constraints for these (downstream) user packages. 

I think it should be the responsibility of the downstream package to define a versioning policy - define when they plan to drop support for a particular Sage version. 

Then we can use an appropriate version constraint in the "install-requires.txt" file for this package in build/pkgs.

Another question: what happens if at the end of a Sage release cycle,
the downstream package is still broken ? Shall we remove the package
from build/pkgs ? Shall we add a temporary patch within the spkg
directory ? Or add a "broken" keyword for the "type" file ? 

Setting type to "experimental" would be good enough, I think. This already issues a scary warning.

[Matthias Koeppe]

On Monday, February 8, 2021 at 10:42:59 AM UTC-8 Matthias Koeppe wrote:

I think it should be the responsibility of the downstream package to define a versioning policy - define when they plan to drop support for a particular Sage version. 

Then we can use an appropriate version constraint in the "install-requires.txt" file for this package in build/pkgs.

I meant "requirements.txt" actually.

 

[Thierry]

Hi,

On Mon, Feb 08, 2021 at 10:42:59AM -0800, Matthias Koeppe wrote:
[...]

> As Vincent has pointed out, we currently do not use version
> constraints for
> these (downstream) user packages.

I think that if we do not provide explicit version of downstream
packages, we are going to run into troubles, as we can not expect users
to use latest Sagemath version when they install some downstream
package.

To take the previous example:

https://github.com/mkauers/ore_algebra/commit/8c678f9c70aee8df211c04211dd2f7e230a4770f

without versionning on our side, if a user tries to install ore_algebra
after 7 November 2017 while using 8.0, they would get import errors.

Note also that this downstream change was done during the preparation of
8.1 (that was released on 7 December 2017), so even up-to-date
non-developer users would have been running 8.0 at that time and
therefore get errors.

On Mon, Feb 08, 2021 at 01:17:34PM +0100, Vincent Delecroix wrote:
[...]

> $ cat ore_algebra/requirements.txt
> git+https://github.com/mkauers/ore_algebra@6826ac49b4cdf66a563449aced21a2fd1fd085c9#egg=ore_algebra
>
> If instead ore_algebra would properly release on PyPI, the
> sage side of things would be simpler.

I do not see how having ore_algebra in PyPI would fix the issue of not
providing downstream versions.

Ciao,
Thierry

[Matthias Koeppe]

On Monday, February 8, 2021 at 10:42:59 AM UTC-8 Matthias Koeppe wrote:

I think it should be the responsibility of the downstream package to define a versioning policy - define when they plan to drop support for a particular Sage version. 

Then we can use an appropriate version constraint in the "install-requires.txt" file for this package in build/pkgs.

Another thing that a downstream package can do is declare an "install_requires" on the Sage library, specifying a supported range of versions. Then we should be able to use the dependency resolution by pip to be sure to install a compatible version.

Note https://trac.sagemath.org/ticket/30912 updates the metadata of the Sage library, defining the distribution name "sagemath-standard". (Another distribution that could be used for versioning is "sage_conf".)

Also note the pip dependency resolver has just changed with the update to pip 20.3.3 in #30589 (see https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020).

 

[Vincent Delecroix]

I think that the most important thing we care about is the answer to

What is the most recent version of package X that
runs on Sage version Y?

On the downstream side, one could just add a requirement

sagemath>=Y

meaning that sagemath < Y is not supported.

However, I don't know enough of pip to understand how to make
"pip install X" retrieve the information. Would that work out
of the box with sage 9.3.beta7?

Best
Vincent

[Simon King]

Hi Thierry,

On 2021-02-08, Thierry <sage-goo...@lma.metelu.net> wrote:
> We should notice that, contrary to other packages, those packages should
> be considered as downstream (not upstream), and this should be reflected
> in our release process.
>
> Indeed, their code get adapted after Sage changes, see e.g. (randomly
> chosen examples):
>
> https://github.com/mkauers/ore_algebra/commit/2d71b50ebad81e62432482facfe3f78cc4961c4f
> https://github.com/mkauers/ore_algebra/commit/8c678f9c70aee8df211c04211dd2f7e230a4770f

Then, I think, p_group_cohomology should be considered downstream, too.
Most of its latest releases were triggered by changes in Sage.

Best regards,
Simon

[Matthias Koeppe]

9.3.beta8 adds:

snappy