Sage CI Failing in SymPy GitHub Actions

[Tirthankar Mazumder]

In the SymPy GitHub repository, currently the only CI jobs that are failing are the Sage CI ones. I was investigating the workflows, and found that there are many instances of this warning:

make -j8 build/make/Makefile --stop

make[1]: Entering directory '/sage'

make -j8 build/make/Makefile --stop

make[2]: Entering directory '/sage'

make[2]: warning: -j8 forced in submake: resetting jobserver mode.

make -j8 build/make/Makefile --stop

make[3]: Entering directory '/sage'

make[3]: warning: -j8 forced in submake: resetting jobserver mode.

make -j8 build/make/Makefile --stop

make[4]: warning: -j8 forced in submake: resetting jobserver mode.

This error is repeated more than 5000 times, and then bash finally kills the process.

According to this StackOverflow answer, to silence this warning, the number of jobs have to stay at the default number. I investigated some of the uploaded artifacts, and found the lines

ARG NUMPROC=8
ENV MAKE="make -j${NUMPROC}"

in the Dockerfile uploaded in the logs.

While I am not sure whether this is the source of the error or not, silencing this warning should make finding the root issue easier. I was unable to figure out how this Dockerfile is constructed, so I am sharing my findings here.

Regards,

Tirthankar Mazumder

[Matthias Koeppe]

Fixed in https://github.com/sympy/sympy/pull/23728 via https://trac.sagemath.org/ticket/34118

[Tirthankar Mazumder]

Sorry to resurrect such an old thread, but the Sage CI is failing for SymPy again. This time, it looks to me like the issue is the fact that GitHub Actions is trying to pull the Sage code from trac, which shouldn't be happening because SageMath moved to GitHub some time back.

Can somebody please update the Sage CI for SymPy so that it works?

Regards,

Tirthankar Mazumder

[Dima Pasechnik]

On Sun, Sep 24, 2023 at 9:21 AM Tirthankar Mazumder
<greenw...@gmail.com> wrote:
>
> Sorry to resurrect such an old thread, but the Sage CI is failing for SymPy again. This time, it looks to me like the issue is the fact that GitHub Actions is trying to pull the Sage code from trac, which shouldn't be happening because SageMath moved to GitHub some time back.

Which actions are you referring to? Something here:
https://github.com/scipy/scipy/actions
?
Somewhere else?
Please point out at concrete GitHub Actions where you see this.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/fa45c22e-f080-4510-b319-8de945c64920n%40googlegroups.com.

[Oscar Benjamin]

On Sun, 24 Sept 2023 at 09:50, Dima Pasechnik <dim...@gmail.com> wrote:
>
> On Sun, Sep 24, 2023 at 9:21 AM Tirthankar Mazumder
> <greenw...@gmail.com> wrote:
> >
> > Sorry to resurrect such an old thread, but the Sage CI is failing for SymPy again. This time, it looks to me like the issue is the fact that GitHub Actions is trying to pull the Sage code from trac, which shouldn't be happening because SageMath moved to GitHub some time back.
>
> Which actions are you referring to? Something here:
> https://github.com/scipy/scipy/actions
> ?
> Somewhere else?
> Please point out at concrete GitHub Actions where you see this.

We're talking about this:

https://github.com/sympy/sympy/blob/master/.github/workflows/ci-sage.yml
https://github.com/sympy/sympy/actions/runs/6284388889/job/17065717301

Before the GitHub transition this CI job was set to test the master
branch of SymPy against a particular trac ticket. The trac ticket
would hold the changes needed for Sage to update to the next SymPy
versions. When this CI job was working properly I submitted updates to
the trac ticket (mostly just updating doctests) so that Sage was ready
to update to the next SymPy version. The job is useful if it usually
passes so that we can see when a change in SymPy breaks something but
it just fails all the time right now.

I'm not sure what the equivalent of that trac ticket would be now (a
pull request somewhere?) or how to set up the CI job so that it can
track that ticket.

Mostly the changes needed before were to do with changes in SymPy's
printing output and also Laplace transform calculations. The Laplace
transform code in SymPy has largely been rewritten in recent releases
including significant changes to inverse_laplace_transform since the
last release that have not been tested against Sage as far as I know.
Probably some Sage doctests need to be updated already.

--
Oscar

[Dima Pasechnik]

OK, sorry, as usual for me I mixed up sympy and scipy :-)

yes, trac tickes became github PRs.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAHVvXxTQ_eazEZyaM%3Dz4oZYoup4rEWfhg0BXFNyLZ9%3Dy7DDOXg%40mail.gmail.com.

[Dima Pasechnik]

it seems that there are many old versions of this ci-sage.yml floating
aroung on the net, see e.g.
https://github.com/pypa/setuptools/blob/main/.github/workflows/ci-sage.yml
I've opened https://github.com/pypa/setuptools/issues/4057

[Matthias Koeppe]

PR https://github.com/sympy/sympy/pull/25728 updates the workflow. Apologies for not submitting this fix earlier.

PRs updating the Sage doctests for changes in SymPy are very welcome. 

[Oscar Benjamin]

On Sun, 24 Sept 2023 at 18:47, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> PR https://github.com/sympy/sympy/pull/25728 updates the workflow. Apologies for not submitting this fix earlier.
> PRs updating the Sage doctests for changes in SymPy are very welcome.

Thanks Matthias.

Where would I send the PR for the Sage doctests?

Previously there was an open Sage ticket that was for updating SymPy
and was intended to be compatible with the SymPy master branch. If
there was a doctest change because of e.g. a change in SymPy between
1.12 and master then the updated doctest just went into the ticket
rather than being merged directly into Sage. Then the SymPy CI job
would use that ticket so that it was testing the combination of SymPy
changes since previous SymPy release with the changes that would also
be needed in Sage to update to the next SymPy release.

Now the CI job says:

> sage_ref: develop

Does that mean that I should send PRs to a Sage develop branch? Would
that already test with SymPy's master branch?

--
Oscar

--
Oscar

[Matthias Koeppe]

I've added this to ci-sage.yml to explain what to do:

      # To test with a Sage PR, use this:
      #sage_ref:                 refs/pull/PR_NUMBER/merge

[Oscar Benjamin]

> On Sunday, September 24, 2023 at 11:46:05 AM UTC-7 Oscar Benjamin wrote:
>
>> Where would I send the PR for the Sage doctests?

On Sun, 24 Sept 2023 at 20:48, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> I've added this to ci-sage.yml to explain what to do:
>
> # To test with a Sage PR, use this:
> #sage_ref: refs/pull/PR_NUMBER/merge

So currently the SymPy Sage CI job points at:
https://github.com/sagemath/sage/pull/36276

So if there are doctest updates needed I guess the workflow is:

- Open a PR against sagemath/sage with the needed (doctest or other) changes
- Make SymPy's Sage CI job point at that PR

Then when the time comes for Sage to update its SymPy version someone
will incorporate the changes from that PR somehow?

--
Oscar

[Matthias Koeppe]

On Sunday, September 24, 2023 at 2:10:26 PM UTC-7 Oscar Benjamin wrote:

> On Sunday, September 24, 2023 at 11:46:05 AM UTC-7 Oscar Benjamin wrote:
>
>> Where would I send the PR for the Sage doctests?

On Sun, 24 Sept 2023 at 20:48, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> I've added this to ci-sage.yml to explain what to do:
>
> # To test with a Sage PR, use this:
> #sage_ref: refs/pull/PR_NUMBER/merge

So currently the SymPy Sage CI job points at:
https://github.com/sagemath/sage/pull/36276

Yes, this PR contains a hotfix for the build system (which missed its way into the currrent beta version 10.2.beta4).

 

So if there are doctest updates needed I guess the workflow is:

- Open a PR against sagemath/sage with the needed (doctest or other) changes
- Make SymPy's Sage CI job point at that PR

Then when the time comes for Sage to update its SymPy version someone
will incorporate the changes from that PR somehow?

Yes.

[Oscar Benjamin]

On Sun, 24 Sept 2023 at 22:43, Matthias Koeppe <matthia...@gmail.com> wrote:
>
> On Sunday, September 24, 2023 at 2:10:26 PM UTC-7 Oscar Benjamin wrote:
>
> > On Sunday, September 24, 2023 at 11:46:05 AM UTC-7 Oscar Benjamin wrote:
> >
> >> Where would I send the PR for the Sage doctests?
>
> On Sun, 24 Sept 2023 at 20:48, Matthias Koeppe <matthia...@gmail.com> wrote:
> >
> > I've added this to ci-sage.yml to explain what to do:
> >
> > # To test with a Sage PR, use this:
> > #sage_ref: refs/pull/PR_NUMBER/merge
>
> So currently the SymPy Sage CI job points at:
> https://github.com/sagemath/sage/pull/36276
>
> Yes, this PR contains a hotfix for the build system (which missed its way into the currrent beta version 10.2.beta4).

That is a PR against the sagemath/sage develop branch.

Should any new PR with doctest updates be based off of that PR and set
to merge into develop?

There are some doctest failures:
https://github.com/sympy/sympy/actions/runs/6292982121/job/17082929352

Some are due to this printing change:
https://github.com/sympy/sympy/pull/25673#issuecomment-1719198065

I'm not sure yet that I like that printing change but also not sure
what Sage would want to do about it if SymPy keeps it. Would Sage just
want to accept that change in output?

An example failure is:

**********************************************************************
8567File "src/sage/doctest/forker.py", line 151, in
sage.doctest.forker.init_sage
8568Failed example:
8569 print(PrettyPrinter(settings={'wrap_line': True}).doprint(s))
8570Expected:
8571 29 28 27 26 25 24 23 22 21 20 19 18 17
8572 x + x + x + x + x + x + x + x + x + x + x + x + x +
8573 <BLANKLINE>
8574 16 15 14 13 12 11 10 9 8 7 6 5 4 3
8575 x + x + x + x + x + x + x + x + x + x + x + x + x + x + x
8576 <BLANKLINE>
8577 2
8578 + x
8579Got:
8580 29 28 27 26 25 24 23 22 21 20 19 18 17 ↪
8581 x + x + x + x + x + x + x + x + x + x + x + x + x + ↪
8582 <BLANKLINE>
8583 ↪ 16 15 14 13 12 11 10 9 8 7 6 5 4 3 ↪
8584 ↪ x + x + x + x + x + x + x + x + x + x + x + x + x + x + ↪
8585 <BLANKLINE>
8586 ↪ 2
8587 ↪ x + x

Basically some ↪ arrows are used when wrapping long output lines.

Another doctest failure is due to this change in hyper. The doctest
just needs updating:
https://github.com/sympy/sympy/pull/25641

Failed example:
8800 hypergeometric((5, 4), (4, 4), 3)._sympy_() # needs sage.symbolic
8801Expected:
8802 hyper((5, 4), (4, 4), 3)
8803Got:
8804 hyper((5,), (4,), 3)

There is another doctest failure that I don't understand:

File "src/sage/groups/finitely_presented.py", line 1771, in
sage.groups.finitely_presented.FinitelyPresentedGroup.characteristic_varieties
9400Failed example:
9401 G.characteristic_varieties(groebner=True)
9402Expected:
9403 [[(f1 - 1, f2 - 1, f3 - 1),
9404 (f1 + 1, f2 - 1, f3 - 1),
9405 (f1 - 1, f2 - 1, f3 + 1),
9406 (f3^2 + 1, f1 - f3, f2 - f3),
9407 (f1 - 1, f2 + 1, f3 - 1)],
9408 [(f1 - 1, f2 - 1, f3 - 1),
9409 (f1*f3 + 1, f2 - 1),
9410 (f1*f2 + 1, f3 - 1),
9411 (f2*f3 + 1, f1 - 1),
9412 (f2*f3 + 1, f1 - f2),
9413 (f2*f3 + 1, f1 - f3),
9414 (f1*f3 + 1, f2 - f3)]]
9415Got:
9416 [[(f1 - 1, f2 - 1, f3 - 1),
9417 (f1 + 1, f2 - 1, f3 - 1),
9418 (f1 - 1, f2 - 1, f3 + 1),
9419 (f3^2 + 1, f1 - f3, f2 - f3),
9420 (f1 - 1, f2 + 1, f3 - 1)],
9421 [(f2*f3 + 1, f1 - f2),
9422 (f2*f3 + 1, f1 - f3),
9423 (f2*f3 + 1, f1 - 1),
9424 (f1*f3 + 1, f2 - f3),
9425 (f1 - 1, f2 - 1, f3 - 1),
9426 (f1*f2 + 1, f3 - 1),
9427 (f1*f3 + 1, f2 - 1)]]

How exactly is Sage using SymPy in this calculation?

It looks like the order of the output has changed but I don't know
what caused that change or if that should be considered a bug (in
either SymPy or Sage). Does the order matter here?

Another doctest failure is just a change in the printing code:

File "src/sage/symbolic/expression.pyx", line 1196, in
sage.symbolic.expression.Expression._unicode_art_
14804Failed example:
14805 unicode_art(SR(1.3 - I))
14806Expected:
14807 1.3 - ⅈ
14808Got:
14809 1.3 - 1.0⋅ⅈ

The SymPy expression being printed here does have a 1.0 in it. I don't
know if it is intentional that Sage puts the 1.0 there but the
printing change is correct for SymPy. Either the docstring needs to be
updated or perhaps Sage should not be putting the 1.0 there (I don't
know what Sage intends to do in this context).

--
Oscar

[Matthias Koeppe]

On Monday, September 25, 2023 at 4:55:01 AM UTC-7 Oscar Benjamin wrote:

> So currently the SymPy Sage CI job points at:
> https://github.com/sagemath/sage/pull/36276

That is a PR against the sagemath/sage develop branch.

Should any new PR with doctest updates be based off of that PR and set
to merge into develop?

Yes, for now, until https://github.com/sagemath/sage/pull/36276 has been merged (hopefully next week). 

In https://github.com/sagemath/sage/pull/36338, I have developed a better mechanism, which will allows us to apply hotfixes from the Sage development side in CI workflows, including the upstream deployments of the Sage CI.

[Oscar Benjamin]

On Sun, 24 Sept 2023 at 22:10, Oscar Benjamin

I have done this now:

https://github.com/sagemath/sage/pull/36641
https://github.com/sympy/sympy/pull/25867

The SymPy CI testing with Sage all passes meaning that SymPy CI is all
green on the master branch:
https://github.com/sympy/sympy/commits/master

Thank you Matthias for fixing up the CI config after Sage's GitHub migration.

Is it worth changing the sage PR so that the Sage CI in the PR tests
with SymPy's master branch? (I don't know how to do that.)

--
Oscar

[Matthias Koeppe]

On Saturday, November 4, 2023 at 5:17:01 AM UTC-7 Oscar Benjamin wrote:

On Sun, 24 Sept 2023 at 22:10, Oscar Benjamin
<oscar.j....@gmail.com> wrote:
> So if there are doctest updates needed I guess the workflow is:
>
> - Open a PR against sagemath/sage with the needed (doctest or other) changes
> - Make SymPy's Sage CI job point at that PR
>
> Then when the time comes for Sage to update its SymPy version someone
> will incorporate the changes from that PR somehow?

I have done this now:

https://github.com/sagemath/sage/pull/36641
https://github.com/sympy/sympy/pull/25867

The SymPy CI testing with Sage all passes meaning that SymPy CI is all
green on the master branch:
https://github.com/sympy/sympy/commits/master

Great, thank you!

Is it worth changing the sage PR so that the Sage CI in the PR tests
with SymPy's master branch? (I don't know how to do that.)

I've pushed a change to your PR https://github.com/sagemath/sage/pull/36641 that will do this. When the new SymPy release is out, we can just replace this change by the package upgrade.