missing docker images of sagemath

[Sébastien Labbé]

Bonjour,

Thanks to Vincent, I learned recently how to use continuous integration in gitlab and the docker images of previous versions of sage posted https://hub.docker.com/r/sagemath/sagemath-dev/ in order to test that my optional package works with all those versions. It basically made me realize that my package was broken on most versions of sage except the most recent. And hopefully soon, I will be able to fix those issues. So thanks to those involved in having these various docker images available.

For the curious, the most recent run is available here:

https://gitlab.com/seblabbe/slabbe/-/pipelines/202932270

it shows that my package works for 9.1.rc5 but not 9.0 or earlier versions (I need to work on it).

Unfortunately, it seems the docker images of various versions of sage are not uploaded since 4 months. Maybe it was too much energy to upload all beta versions there. In fact, I don't really need all beta in there. What I think is important to have are the final releases. With that respect, the following are missing:

8.6: ok

8.7: ok

8.8: ok

8.9 is missing or is broken

9.0 : ok

9.1 is missing (so I am using 9.1.rc5 and 9.1.rc5-py3)

No 9.2.betaX are there, but I would be happy that 9.2 be there when released.

Thanks to the people involved.

Sincerely,

Sébastien

[Frédéric Chapoton]

It seems that all pipelines are failing since 9.2.beta4. Could someone please have a look ?

https://gitlab.com/sagemath/sage/-/pipelines

[Frédéric Chapoton]

I can see recent tentatives from Orsay, but they failed. Maybe Erik did launch a runner there ?

Our .ci setting needs to be adapted to use python3 by default. And there is a complaint about

.ci/build-docker.sh: line 60: syntax error: unterminated quoted string

but the script looks ok to me.

[Frédéric Chapoton]

I have made https://trac.sagemath.org/ticket/30820 for some small fixes.

[Frédéric Chapoton]

Build failure comes from not being able to pull something :

+ docker pull source-clean

Error response from daemon: pull access denied for source-clean, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

Le vendredi 23 octobre 2020 à 20:42:26 UTC+2, Frédéric Chapoton a écrit :

[Frédéric Chapoton]

Could some please review https://trac.sagemath.org/ticket/30820 ?

Maybe also relevant : https://trac.sagemath.org/ticket/30736 (already positive review)

[Sébastien Labbé]

At

https://hub.docker.com/r/sagemath/sagemath-dev/tags

tags are still 4 months old.

I notice that #30736 is in 9.3.beta0 but not #30820 yet. This should be fixed automatically if #30820 is in the next beta?

[Frédéric Chapoton]

The 9.3.beta1 build on gitlab is blocked by a failure with "planarity"

see 
https://gitlab.com/sagemath/dev/trac/-/pipelines/213154052

[Frédéric Chapoton]

artifact complains about our patches not applying:

[planarity-3.0.0.5.p0] patching file c/planarity.h

[planarity-3.0.0.5.p0] Hunk #1 FAILED at 77 (different line endings).

[planarity-3.0.0.5.p0] Hunk #2 FAILED at 88 (different line endings).

[planarity-3.0.0.5.p0] 2 out of 2 hunks FAILED -- saving rejects to file c/planarity.h.rej

[givaro-4.1.1] Making all in gmp++

[planarity-3.0.0.5.p0] Error applying '../patches/extern.patch'

[Frédéric Chapoton]

and this seems to be due to automatic replacement of CRLF by LF 

warning: CRLF will be replaced by LF in build/pkgs/planarity/patches/extern.patch.

The file will have its original line endings in your working directory.

[Frédéric Chapoton]

A .gitattributes file was introduced in https://trac.sagemath.org/ticket/29733 that enforces this CRLF to LF conversion

[Frédéric Chapoton]

the build for 9.3.beta3 has passed successfully, so there should be a docker for that, and a docker for develop that should point to the same.

https://gitlab.com/sagemath/sage/-/pipelines

Frédéric

Le jeudi 15 octobre 2020 à 12:09:15 UTC+2, Sébastien Labbé a écrit :

[Eric Gourgoulhon]

Le mardi 8 décembre 2020 à 09:33:03 UTC+1, Frédéric Chapoton a écrit :

the build for 9.3.beta3 has passed successfully, so there should be a docker for that, and a docker for develop that should point to the same.

Cool, thank you Frédéric!

On the page

https://hub.docker.com/r/sagemath/sagemath/tags?page=1&ordering=last_updated

we see the Docker images develop and develop-py3, as well as 9.3.beta3 and 9.3.beta3-py3. I guess this is a duplicate and that there should no longer be any stuff marked '*-py3' now...

Eric.

[Dima Pasechnik]

it's from .ci/push-dockerhub.sh :

# For historical reasons, we also provide a -py3 tag. It's identical
to the non-py3 tag.

docker tag ${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG
${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG-py3

docker push ${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG-py3

Can these lines just be removed?

I don't see any "historical reason" here...


>
> Eric.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bc7a95c5-c6a7-47ea-8241-ad7f99ff78afn%40googlegroups.com.

[Dima Pasechnik]

On Tue, Dec 8, 2020 at 8:33 AM Frédéric Chapoton <fchap...@gmail.com> wrote:
>
> the build for 9.3.beta3 has passed successfully, so there should be a docker for that, and a docker for develop that should point to the same.
>
> https://gitlab.com/sagemath/sage/-/pipelines
>

Can we get this documented in the Developer manual please, with
few minimal commands, sufficient for pulling the image and
running Sage in the container?

> Frédéric
>
> Le jeudi 15 octobre 2020 à 12:09:15 UTC+2, Sébastien Labbé a écrit :
>>
>> Bonjour,
>>
>> Thanks to Vincent, I learned recently how to use continuous integration in gitlab and the docker images of previous versions of sage posted https://hub.docker.com/r/sagemath/sagemath-dev/ in order to test that my optional package works with all those versions. It basically made me realize that my package was broken on most versions of sage except the most recent. And hopefully soon, I will be able to fix those issues. So thanks to those involved in having these various docker images available.
>>
>> For the curious, the most recent run is available here:
>> https://gitlab.com/seblabbe/slabbe/-/pipelines/202932270
>> it shows that my package works for 9.1.rc5 but not 9.0 or earlier versions (I need to work on it).
>>
>> Unfortunately, it seems the docker images of various versions of sage are not uploaded since 4 months. Maybe it was too much energy to upload all beta versions there. In fact, I don't really need all beta in there. What I think is important to have are the final releases. With that respect, the following are missing:
>>
>> 8.6: ok
>> 8.7: ok
>> 8.8: ok
>> 8.9 is missing or is broken
>> 9.0 : ok
>> 9.1 is missing (so I am using 9.1.rc5 and 9.1.rc5-py3)
>>
>> No 9.2.betaX are there, but I would be happy that 9.2 be there when released.
>>
>> Thanks to the people involved.
>>
>> Sincerely,
>>
>> Sébastien
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/8196c783-fbfd-475f-8a29-2b7b13da6325n%40googlegroups.com.

[Matthias Koeppe]

On Tuesday, December 8, 2020 at 8:16:53 AM UTC-8 Dima Pasechnik wrote:

On Tue, Dec 8, 2020 at 3:56 PM Eric Gourgoulhon <egourg...@gmail.com> wrote:
> On the page
> https://hub.docker.com/r/sagemath/sagemath/tags?page=1&ordering=last_updated
> we see the Docker images develop and develop-py3, as well as 9.3.beta3 and 9.3.beta3-py3. I guess this is a duplicate and that there should no longer be any stuff marked '*-py3' now...

it's from .ci/push-dockerhub.sh :

# For historical reasons, we also provide a -py3 tag. It's identical
to the non-py3 tag.

docker tag ${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG
${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG-py3

docker push ${DOCKER_NAMESPACE:-sagemath}/$1:$DOCKER_TAG-py3

Can these lines just be removed?

I don't see any "historical reason" here...

User packages' CI scripts may be referring to this tag.

 

[Sébastien Labbé]

On Tuesday, December 8, 2020 at 6:17:52 PM UTC+1 Matthias Koeppe wrote:

User packages' CI scripts may be referring to this tag.

I would remove the `-py3` suffix or otherwise, we think that the `sage9.3.beta3` tag is not using Python 3.

[Dima Pasechnik]

fine, but this does not need several "docker push", as we are tagging
the same image, right?

docker image push --all-tags

should do all this in one go, no?

>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/010a3525-a4b5-401e-8cc6-09b19586d232n%40googlegroups.com.

[seb....@gmail.com]

It seems that there is still (or again) something broken. For the images of the beta releases since the 8th of December Sage crashes if I try to start it:

Sage build/upgrade complete!
make[1]: Leaving directory '/home/sage/sage'
sage@8711c66abafe:~/sage$ ./sage
┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 9.3.beta5, Release Date: 2020-12-27               │
│ Using Python 3.8.5. Type "help()" for help.                        │
└────────────────────────────────────────────────────────────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Warning: this is a prerelease version, and it may be unstable.     ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
------------------------------------------------------------------------
/home/sage/sage/local/lib/python3.8/site-packages/cysignals/signals.cpython-38-x86_64-linux-gnu.so(+0x6b28)[0x7ff042b33b28]
/home/sage/sage/local/lib/python3.8/site-packages/cysignals/signals.cpython-38-x86_64-linux-gnu.so(+0x6d08)[0x7ff042b33d08]
/home/sage/sage/local/lib/python3.8/site-packages/cysignals/signals.cpython-38-x86_64-linux-gnu.so(+0x93a0)[0x7ff042b363a0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7ff0523f9390]
/home/sage/sage/local/lib/libgmp.so.10(__gmpz_sizeinbase+0x47)[0x7ff049df0597]
/home/sage/sage/local/lib/python3.8/site-packages/sage/rings/rational.cpython-38-x86_64-linux-gnu.so(+0x1f78c)[0x7ff036e2978c]
/home/sage/sage/local/lib/python3.8/site-packages/sage/rings/rational.cpython-38-x86_64-linux-gnu.so(+0x370d5)[0x7ff036e410d5]
............
/home/sage/sage/local/lib/libpython3.8.so.1.0(+0x185260)[0x7ff05278a260]
/home/sage/sage/local/lib/libpython3.8.so.1.0(PyRun_FileExFlags+0x95)[0x7ff05278cb45]
/home/sage/sage/local/lib/libpython3.8.so.1.0(PyRun_SimpleFileExFlags+0xf6)[0x7ff05278ccb6]
/home/sage/sage/local/lib/libpython3.8.so.1.0(Py_RunMain+0x748)[0x7ff0527a7398]
/home/sage/sage/local/lib/libpython3.8.so.1.0(Py_BytesMain+0x39)[0x7ff0527a77c9]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff05203e840]
python3(_start+0x29)[0x400709]
------------------------------------------------------------------------
Attaching gdb to process id 9075.
Cannot find gdb installed
GDB is not installed.
Install gdb for enhanced tracebacks.
------------------------------------------------------------------------
Unhandled SIGILL: An illegal instruction occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------
Illegal instruction (core dumped)

Furthermore, all pipelines on individual branches since that date failed.

​

[Dima Pasechnik]

I don't know anything about docker images, but "illegal instruction" is usually an indication that the binary was made for a CPU that accepts more commands than the one you use.

--

You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/1cb3c7fb-f98b-40e9-90c3-c1961235be43n%40googlegroups.com.

[Frédéric Chapoton]

did you run

docker run -it sagemath/sagemath-dev:develop

or something else ?

[Sebastian Oehms]

Thanks to you both for the hints! Yes, instead of develop I used the
version tag directly. Running docker as you suggested solves the
problem. But, the version tag works on a newer CPU (the one causing
the crash was an i5 of the 2. generation), as well, and for 9.1.beta1
even on the older CPU.

What about my second point concerning the trac ticket branches, I mean this?

> --
> You received this message because you are subscribed to a topic in the Google Groups "sage-devel" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/sage-devel/Dbd4nFi8R7I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/f00269b2-ce0d-4724-99b5-d8e97054679bn%40googlegroups.com.

[Sebastian Oehms]

https://gitlab.com/sagemath/dev/trac/-/pipelines

[Dima Pasechnik]

On Mon, Dec 28, 2020 at 10:05 PM Sebastian Oehms <seb....@gmail.com> wrote:
>
> https://gitlab.com/sagemath/dev/trac/-/pipelines


so docker build does some patching (why? I don't know) and there are
failures (why? I don't know),
and then an attempt to do something with these rejected patches fails:

cf. e.g. https://gitlab.com/sagemath/dev/trac/-/jobs/935852462

patching file src/sage/combinat/crystals/star_crystal.py
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file
src/sage/combinat/crystals/star_crystal.py.rej
patching file src/sage/ext_data/doctest/rich_output/example.png
patching file src/sage/ext_data/notebook-ipython/logo-64x64.png
patching file src/sage/misc/element_with_label.py
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file
src/sage/misc/element_with_label.py.rej
patching file src/sage/rings/invariants/__init__.py
Reversed (or previously applied) patch detected! Assume -R? [n]
Apply anyway? [n]
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file
src/sage/rings/invariants/__init__.py.rej
The command '/bin/sh -c patch -p1 < "$HOME"/sage-context.patch'
returned a non-zero code: 1
Command exited with non-zero status 1

I guess it's because it's a new codepath (with these rejected patches) in
docker/Dockerfile

Over to Julian, I think.

> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAPEyD1D-p5x2MuB0K-7-Q88cAt0hzXYbovnrHh-tjH2_BVUtAA%40mail.gmail.com.

[julian...@fsfe.org]

Hi Frédéric,

docker pull source-clean is expected to fail. We should probably add a message there that explains that. It says `docker pul ... || true` in the script there.

[julian...@fsfe.org]

Hi Dima,

the patching that fails is only meant to make the docker image work in a local developer setup where you have your sage source tree (with changes and untracked files) mounted into the docker container. In the CI context the line that reads `git status --porcelain` in the Dockerfile should not return anything but it finds lots of unstaged changes. I guess that's something to do with some line endings.

I'll try to look into this tomorrow.

julian