Sage 8.0 Debian packages for aarch64/arm64 and ppc64el

[Ximin Luo]

The Debian build for Sage 8.0 worked on ppc64el:

https://buildd.debian.org/status/package.php?p=sagemath&suite=experimental

That was a nice surprise, I was not expecting it. It failed on arm64 (aarch64) only because 11 extra test failures put it above our arbitrary "ignore < 40 test failures" threshold:

**********************************************************************
File "src/sage/combinat/partitions.pyx", line 56, in sage.combinat.partitions.number_of_partitions
Failed example:
number_of_partitions(100000)
Expected:
27493510569775696512677516320986352688173429315980054758203125984302147328114964173055050741660736621590157844774296248940493063070200461792764493033510116079342457190155718943509725312466108452006369558934464248716828789832182345009262853831404597021307130674510624419227311238999702284408609370935531629697851569569892196108480158600569421098519
Got:
27493510569775696512677516320986352688173429315980054758203125984302147328114964173055050741660736621590157844774296248940493063070200461792764493033510116079342457190155718943509725312466108452006369558934464248716828789832182345009262853831404597021307130674510624419227311238999702284408609370935531629697851569569892196108480158600569425643053
[..]
**********************************************************************
File "src/sage/combinat/partitions.pyx", line 62, in sage.combinat.partitions.number_of_partitions
Failed example:
number_of_partitions( n - (n % 385) + 369) % 385 == 0
Expected:
True
Got:
False
[.. several more of these, then ..]
**********************************************************************
File "src/sage/combinat/partitions.pyx", line 91, in sage.combinat.partitions.number_of_partitions
Failed example:
len([n for n in [1..500] if number_of_partitions(n) != Partitions(n).cardinality(algorithm='pari')])
Expected:
0
Got:
203
**********************************************************************

Full log here: https://buildd.debian.org/status/fetch.php?pkg=sagemath&arch=arm64&ver=8.0-3&stamp=1502987084&raw=0

I wonder if anyone else has experience running Sage on these platforms? Should we ship it despite these test failures, and how thoroughly are these platforms tested by the Sage team in general?

X

--
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
https://github.com/infinity0/pubkeys.git

[Dima Pasechnik]

We used to test on arm32, until the hardware we used was gone about two years ago.
I am in process of getting an arm64 box, perhaps we could start testing on it soon.

[Dima Pasechnik]

I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
with "The collector has not been ported to this machine/OS combination"
(the gc bundled with Sage is kind of old, though, it's 7.2f).

I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)

[Ximin Luo]

Dima Pasechnik:
> [..]

>
> I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
> with "The collector has not been ported to this machine/OS combination"
> (the gc bundled with Sage is kind of old, though, it's 7.2f).
>
> I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)
>

We're using gc version 7.4.2, the versions of the other packages we use are here:

https://people.debian.org/~thansen/debian-sage-status.html

[Dima Pasechnik]

On Wed, Aug 23, 2017 at 2:04 PM, Ximin Luo <infi...@debian.org> wrote:
> Dima Pasechnik:
>> [..]
>>
>> I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
>> with "The collector has not been ported to this machine/OS combination"
>> (the gc bundled with Sage is kind of old, though, it's 7.2f).
>>
>> I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)
>>
>
> We're using gc version 7.4.2, the versions of the other packages we use are here:
>
> https://people.debian.org/~thansen/debian-sage-status.html

Thanks, we'll look into this. I've opened https://trac.sagemath.org/ticket/23687
to track the progress on it.

By the way, gc version 7.4 or newer apparently needs
libatomic_ops, at least according to
http://www.hboehm.info/gc/

How does debian deal with the latter?
Thanks,
Dima

[Ximin Luo]

Dima Pasechnik:

I think it's possible to just delete/omit the libatomic_ops subdirectory and then the build will use the system version by looking through pkg-config:

https://sources.debian.net/src/libgc/1:7.4.2-8/debian/rules/#L14

libgc and libatomic-ops both have extra patches on Debian, not sure how important they are:
https://sources.debian.net/src/libatomic-ops/7.4.4-3/debian/patches/
https://sources.debian.net/src/libgc/1:7.4.2-8/debian/patches/

If you have more issues you can CC the Debian libgc maintainer Christoph Egger <chri...@debian.org> for more precise answers, I am mostly just guessing the above from the source code.

[Dima Pasechnik]

On Thursday, August 17, 2017 at 7:55:50 PM UTC+1, Ximin Luo wrote:

> The Debian build for Sage 8.0 worked on ppc64el:
>
> https://buildd.debian.org/status/package.php?p=sagemath&suite=experimental
>
> That was a nice surprise, I was not expecting it. It failed on arm64 (aarch64) only because 11 extra test failures put it above our arbitrary "ignore < 40 test failures" threshold:
>

We also saw that we need to update tachyon to 0.99b6, see
https://trac.sagemath.org/ticket/23712

It appears that Debian autotools patches have a bug that only manifests itself in a non-Debian setup. As these are meant to be upstreamed, it's probably a true Debian bug, see
https://trac.sagemath.org/ticket/23712#comment:9

Not sure how and if this should be properly reported.

Thanks,
Dima

[Jerome BENOIT]

Hello Dima, thanks for your email.

I am maintaining tachyon on behalf of the Debian Science Team.

It should not be reported because it is not a bug at all :-)
The answer is probably in debian/control :
the gnulib package must be installed properly:
it furnishes the macro

gl_LD_VERSION_SCRIPT

defined in the m4 file (on Debain Strech)

/usr/share/gnulib/m4/ld-version-script.m4

and properly added in configure.ac .

Thanks,
Jerome

>
> Thanks, Dima
>

--
Jerome BENOIT | calculus+at-rezozer^dot*net
https://qa.debian.org/developer.php?login=calc...@rezozer.net
AE28 AE15 710D FF1D 87E5 A762 3F92 19A6 7F36 C68B

[Dima Pasechnik]

Hi Jerome,
thanks for the message.

On Mon, Aug 28, 2017 at 7:28 PM, Jerome BENOIT <calc...@rezozer.net> wrote:
> I am maintaining tachyon on behalf of the Debian Science Team.
>
> On 28/08/17 15:32, Dima Pasechnik wrote:
>> On Thursday, August 17, 2017 at 7:55:50 PM UTC+1, Ximin Luo wrote:
>>> The Debian build for Sage 8.0 worked on ppc64el:
>>>
>>> https://buildd.debian.org/status/package.php?p=sagemath&suite=experimental
>>>
>>>
>>>
> That was a nice surprise, I was not expecting it. It failed on arm64 (aarch64) only because 11 extra test failures put it above our arbitrary "ignore < 40 test failures" threshold:
>>>
>>
>> We also saw that we need to update tachyon to 0.99b6, see
>> https://trac.sagemath.org/ticket/23712
>>
>> It appears that Debian autotools patches have a bug that only
>> manifests itself in a non-Debian setup. As these are meant to be
>> upstreamed, it's probably a true Debian bug, see
>> https://trac.sagemath.org/ticket/23712#comment:9
>>
>> Not sure how and if this should be properly reported.
>
> It should not be reported because it is not a bug at all :-)

My guess was that the patches named upstream-*
were meant to be a complete autotoolization of
tachyon. It turns out it`s not really the case,
right?

> The answer is probably in debian/control :
> the gnulib package must be installed properly:
> it furnishes the macro
>
> gl_LD_VERSION_SCRIPT
>
> defined in the m4 file (on Debain Strech)
>
> /usr/share/gnulib/m4/ld-version-script.m4
>
> and properly added in configure.ac .

If LD_VERSION_SCRIPT is always provided in Debian,
then, I don`t see why you need the conditional

if HAVE_LD_VERSION_SCRIPT

in src/Makefile.am at all?

If it`s not always provided then in seems you do need

AM_CONDITIONAL([HAVE_LD_VERSION_SCRIPT], ... )

in configure.am to make autoreconf happy.


Anyway, could you perhaps also explain how to build tachyon
executable with the patches in question, it seems
that it`s also only supported on Debian at the moment...

Thanks,
Dima

[Jerome BENOIT]

Hello,

On 29/08/17 00:08, Dima Pasechnik wrote:
> Hi Jerome,
> thanks for the message.
> On Mon, Aug 28, 2017 at 7:28 PM, Jerome BENOIT <calc...@rezozer.net> wrote:
>> I am maintaining tachyon on behalf of the Debian Science Team.
>>
>> On 28/08/17 15:32, Dima Pasechnik wrote:
>>> On Thursday, August 17, 2017 at 7:55:50 PM UTC+1, Ximin Luo wrote:
>>>> The Debian build for Sage 8.0 worked on ppc64el:
>>>>
>>>> https://buildd.debian.org/status/package.php?p=sagemath&suite=experimental
>>>>
>>>>
>>>>
>> That was a nice surprise, I was not expecting it. It failed on arm64 (aarch64) only because 11 extra test failures put it above our arbitrary "ignore < 40 test failures" threshold:
>>>>
>>>
>>> We also saw that we need to update tachyon to 0.99b6, see
>>> https://trac.sagemath.org/ticket/23712
>>>
>>> It appears that Debian autotools patches have a bug that only
>>> manifests itself in a non-Debian setup. As these are meant to be
>>> upstreamed, it's probably a true Debian bug, see
>>> https://trac.sagemath.org/ticket/23712#comment:9
>>>
>>> Not sure how and if this should be properly reported.
>>
>> It should not be reported because it is not a bug at all :-)
>
> My guess was that the patches named upstream-*
> were meant to be a complete autotoolization of
> tachyon. It turns out it`s not really the case,
> right?

I updated the tachyon Debian material a long time ago.
So far I can remember, the upstream package does not
follow any traditional approach. So I attempted to build
from scratch an autotools approach.
The upstream patch are essentially not Debian centric but meant
to be GNU generic and, as such, to be submitted to the mainstream author.
This is a personal approach compatible with the Debian way.

>
>> The answer is probably in debian/control :
>> the gnulib package must be installed properly:
>> it furnishes the macro
>>
>> gl_LD_VERSION_SCRIPT
>>
>> defined in the m4 file (on Debain Strech)
>>
>> /usr/share/gnulib/m4/ld-version-script.m4
>>
>> and properly added in configure.ac .
>
> If LD_VERSION_SCRIPT is always provided in Debian,
> then, I don`t see why you need the conditional
>
> if HAVE_LD_VERSION_SCRIPT
>
> in src/Makefile.am at all?

because the involved patch is not meant to be Debian centric.

The Debian centric patches have debianization for prefix.

>
> If it`s not always provided then in seems you do need
>
> AM_CONDITIONAL([HAVE_LD_VERSION_SCRIPT], ... )
>
> in configure.am to make autoreconf happy.

Once all patches are applied, there is no

AM_CONDITIONAL([HAVE_LD_VERSION_SCRIPT], ... )

line in the configure.ac file.

But there is the following line:

gl_LD_VERSION_SCRIPT

The conditional in the automake file
are meant to work with this macro;
this is the generic approach.

>
>
> Anyway, could you perhaps also explain how to build tachyon
> executable with the patches in question, it seems
> that it`s also only supported on Debian at the moment...

My role is to package tachyon for Debian and to submit/suggest patches
to the upstream maintainer: the non debian centric patches were submitted,
but my understanding is that the upstream maintainer has been overwhelmed
since he got an academic job (the lucky guy !)

So far I can remember, building with the distributed material is not easy
and not compatible with the debian helpers which expects some conventional
behaviours: using autotools is a good approach in general, and particularly here.

I would suggest to build the tachyon Debian package from the Debian source ball
on a Debian machine to see how it works. Otherwise, you can stick to the upstream
source and submit to upstream your own patches.

Jerome

[Dima Pasechnik]

On Monday, August 28, 2017 at 10:07:59 PM UTC+1, Jerome BENOIT wrote:
> On 29/08/17 00:08, Dima Pasechnik wrote:

OK, so these to be upstreamed patches either need gnulib installed, or
AM_CONDITIONAL([HAVE_LD_VERSION_SCRIPT], ... ) I mentioned above,
or some other way to deal with this issue, perhaps dropping LD_VERSION_SCRIPT
all together, right?

With the AM_CONDITIONAL() added, I am sort of able to do the rest, by
./configure --prefix=$SAGE_LOCAL --with-libflavour=serial
where the latter is needed for tachyon executables to be built, as
configure.ac says

+bin_PROGRAMS =
+man_MANS =
+if LIBFLAVOUR_IS_SERIAL
+bin_PROGRAMS += tachyon-nox tachyon-ogl
+man_MANS += tachyon-nox.1 tachyon-ogl.1
+endif

This strikes me as a bit strange, as surely the executable would work
in the multithreaded setting too, no?
Could you comment on this?

In fact, I see that on Debian the corresponding executable is linked to
libpthread, while out of Debian I only (unsurprisingly) get an executable which is not linked to libpthread, and thus, I presume, single-threaded. So there is some Debian-only building mechanism in action here...

Thanks,
Dima

[Jerome BENOIT]

Hello,

no because we are dealing with m4 files:
it is understood that the upstream maintainer had to provide
the gnulib file ld-version-script.m4 in m4 (as other *.m4 file are provided);
this is the custom with autotools.

For the Debian package, I assumed that gnulib is installed for at least two reasons:
1] the used version of ld-version-script.m4 will be the one provided by the debian package gnulib;
2] no special entry is needed in debian/copyright.

>
> With the AM_CONDITIONAL() added, I am sort of able to do the rest, by
> ./configure --prefix=$SAGE_LOCAL --with-libflavour=serial
> where the latter is needed for tachyon executables to be built, as
> configure.ac says
>
> +bin_PROGRAMS =
> +man_MANS =
> +if LIBFLAVOUR_IS_SERIAL
> +bin_PROGRAMS += tachyon-nox tachyon-ogl
> +man_MANS += tachyon-nox.1 tachyon-ogl.1
> +endif
>
> This strikes me as a bit strange, as surely the executable would work
> in the multithreaded setting too, no?
> Could you comment on this?
>
> In fact, I see that on Debian the corresponding executable is linked to
> libpthread, while out of Debian I only (unsurprisingly) get an executable which is not linked to libpthread, and thus, I presume, single-threaded. So there is some Debian-only building mechanism in action here...

Indeed. Debian distributes different variants (libraries) and flavours (commandline tools) of tachyon as explain in
<DEBIANSOURCE>/debian/README.Debian
or
/usr/share/doc/libtachyon-serial-0/README.Debian

For a quick overview:
https://packages.debian.org/search?keywords=tachyon&searchon=names&suite=stable&section=all

The `Debian-only building mechanism' follows the wish of the upstream maintainer
as express in <UPSTREAMSOURCE>/unix/*

The detail of the Debian machinery is in <DEBIANSOURCE>/debian/rules (make script): be aware:
(1) this part is meant to feed the debian packaging tools, as such it is Debian centric ;
(2) for this very Debian package, it is quite tricky.

Thanks,
Jerome

[Ximin Luo]

Ximin Luo:

Interestingly, we just got these exact same failures (with the same unexpected values) on mips64el:

https://buildd.debian.org/status/fetch.php?pkg=sagemath&arch=mips64el&ver=8.0-7&stamp=1505172408&raw=0

So in summary:

Succeeds: amd64/x86-64, i386/x86, ppc64el
Fails: arm64/aarch64, mips64el

I'm not sure what the difference is here, these are all little-endian machines. Apart from standard system libraries, partitions.so only links to:

libmpfr.so.4 => /usr/lib/x86_64-linux-gnu/libmpfr.so.4 (0x00007feb9f949000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007feb9f6c6000)

[Dima Pasechnik]

On Wednesday, August 23, 2017 at 2:04:22 PM UTC+1, Ximin Luo wrote:
> Dima Pasechnik:
> > [..]
> >
> > I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
> > with "The collector has not been ported to this machine/OS combination"
> > (the gc bundled with Sage is kind of old, though, it's 7.2f).
> >
> > I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)
> >
>
> We're using gc version 7.4.2, the versions of the other packages we use are here:
>
> https://people.debian.org/~thansen/debian-sage-status.html

Could you test for me whether in your setup with gc-7.4.2 (patched, or not) you don't hit this,
https://trac.sagemath.org/ticket/23700#comment:11

sage: from sage.libs.ecl import *
sage: sage.interfaces.maxima_lib import <TAB>

results in segfaults for all gc versions I tried on Linux (Fedora 26) which are younger than gc-7.2f.

Thanks,
Dima

[Tobias Hansen]

On 09/28/2017 01:45 PM, Dima Pasechnik wrote:
> On Wednesday, August 23, 2017 at 2:04:22 PM UTC+1, Ximin Luo wrote:
>> Dima Pasechnik:
>>> [..]
>>>
>>> I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
>>> with "The collector has not been ported to this machine/OS combination"
>>> (the gc bundled with Sage is kind of old, though, it's 7.2f).
>>>
>>> I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)
>>>
>> We're using gc version 7.4.2, the versions of the other packages we use are here:
>>
>> https://people.debian.org/~thansen/debian-sage-status.html
> Could you test for me whether in your setup with gc-7.4.2 (patched, or not) you don't hit this,
> https://trac.sagemath.org/ticket/23700#comment:11
>
> sage: from sage.libs.ecl import *
> sage: sage.interfaces.maxima_lib import <TAB>
>
> results in segfaults for all gc versions I tried on Linux (Fedora 26) which are younger than gc-7.2f.
>
> Thanks,
> Dima

Hi,

doesn't crash for me on Debian unstable with sagemath 8.0-8 and libgc1c2
7.4.2-8.

To see the patches applied to gc in that version you can look into this
file:
http://http.debian.net/debian/pool/main/libg/libgc/libgc_7.4.2-8.debian.tar.xz

Best,
Tobias

[Dima Pasechnik]

On Thursday, September 28, 2017 at 2:13:48 PM UTC+1, Tobias Hansen wrote:
> On 09/28/2017 01:45 PM, Dima Pasechnik wrote:
> > On Wednesday, August 23, 2017 at 2:04:22 PM UTC+1, Ximin Luo wrote:
> >> Dima Pasechnik:
> >>> [..]
> >>>
> >>> I wonder how your arm64/aarch64 build worked, as for me in fails in building gc,
> >>> with "The collector has not been ported to this machine/OS combination"
> >>> (the gc bundled with Sage is kind of old, though, it's 7.2f).
> >>>
> >>> I tried building Sage 8.1.beta3 on Pinebook (https://www.pine64.org/?page_id=3707)
> >>>
> >> We're using gc version 7.4.2, the versions of the other packages we use are here:
> >>
> >> https://people.debian.org/~thansen/debian-sage-status.html
> > Could you test for me whether in your setup with gc-7.4.2 (patched, or not) you don't hit this,
> > https://trac.sagemath.org/ticket/23700#comment:11
> >
> > sage: from sage.libs.ecl import *
> > sage: sage.interfaces.maxima_lib import <TAB>
> >
> > results in segfaults for all gc versions I tried on Linux (Fedora 26) which are younger than gc-7.2f.
> >
> > Thanks,
> > Dima
>
> Hi,
>
> doesn't crash for me on Debian unstable with sagemath 8.0-8 and libgc1c2
> 7.4.2-8.
>

how about https://trac.sagemath.org/ticket/22766 ?
(note that to trigger this you might need to delete ~/.sage/maxima_commandlist_cache.sobj
- or whatever the Debian equavalent of it is)

> To see the patches applied to gc in that version you can look into this
> file:
> http://http.debian.net/debian/pool/main/libg/libgc/libgc_7.4.2-8.debian.tar.xz

I guess you are doing something to IPython - namely, disabling running tab completion in a separate thread, no?

Thanks,
Dima

[Tobias Hansen]

That gives a segmentation fault.

>> To see the patches applied to gc in that version you can look into this
>> file:
>> http://http.debian.net/debian/pool/main/libg/libgc/libgc_7.4.2-8.debian.tar.xz
> I guess you are doing something to IPython - namely, disabling running tab completion in a separate thread, no?

Ipython doesn't seem to have a patch that does this and I don't know how you would disable this.

Best,
Tobias