Is there a way to force Sage to use system's openblas ?

[Emmanuel Charpentier]

Dear list,

I started to get strange errors when installing some R packages ("sp" and "spdep", for the record). Ultimately, it boils down to the fact that those libraries need to use library system calls that exist in the system's openblas (these packages do install fine in system's R) but not in "our" openblas.

A possible workaround would be to install Atlas systemwide and to point SAGE_ATLAS_LIB to that. But  using ATLAS has some drawbacks (e. g. necessity to manually recompile in order to optimize BLAS and LAPACK). Hence my question :

Is there a way to install (i. e. compile) Sage using system's BLAS and LAPACK provided by system's openblas ?

PS : another workaround would be ti upgrade "our" openblas to a version supporting the needed calls. I do not know how to do this, and it sems soewhat intricate.

[François Bissey]

Can you send us the error message you get when installing sp/spdep?
What version of openblas do you have on your system? The latest beta
of sage uses openblas 0.3.5 which is latest available release.

François

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To post to this group, send email to sage-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sage-devel.
> For more options, visit https://groups.google.com/d/optout.

[Emmanuel Charpentier]

Ahem...

It turns put that I can't reproduce my *origonal* problem : sustem's upgrade seems to ghave fixed rthe problem.

Sorry for the noise...

However, "spdep" still doesn't install, but the problem si totally different : its dependency "sf" doesn't compile due to a missing library call in ... libgdal.so, of all things, which claims :

/usr/bin/ld: /usr/lib/libgdal.so: undefined reference to `sqlite3_column_origin_name'

/usr/bin/ld: /usr/lib/libgdal.so: undefined reference to `sqlite3_column_table_name'

collect2: error: ld returned 1 exit status

/usr/bin/ld: /usr/lib/libgdal.so: undefined reference to `sqlite3_column_origin_name'

/usr/bin/ld: /usr/lib/libgdal.so: undefined reference to `sqlite3_column_table_name'

collect2: error: ld returned 1 exit status

This is more explainable: "our" sqlite3 is at 3.22, whereas system's sqlite3 is at 3.26+fossilbc891ac6b (whatever that means...).

I might try to update "our" sqlite3 *if* I can figure what is the "right" source.... Debian unstable has 3.27 ; I might try that.

Hints ? Suggestions ?

[François Bissey]

Good diagnostics there. I’d recommend
https://sqlite.org/2019/sqlite-autoconf-3270100.tar.gz
as the sources to use. Pushing to use system sqlite is
also an healthy option.

François

[Emmanuel Charpentier]

Le samedi 16 février 2019 10:20:51 UTC+1, François Bissey a écrit :

Good diagnostics there. I’d recommend
https://sqlite.org/2019/sqlite-autoconf-3270100.tar.gz
as the sources to use.

Thanks ! I'll try that

 

Pushing to use system sqlite

What do you mean ? Do you mean removing "our" sqlite and dependig on system's sqlite ? Nice idea... except for Cygwin,and/or WSL...

[François Bissey]

> On 16/02/2019, at 22:25, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:
>
> Pushing to use system sqlite
>
> What do you mean ? Do you mean removing "our" sqlite and dependig on system's sqlite ? Nice idea... except for Cygwin,and/or WSL...
>

Erik has been pushing that for a number of package. In a configurable way.
For example
https://github.com/sagemath/sage/blob/develop/build/pkgs/bzip2/spkg-configure.m4

sqlite would make as much sense to take from the system as something like
bzip2.

François

[Emmanuel Charpentier]

Dear François

Le samedi 16 février 2019 10:30:14 UTC+1, François Bissey a écrit :

> On 16/02/2019, at 22:25, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:
>
> Pushing to use system sqlite
>
> What do you mean ? Do you mean removing "our" sqlite and dependig on system's sqlite ? Nice idea... except for Cygwin,and/or WSL...
>  

Erik has been pushing that for a number of package. In a configurable way.
For example
https://github.com/sagemath/sage/blob/develop/build/pkgs/bzip2/spkg-configure.m4

I see (modulo the legendary readability of m4, which has been compared to line noise or a cat's nap on the keyboard...).

And, ndeed, it should also apply to a lot of "our" libraries, and possibly "our" interpreters (ideally, we should be able to reduce Sage-the-distribution to Sage-the-library...).

But this is a secondary goal. Some systems may need "our" sqlite (e. g. when the system's sqlite is too old and can't be upgraded for some reason...).

So I'll propose a patch to upgrade *our* sqlite, and, possibly, another one to render "our" version optional.

Comments ? 

[François Bissey]

> On 16/02/2019, at 22:40, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:
>
> I see (modulo the legendary readability of m4, which has been compared to line noise or a cat's nap on the keyboard...).
>

Any language is a bit like that when you don’t know it to be fair.
And you can produce line noise in most programming languages
but I’d pick perl and lisp over m4 for that.

Do open a ticket for sqlite upgrade, there may be other interesting
reasons to upgrade.

There may already be a ticket for making sqlite optional and if
you think m4 is looks like line noise you probably should leave
it to the pros.

[Emmanuel Charpentier]

Le samedi 16 février 2019 10:47:05 UTC+1, François Bissey a écrit :

> On 16/02/2019, at 22:40, Emmanuel Charpentier <emanuel.c...@gmail.com> wrote:
>
> I see (modulo the legendary readability of m4, which has been compared to line noise or a cat's nap on the keyboard...).
>

Any language is a bit like that when you don’t know it to be fair.
And you can produce line noise in most programming languages
but I’d pick perl and lisp over m4 for that.

Regexps as used from R aren't bad, either... ;-).

 

Do open a ticket for sqlite upgrade, there may be other interesting
reasons to upgrade.

Done. That builds without a hitch. make ptestlong underway (it will be a while : make started by recompiling both pythons. I think that this will recompile almost anything...)

 

There may already be a ticket for making sqlite optional and if
you think m4 is looks like line noise you probably should leave
it to the pros.

Or *really* learn m4... which is a bit orthogonal to my initial preoccupations (evidence-based medicine and dentistry).

 

[Emmanuel Charpentier]

Trac#27303 needs review.

[Dima Pasechnik]

Our plan is to enable the use of the system R install in Sage - for this we need to understand whether there would be any problems for rpy2 if R used a different, from the one in Sage, Blas library.

--

[Emmanuel Charpentier]

Le dimanche 17 février 2019 09:34:10 UTC+1, Dima Pasechnik a écrit :

Our plan is to enable the use of the system R install in Sage - for this we need to understand whether there would be any problems for rpy2 if R used a different, from the one in Sage, Blas library.

Quite probably : In order to use the rjags package (interface to the JAGS Bayesian modelling sampler, I had to use the Sage shell in order to compile and install a version of JAGS in $SAGE_LOCAL : the system JAGS choked on BLAS/LAPACK calls not available in Sage's libraries.

Things would be much simpler if Sage used the systemwide numeric libraries. I *think* that this is possible when these libraries are ATLAS : pointing SAGE_ATLAS_LIB to the systemwide ATLAS library makes Sage use them. ISTR that Erik Madison Bray recommended this to compile Sage on Cygwin (but I lost the reference to the relevant Sage Wiki post...). 

My question was to know if this is possible when openblas provides the systemwide numerical libraries...

[Dima Pasechnik]

On Sun, 17 Feb 2019 12:48 Emmanuel Charpentier <emanuel.c...@gmail.com wrote:

Le dimanche 17 février 2019 09:34:10 UTC+1, Dima Pasechnik a écrit :

Our plan is to enable the use of the system R install in Sage - for this we need to understand whether there would be any problems for rpy2 if R used a different, from the one in Sage, Blas library.

Quite probably : In order to use the rjags package (interface to the JAGS Bayesian modelling sampler, I had to use the Sage shell in order to compile and install a version of JAGS in $SAGE_LOCAL : the system JAGS choked on BLAS/LAPACK calls not available in Sage's libraries.

Things would be much simpler if Sage used the systemwide numeric libraries. I *think* that this is possible when these libraries are ATLAS : pointing SAGE_ATLAS_LIB to the systemwide ATLAS library makes Sage use them. ISTR that Erik Madison Bray recommended this to compile Sage on Cygwin (but I lost the reference to the relevant Sage Wiki post...). 

My question was to know if this is possible when openblas provides the systemwide numerical libraries...

Not at the moment, but there are plans to make it possible.

One should do more spkg-configure.m4 along the lines of #27212, where this is done for gmp and mpir.

[Timo Kaufmann]

Am Sonntag, 17. Februar 2019 12:48:52 UTC+1 schrieb Emmanuel Charpentier:

Le dimanche 17 février 2019 09:34:10 UTC+1, Dima Pasechnik a écrit :

Our plan is to enable the use of the system R install in Sage - for this we need to understand whether there would be any problems for rpy2 if R used a different, from the one in Sage, Blas library.

Quite probably : In order to use the rjags package (interface to the JAGS Bayesian modelling sampler, I had to use the Sage shell in order to compile and install a version of JAGS in $SAGE_LOCAL : the system JAGS choked on BLAS/LAPACK calls not available in Sage's libraries.

Things would be much simpler if Sage used the systemwide numeric libraries. I *think* that this is possible when these libraries are ATLAS : pointing SAGE_ATLAS_LIB to the systemwide ATLAS library makes Sage use them. ISTR that Erik Madison Bray recommended this to compile Sage on Cygwin (but I lost the reference to the relevant Sage Wiki post...). 

My question was to know if this is possible when openblas provides the systemwide numerical libraries...

Setting `LD_DEBUG=libs` shows you which libraries are used at runtime. `LD_PRELOAD` can be useful to force the use of a particular blas implementation.

[Samuel Lelievre]

Sun 2019-02-17 12:48:52 UTC+1, Emmanuel Charpentier:

>
> Things would be much simpler if Sage used the systemwide numeric

> libraries. I *think* that this is possible when these libraries are ATLAS:

> pointing SAGE_ATLAS_LIB to the systemwide ATLAS library makes

> Sage use them. ISTR that Erik Madison Bray recommended this to

> compile Sage on Cygwin (but I lost the reference to the relevant

> Sage Wiki post...). 

You might be referring to this page on the Sage Trac Wiki:

    https://trac.sagemath.org/wiki/Cygwin64Port#BuildingSage

[Emmanuel Charpentier]

Indeed.

Thanks for coming to the rescue of my aging neurons...

[Emmanuel Charpentier]

A couple of remarks :

Le dimanche 17 février 2019 13:17:12 UTC+1, Dima Pasechnik a écrit :

On Sun, 17 Feb 2019 12:48 Emmanuel Charpentier <emanuel.c...@gmail.com wrote:

Le dimanche 17 février 2019 09:34:10 UTC+1, Dima Pasechnik a écrit :

Our plan is to enable the use of the system R install in Sage - for this we need to understand whether there would be any problems for rpy2 if R used a different, from the one in Sage, Blas library.

We should be cautious with this one :

0.8.2 is our current reference because, AFAICT, it is the last version of rpy2 that supports Python2. If/when our plans to switch to Python3 succed (at last !), we should explore its newer versions. And switch to them if practical. But their requirements for numerics may vary...

Quite probably : In order to use the rjags package (interface to the JAGS Bayesian modelling sampler, I had to use the Sage shell in order to compile and install a version of JAGS in $SAGE_LOCAL : the system JAGS choked on BLAS/LAPACK calls not available in Sage's libraries.

Things would be much simpler if Sage used the systemwide numeric libraries. I *think* that this is possible when these libraries are ATLAS : pointing SAGE_ATLAS_LIB to the systemwide ATLAS library makes Sage use them. ISTR that Erik Madison Bray recommended this to compile Sage on Cygwin (but I lost the reference to the relevant Sage Wiki post...). 

My question was to know if this is possible when openblas provides the systemwide numerical libraries...

Not at the moment, but there are plans to make it possible.

One should do more spkg-configure.m4 along the lines of #27212, where this is done for gmp and mpir.

Is that possible for all of the current packages in Sage-the distribution ? Depending on one set (or possibly a coice of sets) of numerical libraries would allow us to get rid of them in Sage-the-distribution, which should lighten it considerably...

[E. Madison Bray]

On Sun, Feb 17, 2019 at 1:50 PM Emmanuel Charpentier
<emanuel.c...@gmail.com> wrote:
> Is that possible for all of the current packages in Sage-the distribution ? Depending on one set (or possibly a coice of sets) of numerical libraries would allow us to get rid of them in Sage-the-distribution, which should lighten it considerably...

That is the plan, but "all" will be an uphill battle, just in terms of
sheer numbers, though that is why we are focusing primarily on
low-level and/or hard to build and/or common system-level packages
first.

Of course, one thing sage-the-distribution does well is ensuring a
compatible cross-section of versions for all of Sage's dependencies,
which is not easy to do, and in the past has not been done on common
Linux distributions (but significant progress has been made there).
The downside to that of course is that Sage ends up only being built
and tested on a very narrow, fine-tuned set of dependency versions.
This makes it easier to build/run consistently, but also makes it more
fragile to run on more arbitrary configurations (and I would argue
more fragile in general).

For that reason I think sage-the-distribution will continue to serve a
purpose (even if not in the same form it's in currently, such as if we
switch over to just using conda). But for a *lot* of the
dependencies, especially on reasonably modern OS distributions, it is
quite possible to get those dependencies from the system and we just
don't. Fixing that's something I've been working toward for
practically as long as I've been working on the project, and we've
been making real progress lately, which you are encouraged to join in
and help with...

[Thierry]

Hi,

On Mon, Feb 18, 2019 at 04:07:07PM +0100, E. Madison Bray wrote:
[...]

> Fixing that's something I've been working toward for practically as
> long as I've been working on the project, and we've been making real
> progress lately, which you are encouraged to join in and help with...

Perhaps could you provide a task ticket, plus some pointers to readable
documentation, howtos, etc.

Ciao,
Thierry

[Dima Pasechnik]

Trac keyword spkg-configure gives a list of relevant tickets.

[Samuel Lelièvre]

Tue 2019-02-19 18:13 UTC, Dima:

>
> Trac keyword spkg-configure gives a list of relevant tickets.

Direct link:
https://trac.sagemath.org/query?order=id&desc=1&keywords=~spkg-configure

[E. Madison Bray]

I have several task tickets for several aspects of this, but a lot of
it is a bit disorganized just because there's so much to do. Some
major ones are:

* For upgrading the installation method for all packages to use
DESTDIR: https://trac.sagemath.org/ticket/24024

* There are still some tickets linked to from there that have been
awaiting review for a long time, including
https://trac.sagemath.org/ticket/25035 and
https://trac.sagemath.org/ticket/25049

* Many packages should still have an spkg-legacy-uninstall:
https://trac.sagemath.org/ticket/25140

* There is a broader task ticket about getting a `make install`
working for Sage. Some disagree with this approach entirely, and I
agree there are some arguments for/against it, though I think if
nothing else it motivates many changes that are worth making for their
own sake: https://trac.sagemath.org/ticket/21495

* Some have liked to the spkg-configure keyword on Trac, but there is
not to my knowledge an umbrella task ticket for adding support for
using more packages from the system (implemented by providing
spkg-configure.m4 files for those packages). I think that would be a
good idea, especially for prioritizing that work. I'll work on making
such a ticket now.

[Samuel Lelievre]

Wed 2019-02-20 17:49:57 UTC+1, E. Madison Bray:
>
> * Some have linked to the spkg-configure keyword on Trac, but there is

> not to my knowledge an umbrella task ticket for adding support for
> using more packages from the system (implemented by providing
> spkg-configure.m4 files for those packages). I think that would be a
> good idea, especially for prioritizing that work. I'll work on making
> such a ticket now.

I see you made one now; link for reference:

- Sage Trac ticket 27330

  Try to use as many system packages as possible
  https://trac.sagemath.org/ticket/27330

Thanks!

[E. Madison Bray]

Yes, I had meant to follow up here, but I was sort of in a rush when I
was putting that together. Thank you for mentioning it here.