spkgs assumed for doctesting but not default installed

[Johan S. H. Rosenkilde]

Hi sage-devel

Today I made a fresh clone of the develop branch, ran "make", and then
tried to run "./sage -t". To my surprise, this didn't work at all!

First off, the doctest framework wouldn't even run, since sympy wasn't
installed. After doing "./sage -pip install sympy", I could now actually
perform doctests.

But I then get thousands of failures all around Sage's library, due to
other uninstalled spkgs: jmol, palp, Cremona's mini database, and
others.

Am I doing something wrong? Or do we currently have a problem with
default spkgs not being installed as they should?

Best,
Johan

[Volker Braun]

sounds like the first "make" failed...

[Johan S. H. Rosenkilde]

> sounds like the first "make" failed...

That's not clear to me. The compile of Sage itself seemed to work but
then make failed during doc-building. This has happened so often during
the years I've worked with Sage, however, that I've learned to ignore
it. Inspecting the error now, I see that this time it has to do with the
missing jmol-package.

Inspecting install.log, I it seems that everything before seems to have
compiled. jmol, sympy, etc. are never mentioned once in the install.log.

A disturbing thing though is that the following line appears a lot:

ar: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/lib/libLLVM-3.8.so)

as well as:

ranlib: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/lib/libLLVM-3.8.so)

However, the packages being compiled report that everything went alright
afterwards. I don't know what the errors mean. I have the Arch linux package
gcc-libs 5.3.0-5 installed.

The install.log is huge and I don't know what might be relevant. I've
put the whole thing here:

http://atuin.dk/install.log.zip

Best,
Johan

[Volker Braun]

On Thursday, June 30, 2016 at 2:26:47 PM UTC+2, Johan S. R. Nielsen wrote:

ar: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required by /usr/lib/libLLVM-3.8.so) 

Are you using llvm? As far as I know that doesnt' work, though I haven't tried it myself. 

[Johan S. H. Rosenkilde]

>> ar: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.22' not found (required
>> by /usr/lib/libLLVM-3.8.so)
>
>
> Are you using llvm? As far as I know that doesnt' work, though I haven't
> tried it myself.

I was baffled about that too. I haven't consciously done anything that
would use llvm. I'm using a fully updated Arch linux, but with gcc
rolled back to 5.3. Then I just cloned the git repo and did 'make'. I
don't seem have any environment variables pointing to llvm either.

Best,
Johan

[leif]

Well, it's apparently just that your toolchain isn't using binutils
consistently; usually LLVM's tools get installed as 'llvm-ar' and
'llvm-ranlib', or they don't get installed into the default PATH.

You may try to reinstall binutils or create appropriate symlinks along
PATH (such that 'ar' and 'ranlib' originate from them), but I have no
idea whether that's at all related to your main problem.


If 'ar' etc. fail but that error isn't catched that's of course a bug.


-leif

P.S.: Downgrading GCC of course (in this case at least) also downgrades
libstdc++ (to 6.0.21), so programs and libraries built against the newer
one are borked. If the newer libstdc++ didn't get removed, it's
probably just a matter of symlinking libstdc++.so.6 back to the more
recent one, presumably libstdc++.so.6.0.22 (from GCC 6.1).

[Johan S. H. Rosenkilde]

Hi leif,

> Well, it's apparently just that your toolchain isn't using binutils
> consistently; usually LLVM's tools get installed as 'llvm-ar' and
> 'llvm-ranlib', or they don't get installed into the default PATH.

I don't have llvm-ar or llvm-ranlib installed AFAIK (they're in the
package llvm which I don't have. I have llvm-libs, lib32-llvm-libs and
clang, though. Also locate shows up nothing.).

> You may try to reinstall binutils or create appropriate symlinks along
> PATH (such that 'ar' and 'ranlib' originate from them), but I have no
> idea whether that's at all related to your main problem.

"pacman -Qo `which ar`" tells me binutils already owns the main ar (and
ranlib).

> P.S.: Downgrading GCC of course (in this case at least) also downgrades
> libstdc++ (to 6.0.21), so programs and libraries built against the newer
> one are borked. If the newer libstdc++ didn't get removed, it's
> probably just a matter of symlinking libstdc++.so.6 back to the more
> recent one, presumably libstdc++.so.6.0.22 (from GCC 6.1).

Oh yeah, that probably explains why llvm expects a GLIBCXX which is not
in my current /usr/lib/libstdc++.so.6. I've downgraded llvm and trying a
"make distclean && make" again.

Best,
Johan

[Johan S. R. Nielsen]

Hi,

I've now recompiled Sage after rolling back llvm-libs to 3.7.1-1 (was 3.8.0.1). That fixed the GLIBCXX errors spewing out of ar and ranlib, but it didn't fix the problem: my newly compiled version of Sage is still born without most of the default spkgs, including sympy, jmol, palp, cremona-minidatabase, R, and probably many other.

I don't see any problems in install.log. llvm is not mentioned anywhere, and neither is any of the above packages. In logs/pkgs I don't have any log files for the above packages (I do have for many other packages though, all of which I guess are installed.)

I'm at a total loss...

Best,
Johan

[leif]

Johan S. R. Nielsen wrote:

$ make --version

?

Did you build in parallel, and if so, does probably building
sequentially fix the problem (such that the missing packages then get
built)?


-leif

[Johan S. H. Rosenkilde]

> $ make --version
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu

> Did you build in parallel, and if so, does probably building
> sequentially fix the problem (such that the missing packages then get
> built)?

I just did "make"; the default is still sequential build, right? I have
no suspect environment variables.

Best,
Johan

[leif]

Johan S. H. Rosenkilde wrote:

Except for the Sage library (and probably docbuilding, not sure), yes.
If you explicitly use 'make -j1', then also the Sage library will be
built sequentially, still using Python's multi-processing though, as
does docbuilding.


Forgot to ask:

How does at all local/var/lib/sage/installed/ look like?


-leif

[Johan S. H. Rosenkilde]

> Except for the Sage library (and probably docbuilding, not sure), yes.
> If you explicitly use 'make -j1', then also the Sage library will be
> built sequentially, still using Python's multi-processing though, as
> does docbuilding.

I've done `make distclean && make -j1`. Let's see how that one goes...

> Forgot to ask:
>
> How does at all local/var/lib/sage/installed/ look like?

See below. It doesn't contain the packages I know are missing. I don't
know if other packages are also missing.

alabaster-0.7.8
arb-2.8.1.p0
atlas-3.10.2.p2
babel-2.3.4
backports_abc-0.4
backports_shutil_get_terminal_size-1.0.0.p0
backports_ssl_match_hostname-3.5.0.1
boost_cropped-1.58.0
brial-0.8.4.3
bzip2-1.0.6-20150304
cephes-2.8.p1
certifi-2016.2.28
cliquer-1.21.p3
conway_polynomials-0.4.p0
cycler-0.10.0
cysignals-1.1.1
cython-0.24.p0
dateutil-2.2
decorator-4.0.6
docutils-0.12
ecl-16.1.2.p0
eclib-20150827
ecm-6.4.4
fflas_ffpack-1.6.0.p2
flask-0.10.1
flask_autoindex-0.6
flask_babel-0.9
flask_oldsessions-0.10
flask_openid-1.2.5
flask_silk-0.2
flint-2.5.2.p0
freetype-2.6.3
future-0.15.2
gap-4.8.3
gc-7.2f.p0
gf2x-1.1.p1
givaro-3.7.1
glpk-4.55.p1
gsl-2.1
iconv-1.14
imagesize-0.7.1
iml-1.0.4p1.p1
ipykernel-4.3.1
ipython-4.2.0
ipython_genutils-0.1.0
itsdangerous-0.24
jinja2-2.8
jupyter_client-4.2.2
jupyter_core-4.1.0
lcalc-1.23.p14
libfplll-20160331
libgap-4.8.3
libgd-2.1.1.1.p0
libpng-1.2.51.p0
linbox-1.3.2.p1
lrcalc-1.2.p0
m4ri-20140914
m4rie-20150908
markupsafe-0.23
mathjax-2.5
matplotlib-1.5.1
maxima-5.35.1.p2
mpc-1.0.3.p0
mpfi-1.5.1
mpfr-3.1.4
mpir-2.7.2
mpmath-0.19
ncurses-5.9.20131221
networkx-1.11
ntl-9.8.1.p0
numpy-1.11.0.p0
pari-2.8-2771-gb70b447.p0
pari_galdata-20080411
pari_seadata_small-20090618
patch-2.7.5
pathlib2-2.1.0
pathpy-7.1
pexpect-4.0.1.p1
pickleshare-0.7.2
pillow-3.2.0
pip-8.1.2
pkgconf-0.9.7
pkgconfig-1.1.0
planarity-2.2.0
ppl-1.1
ptyprocess-0.5.1
pygments-2.1.3
pynac-0.6.7
pyparsing-2.1.4
python2-2.7.10.p2
python_openid-2.2.5
pytz-2016.4
pyzmq-15.2.0
ratpoints-2.1.3.p3
readline-6.3.008
rw-0.7
sagenb-0.13
scipy-0.17.1
setuptools-21.0.0
setuptools_scm-1.11.0
simplegeneric-0.8.1
singledispatch-3.4.0.3
singular-3.1.7p1.p1
six-1.10.0
snowballstemmer-1.2.1
speaklater-1.3
sphinx-1.4.1.p1
sqlite-autoconf-3130000
symmetrica-2.0.p9
tachyon-0.98.9.p5
tornado-4.3
traitlets-4.2.1
twisted-16.2.0
werkzeug-0.11.9
zeromq-4.0.5
zlib-1.2.8.p0
zn_poly-0.9.p11
zope_interface-4.1.3


Best,
Johan

--

[Johan S. H. Rosenkilde]

Hi,

I just noticed, in the very beginning when doing "make" after distclean,
the build process prints a list of packages:

checking package versions...
4ti2-1.6.7
alabaster-0.7.8
arb-2.8.1.p0
...

This list *does* contain the uninstalled packages:

database_cremona_ellcurve-20160211
jmol-14.2.11_2015.01.20
palp-2.1.p1
sympy-1.0.p0
...


This list is not piped out to install.log.

Best,
Johan

[leif]

Johan S. H. Rosenkilde wrote:
>> Except for the Sage library (and probably docbuilding, not sure), yes.
>> If you explicitly use 'make -j1', then also the Sage library will be
>> built sequentially, still using Python's multi-processing though, as
>> does docbuilding.
>
> I've done `make distclean && make -j1`. Let's see how that one goes...

If that still omitted standard packages, I'd suggest to experiment with
GNU make's debug options (you can also add '-n' aka '--dry-run').

'-d' (='--debug=a') is far too verbose, but e.g. '--debug=bvjm',
probably also with '--trace' may give you a clue what's happening (or
why 'make' /doesn't/ build some targets).


-leif

[Johan S. H. Rosenkilde]

Hi,

Sorry for not getting pack to this thread before. I'm still having this
issue that newly compiled Sage is missing many packages.

Overnight I did `make distclean && make --trace --debug=bvjm` on Sage
7.3.beta7 with #20926 applied (gcc6.1 C++ workarounds). I had downgraded
my Make to version 4.1.

The result is the same: Building the docs crashed due to uninstalled
jmol. Other packages are also missing: palp, sympy,
database_cremona_ellcurve, etc.

The missing packages are listed in the beginning of the build:

Makefile:24: update target 'build/make/Makefile' due to: <long list
of dependencies including palp, sympy, jmol, etc.>

They are also mentioned a few pages below:

...
checking for sqrt in -lm... yes
checking multiprecision library... MPIR
checking BLAS library... atlas

checking package versions...
4ti2-1.6.7
alabaster-0.7.8

... <long list of packages including palp, sympy, jmol, etc.>


However, none of the not-installed packages are mentioned again in the
logs! Those that are installed are of course in the log, a few examples:

...
[arb-2.8.1.p0] Finished installing arb-2.8.1.p0.spkg
Successfully remade target file '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/arb-2.8.1.p0'.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/atlas-3.10.2.p2' does not exist.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/python2-2.7.10.p2' does not exist.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/readline-6.3.008' does not exist.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/ncurses-5.9.20131221' does not exist.
Must remake target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/ncurses-5.9.20131221'.
Makefile:2142: target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/ncurses-5.9.20131221' does not exist
sage-logger -p 'sage-spkg ncurses-5.9.20131221' '/home/jsrn/mat/sage_coding_project/logs/pkgs/ncurses-5.9.20131221.log'
...
[ipython_genutils-0.1.0] Finished installing ipython_genutils-0.1.0.spkg
Successfully remade target file '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/ipython_genutils-0.1.0'.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/decorator-4.0.6' does not exist.
Must remake target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/decorator-4.0.6'.
Makefile:1436: update target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/decorator-4.0.6' due to: /home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/setuptools-21.0.0
sage-logger -p 'sage-spkg decorator-4.0.6' '/home/jsrn/mat/sage_coding_project/logs/pkgs/decorator-4.0.6.log'
[decorator-4.0.6] Found local metadata for decorator-4.0.6
...
[sphinx-1.4.1.p1] Finished installing sphinx-1.4.1.p1.spkg
Successfully remade target file '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/sphinx-1.4.1.p1'.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/sagenb-0.13' does not exist.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/flask-0.10.1' does not exist.
File '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/werkzeug-0.11.9' does not exist.
Must remake target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/werkzeug-0.11.9'.
Makefile:2804: update target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/werkzeug-0.11.9' due to: /home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/python2-2.7.10.p2
sage-logger -p 'sage-spkg werkzeug-0.11.9' '/home/jsrn/mat/sage_coding_project/logs/pkgs/werkzeug-0.11.9.log'
...
Makefile:2828: target '/home/jsrn/mat/sage_coding_project/local/var/lib/sage/installed/zlib-1.2.8.p0' does not exist
sage-logger -p 'sage-spkg zlib-1.2.8.p0' '/home/jsrn/mat/sage_coding_project/logs/pkgs/zlib-1.2.8.p0.log'



It looks to me like some explicit dependencies to the not-installed
packages, as has been discussed with Jmol recently, must have been
removed or altered, and that's why Make is not picking up those targets.

Any thoughts?

--