Source Build Fails on GCC-10

161 views
Skip to first unread message

jorda...@gmail.com

unread,
Dec 22, 2021, 1:40:30 PM12/22/21
to sage-devel

I'm trying to build SageMath from source on Ubuntu 20.04. Below is the exact process I tried to follow (taking notes to be able to pass to other students with little/no linux experience--like me--so I try to explain each step), but the build crashed on trying to build gcc-10 with "undefined reference to ZSTD_compress" and a bunch of other ZSTD-related items. Looking deeper into the config log, it turns out that the gcc's I supplied don't manifest the OpenMC flags ("unknown"), causing it to assume it needs to build it. But when I tested the supplied compilers on a "hello world" .c file using "[gcc-7 | gcc-10] hello.c -fopenmp -o hello", it compiled without error, suggesting it has OpenMC available.

So two questions: why doesn't the below install all needed dependencies for the installer gcc-10 build, and more importantly, why can't ./configure figure out that the compiler I gave it supports OpenMC? Thanks!

  • This method will install a new sage python environment that will show up in our anaconda-navigator called sage-pyenv.

  • Install all the prerecs. Highly recommended ones for better experience are after yasm; optional ones to speed up build time after tk-dev (Note: gcc, g++, and gfortran installed to /usr/bin/):

    sudo apt-get install bc binutils bzip2 ca-certificates cliquer cmake curl ecl eclib-tools fflas-ffpack flintqs g++ gcc gfan gfortran glpk-utils gmp-ecm lcalc libatomic-ops-dev libboost-dev libbraiding-dev libbrial-dev libbrial-groebner-dev libbz2-dev libcdd-dev libcdd-tools libcliquer-dev libcurl4-openssl-dev libec-dev libecm-dev libffi-dev libflint-arb-dev libflint-dev libfreetype6-dev libgc-dev libgd-dev libgf2x-dev libgiac-dev libgivaro-dev libglpk-dev libgmp-dev libgsl-dev libhomfly-dev libiml-dev liblfunction-dev liblrcalc-dev liblzma-dev libm4rie-dev libmpc-dev libmpfi-dev libmpfr-dev libncurses5-dev libntl-dev libopenblas-dev libpari-dev libpcre3-dev libplanarity-dev libppl-dev libpython3-dev libreadline-dev librw-dev libsqlite3-dev libssl-dev libsuitesparse-dev libsymmetrica2-dev libz-dev libzmq3-dev libzn-poly-dev m4 make nauty openssl palp pari-doc pari-elldata pari-galdata pari-galpol pari-gp2c pari-seadata patch perl pkg-config planarity ppl-dev python3 python3 python3-distutils r-base-dev r-cran-lattice sqlite3 sympow tachyon tar tox xcas xz-utils yasm gcc-7 g++-7 gfortran-7 dvipng ffmpeg imagemagick texlive texlive-latex-recommended texlive-latex-recommended-doc texlive-fonts-recommended texlive-fonts-recommended-doc texlive-xetex latexmk pandoc default-jdk libavdevice-dev tk tk-dev coinor-cbc coinor-libcbc-dev git graphviz libfile-slurp-perl libigraph-dev libisl-dev libjson-perl libmongodb-perl libnauty-dev libperl-dev libsvg-perl libterm-readkey-perl libterm-readline-gnu-perl libterm-readline-gnu-perl libxml-libxslt-perl libxml-writer-perl libxml2-dev libxml2-dev lrslib ninja-build pari-gp2c

  • Get the source code:

    mkdir ~/Codes

    cd ~/Codes

    git clone -c core.symlinks=true --branch master https://github.com/sagemath/sage.git

  • Set up Python environment:

    cd ~/Codes/sage

    ./bootstrap

    • After building the environment, I was told that conda went ahead and built the environment as Python 3.10, which is too new for Sage. Per this post, 3.8 and 3.9 are supported. It will build a 3.9 during build. So, tweak some things to make sure we get that version.
    • Go into ~/Codes/sage/environment-optional.yml, and
      • change the environment name to sage-pyenv
      • add python=3.9 to dependencies, per this post

    conda env create -f environment-optional.yml

    conda activate sage-pyenv

  • Build it:

    ./bootstrap

    • Point it to system gcc-7 and fortran-7 for compilation, since it seems to fail when trying to build it's own gcc-10. It can’t use gcc-11 as of the time of this writing. Also point it to this conda environment we just created. EDIT: I then tried installing gcc-10, g++-10, and gfortran-10 and pointing to those, but it kept saying that it can't detect OpenMC support (though independent tests showed it is supported, leading it to still try and fail to build gcc-10.3.0).

    CC=gcc-7 CXX=g++-7 FC=gfortran-7 ./configure --prefix=$CONDA_PREFIX --with-python=/home/nuker/anaconda3/envs/sage-pyenv/bin/python

    • This came back with a list of things that will provide extra capability and help build time. I just made sure that those got installed:

      conda install arb cddlib curl eclib ecm libflint giac iml lcalc libgd libhomfly openssl pari pari-elldata pari-galdata pari-galpol pari-seadata pari-galdata pari-seadata-small r r-essentials suitesparse xz lrslib pari-nftables

    • Rerun the ./configure line above (other places suggest ./config.status --recheck && ./config.status, but those would still install a new python-3.9 install, suggesting it lost the --with-python argument).

      make

After make, it would crash out with notes about ZSTD when compiling gcc-10.3.0. I tried to add zstd to the conda install line, and get it with apt-get, but neither could be found when compiling gcc-10.3.0 during make (probably because it's isolated to a different path on purpose).

How can I solve the OpenMC flag issue or provide ZSTD for the gcc-10.3.0 build during make? Thanks!

Matthias Koeppe

unread,
Dec 22, 2021, 1:53:30 PM12/22/21
to sage-devel
Don't try to mix system packages and conda packages. 
Pick a method.
Either work with Debian packages (apt-get) and deactivate all conda. 
Or set up and activate the conda environment; and don't try to override compilers.
Also, do not use "./configure --prefix=$CONDA_PREFIX" -- this is not supported because Sage packages could overwrite conda-installed packages.

jorda...@gmail.com

unread,
Dec 23, 2021, 11:51:37 AM12/23/21
to sage-devel
I guess I misunderstood--I thought that it used both system packages and a python environment. I understood from reading the install guide that installing the system ones were supposed to reduce compile time because Sage could link to system packages and wouldn't have to build them, and the python environment was used for linking things and tying into Jupyter. Sounds like that's not right.

Is Conda installing executable / binaries of packages in its environments, not just python packages? If I use just system packages and skip the python environment, doesn't Sage build itself a python 3.9 environment? Is there a way to create the python environment with just the necessary python stuff so that I can see it in anaconda-navigator, or is the sage environment not something I ever want access to?

I'm not very experienced with building software from source, so I only know enough to understand terminology, but not have any deep understanding of what it's trying to do.

The following did build:

  • This method will install a new sage python environment that will show up in our anaconda-navigator called sage-pyenv. We should never seen to touch it, but I’d like to be able to see what’s in it.

  • Set up a simple python environment for it to build into:

    • Open anaconda-navigator

    • Create a new environment, call it sage-pyenv, and set it to version 3.8.

    • Close anaconda-navigator.

  • We can either use that python environment to install all system prereqs, or do it using system codes. I chose system codes.

  • Install all the prerecs. Highly recommended ones for better experience are after yasm; optional ones to speed up build time after tk-dev (Note: can’t install with gcc/gfortran 11, the system default, so getting 10):

    • sudo apt-get install bc binutils bzip2 ca-certificates cliquer cmake curl ecl eclib-tools fflas-ffpack flintqs g++ gcc gfan gfortran glpk-utils gmp-ecm lcalc libatomic-ops-dev libboost-dev libbraiding-dev libbrial-dev libbrial-groebner-dev libbz2-dev libcdd-dev libcdd-tools libcliquer-dev libcurl4-openssl-dev libec-dev libecm-dev libffi-dev libflint-arb-dev libflint-dev libfreetype6-dev libgc-dev libgd-dev libgf2x-dev libgiac-dev libgivaro-dev libglpk-dev libgmp-dev libgsl-dev libhomfly-dev libiml-dev liblfunction-dev liblrcalc-dev liblzma-dev libm4rie-dev libmpc-dev libmpfi-dev libmpfr-dev libncurses5-dev libntl-dev libopenblas-dev libpari-dev libpcre3-dev libplanarity-dev libppl-dev libpython3-dev libreadline-dev librw-dev libsqlite3-dev libssl-dev libsuitesparse-dev libsymmetrica2-dev libz-dev libzmq3-dev libzn-poly-dev m4 make nauty openssl palp pari-doc pari-elldata pari-galdata pari-galpol pari-gp2c pari-seadata patch perl pkg-config planarity ppl-dev python3 python3 python3-distutils r-base-dev r-cran-lattice sqlite3 sympow tachyon tar tox xcas xz-utils yasm gcc-10 g++-10 gfortran-10 dvipng ffmpeg imagemagick texlive texlive-latex-recommended texlive-latex-recommended-doc texlive-fonts-recommended texlive-fonts-recommended-doc texlive-xetex latexmk pandoc default-jdk libavdevice-dev tk tk-dev coinor-cbc coinor-libcbc-dev git graphviz libfile-slurp-perl libigraph-dev libisl-dev libjson-perl libmongodb-perl libnauty-dev libperl-dev libsvg-perl libterm-readkey-perl libterm-readline-gnu-perl libterm-readline-gnu-perl libxml-libxslt-perl libxml-writer-perl libxml2-dev libxml2-dev lrslib ninja-build pari-gp2c

  • Disable conda, since we don’t need it to do any package management for sage.

    • conda deactivate

  • Configure the build environment

    • cd ~/Codes/sage

    • ./bootstrap

  • Point it to system gcc-10 and fortran-10 for compilation. It can’t use gcc-11 as of the time of this writing. We also want it to build it’s python stuff in the sage-pyenv environment, so let’s try to do that.

    • CC=gcc-10 CXX=g++-10 FC=gfortran-10 ./configure --prefix=/home/nuker/Codes/sage/local --with-python=/home/nuker/anaconda3/envs/sage-pyenv/bin/python

    • This came back with a list of things that will provide extra capability and help build time. I just made sure that those got installed (with optional packages after pari-seadata):

    • sudo apt-get update

    • sudo apt-get install libflint-arb-dev libcdd-dev libcdd-tools ecl libec-dev eclib-tools libflint-dev libgiac-dev xcas lcalc liblfunction-dev pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata texlive-latex-extra texlive-xetex latexmk dvipng default-jdk ffmpeg libavdevice-dev pari-gp2c libigraph-dev lrslib

    • If any were installed (I don’t think it needed to install any more), rerun the ./configure line above (other places suggest ./config.status --recheck && ./config.status, but those didn’t work for me).

  • MAKE='make -j4' make (this will use 4 cores to build SageMath. For me, one core took over 200 minutes. 4 took 77.)
  • Copy it to the regular install location per this post:
    • sudo cp -rl /home/nuker/Codes/sage/local/* /usr/local
Is there any serious problem with doing things this way?

jorda...@gmail.com

unread,
Dec 23, 2021, 11:57:31 AM12/23/21
to sage-devel
I take it back. The above, for which sage tells me /usr/local, but when I try to run it, it says:
Traceback (most recent call last):
  File "././sage-config", line 5, in <module>
    from sage_conf import _main
ModuleNotFoundError: No module named 'sage_conf'
Traceback (most recent call last):
  File "././sage-config", line 5, in <module>
    from sage_conf import _main
ModuleNotFoundError: No module named 'sage_conf'
Traceback (most recent call last):
  File "/home/nuker/Codes/sage/src/bin/sage-ipython", line 9, in <module>
    from sage.misc.banner import banner
ModuleNotFoundError: No module named 'sage'

Matthias Koeppe

unread,
Dec 23, 2021, 12:01:07 PM12/23/21
to sage-devel
On Thursday, December 23, 2021 at 8:51:37 AM UTC-8 jorda...@gmail.com wrote:
Is Conda installing executable / binaries of packages in its environments, not just python packages? 

Yes.
 

jorda...@gmail.com

unread,
Jan 2, 2022, 12:34:07 PM1/2/22
to sage-devel
Here is the code that finally worked.
The user on the Ubuntu 20.04 box is "nuker". Replace paths and user names as appropriate.

    • This method will install a new sage python environment that will show up in our anaconda-navigator called sage-pyenv. We should never seen to touch it, but I’d like to be able to see what’s in it.

    • Set up a simple python environment for it to build into:

      • Open anaconda-navigator

      • Create a new environment, call it sage-pyenv, and set it to version 3.8.

      • Close anaconda-navigator.

    • We can either use that python environment to install all system prereqs, or do it using system codes. I chose system codes.

    • Install all the prerecs. Highly recommended ones for better experience are after yasm; optional ones to speed up build time after tk-dev (Note: can’t install with gcc/gfortran 11, the system default, so getting 10):

      • sudo apt-get install bc binutils bzip2 ca-certificates cliquer cmake curl ecl eclib-tools fflas-ffpack flintqs g++ gcc gfan gfortran glpk-utils gmp-ecm lcalc libatomic-ops-dev libboost-dev libbraiding-dev libbrial-dev libbrial-groebner-dev libbz2-dev libcdd-dev libcdd-tools libcliquer-dev libcurl4-openssl-dev libec-dev libecm-dev libffi-dev libflint-arb-dev libflint-dev libfreetype6-dev libgc-dev libgd-dev libgf2x-dev libgiac-dev libgivaro-dev libglpk-dev libgmp-dev libgsl-dev libhomfly-dev libiml-dev liblfunction-dev liblrcalc-dev liblzma-dev libm4rie-dev libmpc-dev libmpfi-dev libmpfr-dev libncurses5-dev libntl-dev libopenblas-dev libpari-dev libpcre3-dev libplanarity-dev libppl-dev libpython3-dev libreadline-dev librw-dev libsqlite3-dev libssl-dev libsuitesparse-dev libsymmetrica2-dev libz-dev libzmq3-dev libzn-poly-dev m4 make nauty openssl palp pari-doc pari-elldata pari-galdata pari-galpol pari-gp2c pari-seadata patch perl pkg-config planarity ppl-dev python3 python3 python3-distutils r-base-dev r-cran-lattice sqlite3 sympow tachyon tar tox xcas xz-utils yasm gcc-10 g++-10 gfortran-10 dvipng ffmpeg imagemagick texlive texlive-latex-recommended texlive-latex-recommended-doc texlive-fonts-recommended texlive-fonts-recommended-doc texlive-xetex latexmk pandoc default-jdk libavdevice-dev tk tk-dev coinor-cbc coinor-libcbc-dev git graphviz libfile-slurp-perl libigraph-dev libisl-dev libjson-perl libmongodb-perl libnauty-dev libperl-dev libsvg-perl libterm-readkey-perl libterm-readline-gnu-perl libterm-readline-gnu-perl libxml-libxslt-perl libxml-writer-perl libxml2-dev libxml2-dev lrslib ninja-build pari-gp2c

    • Get the source code:

    • Disable conda, since we don’t need it to do any package management for sage.

      • conda deactivate

    • Configure the build environment

      • cd ~/Codes/sage

      • ./bootstrap

    • Point it to system gcc-10 and fortran-10 for compilation. It can’t use gcc-11 as of the time of this writing. We also want it to build it’s python stuff in the sage-pyenv environment, so let’s try to do that.

      • CC=gcc-10 CXX=g++-10 FC=gfortran-10 ./configure --prefix=/home/nuker/Codes/sage/local --with-python=/home/nuker/anaconda3/envs/sage-pyenv/bin/python

      • This came back with a list of things that will provide extra capability and help build time. I just made sure that those got installed (with optional packages after pari-seadata):

      • sudo apt-get update

      • sudo apt-get install libflint-arb-dev libcdd-dev libcdd-tools ecl libec-dev eclib-tools libflint-dev libgiac-dev xcas lcalc liblfunction-dev pari-gp2c libpari-dev pari-doc pari-elldata pari-galdata pari-galpol pari-seadata texlive-latex-extra texlive-xetex latexmk dvipng default-jdk ffmpeg libavdevice-dev pari-gp2c libigraph-dev lrslib

      • If any were installed (I don’t think it needed to install any more), rerun the ./configure line above (other places suggest ./config.status --recheck && ./config.status, but those didn’t work for me).

      • MAKE='make -j4' make (this will use 4 cores to build SageMath. For me, one core took over 200 minutes. 4 took 77.)

    • Test the build (using parallel processes and omitting tests that take a long time, since we aren’t planning to be doing serious work in Sage). This command left me with a few cython doctest failing and one test failing, with a note that pytest wasn’t installed so wouldn’t be used:

      • make ptest

    • Move it to the regular install location per this post:

        • sudo cp -rl /home/nuker/Codes/sage/local/* /usr/local

        • sudo cp /home/nuker/Codes/sage/local/man/man1/* /usr/local/man/man1 (because man is already a link, not a directory from some other install)

      • Finally, install into Jupyter (the install process installed the kernelspec, but it wasn’t working for me)

        • sudo jupyter kernelspec install /home/nuker/Codes/sage/local/share/jupyter/kernels/sagemath

      Reply all
      Reply to author
      Forward
      0 new messages