configuration with LAPACK/BLAS fails

[Ahmad Shahba]

I am trying to compile deal.ii (version 9.1.1) on my Ubuntu laptop with gcc compilers. I have tried two methods to configure deal.ii with LAPACK/BLAS and both failed in a similar manner.

method 1: Using LAPACK/BLAS in my PETSc (version 3.9) installation

Motivated by the instructions given here, I use the following CMake command 

export PREFIX_PETSC_ARCH=$PETSC_DIR/$PETSC_ARCH

cmake \

   -DDEAL_II_COMPONENT_DOCUMENTATION=OFF \

   -DGSL_DIR=$HOME/workspace/gsl \

   -DHDF5_DIR=$PREFIX_PETSC_ARCH \

   -DMETIS_DIR=$PREFIX_PETSC_ARCH \

   -DP4EST_DIR=$HOME/workspace/p4est-2.2/FAST \

   -DDEAL_II_WITH_P4EST=ON \

   -DPETSC_DIR=$PETSC_DIR \

   -DPETSC_ARCH=$PETSC_ARCH \

   -DSLEPC_DIR=$SLEPC_DIR \

   -DZLIB_DIR=$PREFIX_PETSC_ARCH \

   -DDEAL_II_WITH_LAPACK=ON \

   -DLAPACK_FOUND=true \

   -DLAPACK_LIBRARIES="$PREFIX_PETSC_ARCH/lib/libflapack.a;$PREFIX_PETSC_ARCH/lib/libfblas.a" \

   -DLAPACK_LINKER_FLAGS="-lgfortran -lm" \

   -DDEAL_II_WITH_MPI=ON \

   -DCMAKE_INSTALL_PREFIX=path/to/some/installation/directory \

   ../

The configuration fails with prints an error message like

BLAS/LAPACK symbol check failed! This usually means that your BLAS/LAPACK

  installation is incomplete or the link line is broken.

I am attaching some CMake log and error files.

method 2:Linking to the LAPACK/BLAS whick comes with MKL

For another project, I had use gfortran to compile LAPACK/BLAS which comes with Intel MKL. Linking to these libraries also failed in the same manner. Here is the CMake command

export PREFIX_PETSC_ARCH=$PETSC_DIR/$PETSC_ARCH

cmake \

   -DDEAL_II_COMPONENT_DOCUMENTATION=OFF \

   -DGSL_DIR=$HOME/workspace/gsl \

   -DHDF5_DIR=$PREFIX_PETSC_ARCH \

   -DMETIS_DIR=$PREFIX_PETSC_ARCH \

   -DP4EST_DIR=$HOME/workspace/p4est-2.2/FAST \

   -DDEAL_II_WITH_P4EST=ON \

   -DPETSC_DIR=$PETSC_DIR \

   -DPETSC_ARCH=$PETSC_ARCH \

   -DSLEPC_DIR=$SLEPC_DIR \

   -DZLIB_DIR=$PREFIX_PETSC_ARCH \

   -DDEAL_II_WITH_LAPACK=ON \

   -DLAPACK_FOUND=true \

   -DLAPACK_LIBRARIES="${myInstallations}/lib/intel64/libmkl_blas95_lp64.a;${myInstallations}/lib/intel64/libmkl_lapack95_lp64.a;${MKLROOT}/lib/intel64/libmkl_gf_lp64.a;${MKLROOT}/lib/intel64/libmkl_sequential.a;${MKLROOT}/lib/intel64/libmkl_core.a" \

   -DLAPACK_LINKER_FLAGS="-lgfortran -lpthread -lm -ldl" \

   -DDEAL_II_WITH_BLAS=ON \

   -DBLAS_FOUND=true \

   -DBLAS_LIBRARIES="${myInstallations}/lib/intel64/libmkl_blas95_lp64.a;${myInstallations}/lib/intel64/libmkl_lapack95_lp64.a;${MKLROOT}/lib/intel64/libmkl_gf_lp64.a;${MKLROOT}/lib/intel64/libmkl_sequential.a;${MKLROOT}/lib/intel64/libmkl_core.a" \

   -DBLAS_LINKER_FLAGS="-lgfortran -lpthread -lm -ldl" \

   -DSCALAPACK_DIR==$PREFIX_PETSC_ARCH \

   -DDEAL_II_WITH_MPI=ON \

   -DCMAKE_INSTALL_PREFIX=path/to/some/installation/directory \

   ../

Any thoughts how I can get this done? 

Thanks in advance,

Ahmad

 

[Yang Li]

Spack offers a powerful alternative way to install DEAL.II.

--
The deal.II project is located at http://www.dealii.org/
For mailing list/forum options, see https://groups.google.com/d/forum/dealii?hl=en
---
You received this message because you are subscribed to the Google Groups "deal.II User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/60f9bf26-95d8-4b17-a529-1a498030433f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Li Yang

[Ahmad Shahba]

Thanks for your response. I want to know why CMake cannot locate lapack and blas libraries in the path I explicitly provide. I feel that the answer to this question should be simple and yet I don't know what the answer is :( I don't really want to resort to other tools such as spack unless it is absolutely the only way to get around this issue  

To unsubscribe from this group and stop receiving emails from it, send an email to dea...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/60f9bf26-95d8-4b17-a529-1a498030433f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Li Yang

[Jean-Paul Pelteret]

Hi Ahmad,

Although I don’t have an answer for you, I just want to say that what you’re going here looks somewhat reasonable to me. Its unclear to me why this test would fail:

/usr/bin/cc  -DLAPACK_SYMBOL_CHECK -pthread    -rdynamic CMakeFiles/cmTC_3cdf4.dir/src.c.o  -o cmTC_3cdf4 -lgfortran -lm /home/mamiri/workspace/petsc/petsc3.9_openmpi3.2/lib/libflapack.a /home/mamiri/workspace/petsc/petsc3.9_openmpi3.2/lib/libfblas.a 

/home/mamiri/workspace/petsc/petsc3.9_openmpi3.2/lib/libflapack.a(dgelsd.o): In function `dgelsd_':

dgelsd.f:(.text+0x26a): undefined reference to `log’

…. + other missing symbols for math functions…

since you specifically pass the -lm flag on the link line. As for this test:

/usr/bin/c++   -DMKL_SYMBOL_CHECK -pthread    -rdynamic CMakeFiles/cmTC_527e8.dir/src.cxx.o  -o cmTC_527e8 -lgfortran -lm /home/mamiri/workspace/petsc/petsc3.9_openmpi3.2/lib/libflapack.a /home/mamiri/workspace/petsc/petsc3.9_openmpi3.2/lib/libfblas.a 

CMakeFiles/cmTC_527e8.dir/src.cxx.o: In function `main':

src.cxx:(.text+0xe9): undefined reference to `MKL_Domatcopy'

perhaps your missing a linker flag? I’m not familiar with MKL, so maybe there is (genuinely) something missing there. Maybe @Matthias or one of the other developers has some suggestions?

I do have one suggestion: Have you perhaps tried installing OpenBLAS and using that as the BLAS and LAPACK provider? At least then its stand-alone, and you’re not relying on PETSc to download and correctly configure it for you. This is what happens in my Spack setup, which seems to work just fine. If you do this, then don’t forget to point PETSc to this BLAS/LAPACK provider too.

Sorry I can’t be of more help, but hopefully someone else has a concrete solution for you and you can get this resolved soon.

Best,

Jean-Paul

To unsubscribe from this group and stop receiving emails from it, send an email to dealii+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dealii/ed83e369-a6ce-4db1-aa71-efd7bff4672a%40googlegroups.com.

[Bruno Turcksin]

Ahmad,

The first thing is that you should not try to set the flags yourself. You should let CMake take care of that. You should just set the paths (BLAS_DIR and LAPACK_DIR) and let CMake do the rest. The second thing is that I am not sure sure that you are using the right MKL libraries. You can go here to check what you need.

Best,

Bruno

[Ahmad Shahba]

Thanks Jean-Paul and Bruno for your input

Regarding OpenBLAS: I installed OpenBLAS, reconfigured PETSc (to use OpenBLAS). Using the following CMake invocation

export PREFIX_PETSC_ARCH=$PETSC_DIR/$PETSC_ARCH

cmake \

   -DDEAL_II_COMPONENT_DOCUMENTATION=OFF \

   -DGSL_DIR=$HOME/workspace/gsl \

   -DHDF5_DIR=$PREFIX_PETSC_ARCH \

   -DMETIS_DIR=$PREFIX_PETSC_ARCH \

   -DP4EST_DIR=$HOME/workspace/p4est-2.2/FAST \

   -DDEAL_II_WITH_P4EST=ON \

   -DPETSC_DIR=$PETSC_DIR \

   -DPETSC_ARCH=$PETSC_ARCH \

   -DSLEPC_DIR=$SLEPC_DIR \

   -DZLIB_DIR=$PREFIX_PETSC_ARCH \

   -DDEAL_II_WITH_LAPACK=ON \

   -DLAPACK_FOUND=true \

   -DLAPACK_LIBRARIES="/opt/OpenBLAS/lib/libopenblas.so" \

   -DLAPACK_LINKER_FLAGS="-lgfortran -lpthread -lm -ldl" \

   -DDEAL_II_WITH_MPI=ON \

   -DCMAKE_INSTALL_PREFIX=$installationPath \

   ../

Now I can successfully configure deal.ii and compile the library. I am running the post-installation tests by invoking

make test

I see that all the tests pass except for one related to gmsh. It produces an error

This exception -- which is used in many places in the library -- usually indicates that some condition which the author of the code thought must be satisfied at a certain point in an algorithm, is not fulfilled. An example would be that the first part of an algorithm sorts elements of an array in ascending order, and a second part of the algorithm later encounters an element that is not larger than the previous one.

There is usually not very much you can do if you encounter such an exception since it indicates an error in deal.II, not in your own program. 

I am going to post this issue as another question in the forum

Regarding LAPACK/BLAS in MKL: It appears that I was missing one library (libmkl_intel_lp64.so). I reconfigured deal.ii using the following CMake command

export PREFIX_PETSC_ARCH=$PETSC_DIR/$PETSC_ARCH
cmake \
   -DDEAL_II_COMPONENT_DOCUMENTATION=OFF \
   -DGSL_DIR=$HOME/workspace/gsl \
   -DHDF5_DIR=$PREFIX_PETSC_ARCH \
   -DMETIS_DIR=$PREFIX_PETSC_ARCH \
   -DP4EST_DIR=$HOME/workspace/p4est-2.2/FAST \
   -DDEAL_II_WITH_P4EST=ON \
   -DPETSC_DIR=$PETSC_DIR \
   -DPETSC_ARCH=$PETSC_ARCH \
   -DSLEPC_DIR=$SLEPC_DIR \
   -DZLIB_DIR=$PREFIX_PETSC_ARCH \
   -DDEAL_II_WITH_LAPACK=ON \
   -DLAPACK_FOUND=true \

   -DLAPACK_LIBRARIES="${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a;${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a;${MKLROOT}/lib/intel64/libmkl_gf_lp64.a;${MKLROOT}/lib/intel64/libmkl_intel_lp64.so;${MKLROOT}/lib/intel64/libmkl_sequential.so;${MKLROOT}/lib/intel64/libmkl_core.so" \

   -DLAPACK_LINKER_FLAGS="-lgfortran -lpthread -lm -ldl" \
   -DDEAL_II_WITH_BLAS=ON \
   -DBLAS_FOUND=true \

   -DBLAS_LIBRARIES="${MKLROOT}/lib/intel64/libmkl_blas95_lp64.a;${MKLROOT}/lib/intel64/libmkl_lapack95_lp64.a;${MKLROOT}/lib/intel64/libmkl_gf_lp64.a;${MKLROOT}/lib/intel64/libmkl_intel_lp64.so;${MKLROOT}/lib/intel64/libmkl_sequential.so;${MKLROOT}/lib/intel64/libmkl_core.so" \

   -DBLAS_LINKER_FLAGS="-lgfortran -lpthread -lm -ldl" \
   -DSCALAPACK_DIR==$PREFIX_PETSC_ARCH \
   -DDEAL_II_WITH_MPI=ON \

   -DCMAKE_INSTALL_PREFIX=$installationPath \
   ../

CMake still fails to locate LAPACK/BLAS. The CMakeError.Log file is attached. Bruno's comment and previous post made me think of something. The LAPACK and BLAS libraries provided in MKL are static .a libraries. Could this be the reason why CMake cannot locate them?

[Ahmad Shahba]

I fixed the failing gmsh test case. I am reporting it here, just in case someone else faced the same problem. It turned out that the gmsh I was using was linking to the blas library shipped with Ubuntu, but since my manually installed openblas appeared first on my path, gmsh wasn't picking up the right blas library. I ended up installing gmsh on my ubuntu machine and had deal.ii use the manually installed gmsh library, instead. Now all deal.ii tests pass

Thanks everyone for their suggestions and help