Linking MKL

[Jack Poulson]

Hi Alex,

How to link MKL tends to change periodically and it is best to start from the recommendation from their link line advisor:
http://software.intel.com/sites/products/mkl/

If BLAS still cannot be linked properly, then the reason will be given towards the bottom of CMakeFiles/CMakeError.log. You will likely need to specify a Fortran library or two.

Jack

On Dec 23, 2012 9:06 AM, <agra...@gfz-potsdam.de> wrote:

Hello!

I'm trying to build Elemental on cluster using Intel Compiler + Intel MKL + IntelMPI toolchain. 

Using following cmake command:

cmake -D CMAKE_INSTALL_PREFIX=/home/ag/lib/elemental -D CMAKE_CXX_COMPILER=mpiicc -D CMAKE_C_COMPILER=mpiicc -D CMAKE_Fortran_COMPILER=mpiifort -D MATH_LIBS="-L/usr/local/opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lm" ..

I end up with error:

-- Using user-defined MATH_LIBS=-L/usr/local/opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lm

-- Looking for daxpy

-- Looking for daxpy - not found

-- Looking for daxpy_

-- Looking for daxpy_ - not found

CMake Error at CMakeLists.txt:316 (message):

  Could not determine BLAS format.

At the same time nm tells me there is BLAS:

elemental/build> nm /usr/local/opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64/libmkl_intel_lp64.so | grep daxpy                                                                                                                 0000000000113b70 T cblas_daxpy

0000000000113bb0 T cblas_daxpyi

0000000000103c30 T daxpy

0000000000103c30 T daxpy_

0000000000100540 T daxpyi

0000000000100540 T daxpyi_

                 U mkl_blas_daxpy

                 U mkl_blas_daxpyi

Any ideas why cmake can not find BLAS?

Thanks. 

--
 
 

[Yashar Hezaveh]

Hi Jack

 I'm having the same problem (i.e. " Could not determine BLAS format."). Haven't managed to resolve it from your previous response.

I'm installing elemental on a cluster. I have installed the required libraries locally (I installed ATLAS on "/home/yashar/ATLAS3.10" (which has a bin, and include folders in it), and libFlame on "/home/yashar/flame".)

Now to compile elemental, I issue this command:

cmake -D MATH_LIBS="/home/yashar/flame/lib/ ;-L/home/yashar/ATLAS3.10/lib -llapack -lblas -lm" CMAKE_INSTALL_PREFIX=/home/yashar ..

and the CMakeError.log file reads:

[yashar]$ more CMakeError.log 

Determining if the function daxpy exists failed with the following output:

Change Dir: /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec4215778255/fast"

/usr/bin/gmake -f CMakeFiles/cmTryCompileExec4215778255.dir/build.make CMakeFiles/cmTryCompileExec4215778255.dir/build

gmake[1]: Entering directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

/home/yashar/bin/cmake -E cmake_progress_report /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp/CMakeFiles 1

Building C object CMakeFiles/cmTryCompileExec4215778255.dir/CheckFunctionExists.c.o

/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=daxpy   -o CMakeFiles/cmTryCompileExec4215778255.dir/CheckFunctionExists.c.o   -c /home/yashar/share/c

make-2.8/Modules/CheckFunctionExists.c

Linking C executable cmTryCompileExec4215778255

/home/yashar/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec4215778255.dir/link.txt --verbose=1

/usr/bin/cc     -DCHECK_FUNCTION_EXISTS=daxpy    CMakeFiles/cmTryCompileExec4215778255.dir/CheckFunctionExists.c.o  -o cmTryCompileExec42157

78255 -rdynamic -L/home/yashar/ATLAS3.10/lib -llapack -lblas -lm 

/software/base-gcc/bin/ld: cannot find -lblas

collect2: ld returned 1 exit status

gmake[1]: *** [cmTryCompileExec4215778255] Error 1

gmake[1]: Leaving directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

gmake: *** [cmTryCompileExec4215778255/fast] Error 2

Determining if the function daxpy_ exists failed with the following output:

Change Dir: /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec2720716939/fast"

/usr/bin/gmake -f CMakeFiles/cmTryCompileExec2720716939.dir/build.make CMakeFiles/cmTryCompileExec2720716939.dir/build

gmake[1]: Entering directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

/home/yashar/bin/cmake -E cmake_progress_report /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp/CMakeFiles 1

Building C object CMakeFiles/cmTryCompileExec2720716939.dir/CheckFunctionExists.c.o

/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=daxpy_   -o CMakeFiles/cmTryCompileExec2720716939.dir/CheckFunctionExists.c.o   -c /home/yashar/share/

cmake-2.8/Modules/CheckFunctionExists.c

Linking C executable cmTryCompileExec2720716939

/home/yashar/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec2720716939.dir/link.txt --verbose=1

/usr/bin/cc     -DCHECK_FUNCTION_EXISTS=daxpy_    CMakeFiles/cmTryCompileExec2720716939.dir/CheckFunctionExists.c.o  -o cmTryCompileExec2720

716939 -rdynamic -L/home/yashar/ATLAS3.10/lib -llapack -lblas -lm 

/software/base-gcc/bin/ld: cannot find -lblas

collect2: ld returned 1 exit status

gmake[1]: *** [cmTryCompileExec2720716939] Error 1

gmake[1]: Leaving directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

gmake: *** [cmTryCompileExec2720716939/fast] Error 2

I have very limited experience with libraries, linking, etc. so please be as pedantic as you can. 

Thanks a lot!

Yashar

[Jeff Hammond]

You got this error: "/software/base-gcc/bin/ld: cannot find -lblas"

Please look in /home/yashar/ATLAS3.10/lib and see what the filename of
the BLAS library is. If you paste the result of "ls -l" in your
reply, I can help you if that's necessary.

GotoBLAS (http://www.tacc.utexas.edu/tacc-projects/gotoblas2) is
better than ATLAS in terms of performance and also requires only one
library to link. I encourage you to try it instead.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "elemental-maint" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to elemental-mai...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Jeff Hammond
jeff.s...@gmail.com

[Yashar Hezaveh]

Thanks Jeff

here's the list:

[yashar]$ cd /home/yashar/ATLAS3.10/lib

[yashar]$ ls -l

total 24356

-rw-r--r-- 1 yashar vjp-933-01 12485912 Jun 28 08:29 libatlas.a

-rw-r--r-- 1 yashar vjp-933-01   448892 Jun 28 08:29 libcblas.a

-rw-r--r-- 1 yashar vjp-933-01   568984 Jun 28 08:29 libf77blas.a

-rw-r--r-- 1 yashar vjp-933-01 10407894 Jun 28 08:29 liblapack.a

-rw-r--r-- 1 yashar vjp-933-01   449474 Jun 28 08:29 libptcblas.a

-rw-r--r-- 1 yashar vjp-933-01   569380 Jun 28 08:29 libptf77blas.a

Y

[Yashar Hezaveh]

I also installed GotoBLAS, this is what I got at the end of "make" (for GotoBLAS):

make[1]: Leaving directory `/home/yashar/Elemental/GotoBLAS2/exports'

ln -fs libgoto2_nehalemp-r1.13.so libgoto2.so

 GotoBLAS build complete.

  OS               ... Linux             

  Architecture     ... x86_64               

  BINARY           ... 64bit                 

  C compiler       ... GCC  (command line : gcc)

  Fortran compiler ... GFORTRAN  (command line : gfortran)

  Library Name     ... libgoto2_nehalemp-r1.13.a (Multi threaded; Max num-threads is 8)

so now I go back to "elemental/build" folder and type:

cmake -D MATH_LIBS="/home/yashar/flame/lib/ ;-L/home/yashar/Elemental/GotoBLAS2 -llapack -lblas -lm" CMAKE_INSTALL_PREFIX=/home/yashar ..

but i still get the "Could not determine BLAS format." error. 

Thanks

[Jeff Hammond]

then -lf77blas is what you need, not -lblas

[Jeff Hammond]

> Library Name ... libgoto2_nehalemp-r1.13.a (Multi threaded; Max num-threads is 8)

library name is the name of the library you need to link against.
-lblas is no more of a correct pointer to libgoto2_nehalemp-r1.13.a
than my name is Bananapants McGee.

try -lgoto2_nehalemp-r1.13 if you want to link against gotoblas or
soft link libblas.a to it (ln -s libgoto2_nehalemp-r1.13.a libblas.a)
so the linker can figure it out.

jeff

[Yashar Hezaveh]

Thanks Jeff

 I'm still doing something wrong. Still get "Could not determine BLAS format." when I enter:

cmake -D MATH_LIBS="/home/yashar/flame/lib/ ;-L/home/yashar/ATLAS3.10/lib -llapack -lf77blas -lm" CMAKE_INSTALL_PREFIX=/home/yashar ..

the error log:

[yashar]$ more CMakeFiles/CMakeError.log 

Determining if the function daxpy exists failed with the following output:

Change Dir: /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec4009289716/fast"

/usr/bin/gmake -f CMakeFiles/cmTryCompileExec4009289716.dir/build.make CMakeFiles/cmTryCompileExec4009289716.dir/build

gmake[1]: Entering directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

/home/yashar/bin/cmake -E cmake_progress_report /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp/CMakeFiles 1

Building C object CMakeFiles/cmTryCompileExec4009289716.dir/CheckFunctionExists.c.o

/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=daxpy   -o CMakeFiles/cmTryCompileExec4009289716.dir/CheckFunctionExists.c.o   -c /home/yashar/share/c

make-2.8/Modules/CheckFunctionExists.c

Linking C executable cmTryCompileExec4009289716

/home/yashar/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec4009289716.dir/link.txt --verbose=1

/usr/bin/cc     -DCHECK_FUNCTION_EXISTS=daxpy    CMakeFiles/cmTryCompileExec4009289716.dir/CheckFunctionExists.c.o  -o cmTryCompileExec40092

89716 -rdynamic -L/home/yashar/ATLAS3.10/lib -llapack -lf77blas -lm 

CMakeFiles/cmTryCompileExec4009289716.dir/CheckFunctionExists.c.o: In function `main':

CheckFunctionExists.c:(.text+0x15): undefined reference to `daxpy'

collect2: ld returned 1 exit status

gmake[1]: *** [cmTryCompileExec4009289716] Error 1

gmake[1]: Leaving directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

gmake: *** [cmTryCompileExec4009289716/fast] Error 2

Determining if the function daxpy_ exists failed with the following output:

Change Dir: /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp

Run Build Command:/usr/bin/gmake "cmTryCompileExec1328063564/fast"

/usr/bin/gmake -f CMakeFiles/cmTryCompileExec1328063564.dir/build.make CMakeFiles/cmTryCompileExec1328063564.dir/build

gmake[1]: Entering directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

/home/yashar/bin/cmake -E cmake_progress_report /home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp/CMakeFiles 1

Building C object CMakeFiles/cmTryCompileExec1328063564.dir/CheckFunctionExists.c.o

/usr/bin/cc   -DCHECK_FUNCTION_EXISTS=daxpy_   -o CMakeFiles/cmTryCompileExec1328063564.dir/CheckFunctionExists.c.o   -c /home/yashar/share/

cmake-2.8/Modules/CheckFunctionExists.c

Linking C executable cmTryCompileExec1328063564

/home/yashar/bin/cmake -E cmake_link_script CMakeFiles/cmTryCompileExec1328063564.dir/link.txt --verbose=1

/usr/bin/cc     -DCHECK_FUNCTION_EXISTS=daxpy_    CMakeFiles/cmTryCompileExec1328063564.dir/CheckFunctionExists.c.o  -o cmTryCompileExec1328

063564 -rdynamic -L/home/yashar/ATLAS3.10/lib -llapack -lf77blas -lm 

/home/yashar/ATLAS3.10/lib/libf77blas.a(ATL_F77wrap_daxpy.o): In function `atl_f77wrap_daxpy_':

ATL_F77wrap_axpy.c:(.text+0x24): undefined reference to `ATL_daxpy'

ATL_F77wrap_axpy.c:(.text+0x4b): undefined reference to `ATL_daxpy'

ATL_F77wrap_axpy.c:(.text+0x79): undefined reference to `ATL_daxpy'

ATL_F77wrap_axpy.c:(.text+0x90): undefined reference to `ATL_daxpy'

collect2: ld returned 1 exit status

gmake[1]: *** [cmTryCompileExec1328063564] Error 1

gmake[1]: Leaving directory `/home/yashar/Elemental/elemental/build/CMakeFiles/CMakeTmp'

gmake: *** [cmTryCompileExec1328063564/fast] Error 2

[Jack Poulson]

Hi Yashar,

You need to add -latlas at the very end of your link line, as it is apparently a dependency of f77blas. I highly recommend getting familiar with the 'nm' command if you are interested in how to resolve these issues in the future, e.g.,

    nm libatlas.a | grep ATL_daxpy
should return a line beginning with a 'T' if the ATL_daxpy symbol exists and is defined in that library. If the symbol exists but is undefined, then you will see said line, but beginning with a 'U' instead of a 'T'.

Jack

[Yashar Hezaveh]

Thanks for your help!

It works fine now.

Y