Add mkl_avx2 or mkl_def can fix it

[cy yan]

I compiler numpy with MKL, everything is ok. But I come across a peculiar question. I have three case:

•   case_1(perinfoMKL1): I add only `mkl_rt` lib in site.cfg file
  
•   case_2(perinfoMKL2): I add `mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt` lib in site.cfg file
  
•   case_3(perinfoMKL3): I and `mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt, mkl_avx2` in site.cfg file
  

And then, I compiler it, build and install process are ok, no problems. But when I use case_2, a error is occured: Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so. I setted LD_LIBRARY_PATH, but when I add mkl_avx2 libs and compiler it again after, test ok. 

Based on LD_DEBUG, I use LD_DEBUG="files libs" LD_DEBUG_OUTPUT=log ./test.py to display process for input file. I see relocation dependency words, so I **guess** the problem is related to this. I am not clear in underlying computer inspect, someone can explain it, thanks. I am confused of it a little.  Someone can explain it, thanks.

My understanding: Because mkl_rtaccording to relocation meaning, I guess that when group_1(mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5) and group_2(mkl_rt) exist together, executable program will find a new symbol table rather than specific one libs(such as: group_1)， and some aren't associated with the symbol table to link other symbol table. And then, finally executable program will not find a symbols, because function of libs relocataion again, so it's not find. When I add new lib to compiler again, errored symbols link new symbols successfully.

But according to Intel Forums, someone says that it's bug. So, I am confused of it again:-(                                https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/300857

Environments: 

  hp computer, CPU——Intel Core i5 4 core，Memory——8G, OS——CentOS 7, miniconda3(Python3.6), numpy 1.13(numpy with mkl, I configure some configuration MKL-information to site.cfg file) I compiled from source code.

  

Below followings are my process and attention points:

1. I seted LD_LIBRARY_PATH. My setting: LD_LIBRARY_PATH="/tmp/mkl-nfs/lib" (Since I will make my environment clean and tidy)
   
2.  Same test code: test.py  , and every test-code will be modified suitable Python interpreter path.
   
3. Compiler source code is ok for every case.
   

$ cat test.py

#!/home/yancy/miniconda3/envs/perinfoMKL1/bin/python

# -*- coding: utf-8 -*-

import numpy as np

import time

start_time = time.time()

a = 10 ** 4

A = np.random.random((a, a))

B = np.random.random((a, a))

C = A.dot(B)

print("Time: ", time.time() - start_time)

------------

(perinfoMKL1)$ tail numpy/site.cfg

#[fftw]

#libraries = fftw3

#

# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a .

#[djbfft]

#include_dirs = /usr/local/djbfft/include

#library_dirs = /usr/local/djbfft/lib

[mkl]

library_dirs = /tmp/mkl-nfs/lib

mkl_libs = mkl_rt

# No error, execute successfully

(perinfoMKL1) $ ./test.py

Time:  35.454288959503174

(perinfoMKL1)$ grep "libmkl_rt.so" log.27250

     27250:        file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL1/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]

    27250:     find library=libmkl_rt.so [0]; searching

    27250:       trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/tls/x86_64/libmkl_rt.so

    27250:      trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/tls/libmkl_rt.so

    27250:     trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/x86_64/libmkl_rt.so

    27250:          trying file=/home/yancy/miniconda3/envs/perinfoMKL1/lib/libmkl_rt.so

    27250:         trying file=/home/yancy/miniconda3/envs/perinfoMKL1/bin/../lib/libmkl_rt.so

    27250:          trying file=/tmp/mkl-nfs/lib/libmkl_rt.so

    27250:  file=libmkl_rt.so [0];  generating link map

    27250:  calling init: /tmp/mkl-nfs/lib/libmkl_rt.so

    27250:  file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]

    27250:   file=/tmp/mkl-nfs/lib/libiomp5.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]

    27250:      file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]

    27250:   file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0]

    27250:     calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]

(perinfoMKL1)]$ grep "libmkl_avx2.so" log.27250

     27250:      file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]

    27250: file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  generating link map

    27250:       file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    27250:     calling init: /tmp/mkl-nfs/lib/libmkl_avx2.so

    27250:        opening file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0]; direct_opencount=1

    27250: file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    27250:     calling fini: /tmp/mkl-nfs/lib/libmkl_avx2.so [0]

---------------

(perinfoMKL2) $ tail numpy/site.cfg

#[fftw]

#libraries = fftw3

#

# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a .

#[djbfft]

#include_dirs = /usr/local/djbfft/include

#library_dirs = /usr/local/djbfft/lib

[mkl]

library_dirs = /tmp/mkl-nfs/lib

mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt

(perinfoMKL2)$ ./test.py

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

# Error occur, LD_LIBRARY_PATH I setted.

(perinfoMKL2)$ ./test.py

Intel MKL FATAL ERROR: Cannot load libmkl_avx2.so or libmkl_def.so.

(perinfoMKL2)$ grep "libmkl_rt.so" test.log.26855

     26855:       file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL2/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]

    26855:     find library=libmkl_rt.so [0]; searching

    26855:       trying file=/home/yancy/miniconda3/envs/perinfoMKL2/lib/libmkl_rt.so

    26855:         trying file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/../lib/libmkl_rt.so

    26855:          trying file=/tmp/mkl-nfs/lib/libmkl_rt.so

    26855:  file=libmkl_rt.so [0];  generating link map

    26855:  file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)

    26855: file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)

    26855:       calling init: /tmp/mkl-nfs/lib/libmkl_rt.so

    26855:  calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]

(perinfoMKL2) $ grep "libmkl_avx2.so" test.log.26855

     26855: file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]

    26855: file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  generating link map

    26855:       /tmp/mkl-nfs/lib/libmkl_avx2.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

    26855:     file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  destroying link map

    26855:       file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]

    26855:      file=libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]

    26855:  find library=libmkl_avx2.so [0]; searching

    26855:     trying file=/home/yancy/miniconda3/envs/perinfoMKL2/lib/libmkl_avx2.so

    26855:       trying file=/home/yancy/miniconda3/envs/perinfoMKL2/bin/../lib/libmkl_avx2.so

    26855:        trying file=/tmp/mkl-nfs/lib/libmkl_avx2.so

    26855:        file=libmkl_avx2.so [0];  generating link map

    26855:        /tmp/mkl-nfs/lib/libmkl_avx2.so: error: symbol lookup error: undefined symbol: mkl_dft_fft_fix_twiddle_table_32f (fatal)

    26855:     file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  destroying link map

-------------

(perinfoMKL3)$ tail numpy/site.cfg

#[fftw]

#libraries = fftw3

#

# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a .

#[djbfft]

#include_dirs = /usr/local/djbfft/include

#library_dirs = /usr/local/djbfft/lib

[mkl]

library_dirs = /tmp/mkl-nfs/lib

mkl_libs = mkl_intel_lp64, mkl_intel_thread, mkl_core, iomp5, mkl_rt, mkl_avx2

# No error when I add mkl_avx2

(perinfoMKL3)$ ./test.py

Time:  33.996660232543945

(perinfoMKL3) $ grep "libmkl_rt.so" log.27384

     27384:       file=libmkl_rt.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL3/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]

    27384:     find library=libmkl_rt.so [0]; searching

    27384:       trying file=/home/yancy/miniconda3/envs/perinfoMKL3/lib/libmkl_rt.so

    27384:         trying file=/home/yancy/miniconda3/envs/perinfoMKL3/bin/../lib/libmkl_rt.so

    27384:          trying file=/tmp/mkl-nfs/lib/libmkl_rt.so

    27384:  file=libmkl_rt.so [0];  generating link map

    27384:  file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)

    27384: file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_rt.so [0] (relocation dependency)

    27384:       calling init: /tmp/mkl-nfs/lib/libmkl_rt.so

    27384:  calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]

(perinfoMKL3)$ grep "libmkl.avx2.so" log.27384

     27384:       file=libmkl_avx2.so [0];  needed by /home/yancy/miniconda3/envs/perinfoMKL3/lib/python3.6/site-packages/numpy-1.13.0.dev0+4408f74-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]

    27384:   find library=libmkl_avx2.so [0]; searching

    27384:     trying file=/home/yancy/miniconda3/envs/perinfoMKL3/lib/libmkl_avx2.so

    27384:       trying file=/home/yancy/miniconda3/envs/perinfoMKL3/bin/../lib/libmkl_avx2.so

    27384:        trying file=/tmp/mkl-nfs/lib/libmkl_avx2.so

    27384:        file=libmkl_avx2.so [0];  generating link map

    27384:        file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    27384:     file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    27384:       file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    27384:     calling init: /tmp/mkl-nfs/lib/libmkl_avx2.so

    27384:        opening file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0]; direct_opencount=1

    27384: calling fini: /tmp/mkl-nfs/lib/libmkl_avx2.so [0]

[Nehal J Wani]

I have faced this issue before, and the explanation at
https://software.intel.com/en-us/articles/a-new-linking-model-single-dynamic-library-mkl_rt-since-intel-mkl-103
helps a bit.

Cheers!

Nehal J Wani

> --
> You received this message because you are subscribed to the Google Groups
> "conda - Public" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to conda+un...@continuum.io.
> To post to this group, send email to co...@continuum.io.
> Visit this group at https://groups.google.com/a/continuum.io/group/conda/.
> To view this discussion on the web visit
> https://groups.google.com/a/continuum.io/d/msgid/conda/b478679e-1156-4173-8ef2-69f7039d561b%40continuum.io.
> For more options, visit https://groups.google.com/a/continuum.io/d/optout.

[cy yan]

@Nehal J Wani     I am very sorry for replying late:-( When I see reply message, I'm very happy that you can reply to me.

So, I have read this article. I have known mkl_rt is the wrapper of (interface, threading, and computational libraries)  https://software.intel.com/en-us/node/528514. In a word, I think the reason is mkl_rt exists with other libs, but this is a my guess. I don't know how did you solve this similar issues or other explanation. 

Based on mkl_rt lib, I simply do the test and find some another question:

$ tail numpy/site.cfg

#[fftw]

#libraries = fftw3

#

# For djbfft, numpy.distutils will look for either djbfft.a or libdjbfft.a .

#[djbfft]

#include_dirs = /usr/local/djbfft/include

#library_dirs = /usr/local/djbfft/lib

[mkl]

library_dirs = /tmp/mkl-nfs/lib

mkl_libs = mkl_rt

$ LD_DEBUG=files ./test.py 2> files.log

$ grep "libmkl_rt.so" files.log

     24311:      file=libmkl_rt.so [0];  needed by /home/yancy/.conda/envs/perinfoMKL/lib/python3.6/site-packages/numpy-1.13.0.dev0+1a9f43f-py3.6-linux-x86_64.egg/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so [0]                        

     24311: file=libmkl_rt.so [0];  generating link map

    24311:  calling init: /tmp/mkl-nfs/lib/libmkl_rt.so

    24311:  file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0] <==========

    24311:       file=/tmp/mkl-nfs/lib/libiomp5.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0] <==========

    24311:  file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0] <==========

    24311:       file=/tmp/mkl-nfs/lib/libmkl_intel_lp64.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_rt.so [0] <==========

    24311: calling fini: /tmp/mkl-nfs/lib/libmkl_rt.so [0]

$ grep "libmkl_avx2.so" files.log

     24311:    file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  dynamically loaded by /tmp/mkl-nfs/lib/libmkl_core.so [0]

    24311: file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0];  generating link map

    24311:       file=/tmp/mkl-nfs/lib/libmkl_intel_thread.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    24311:     file=/tmp/mkl-nfs/lib/libmkl_core.so [0];  needed by /tmp/mkl-nfs/lib/libmkl_avx2.so [0] (relocation dependency)

    24311:     calling init: /tmp/mkl-nfs/lib/libmkl_avx2.so

    24311:        opening file=/tmp/mkl-nfs/lib/libmkl_avx2.so [0]; direct_opencount=1

    24311: calling fini: /tmp/mkl-nfs/lib/libmkl_avx2.so [0]

Now, we can see interface, threading, computational and evenrun-time libraries (RTL) libraries all loaded by mkl_rt. So, I am confused it, this output is different from Intel OFFICIAL-DOC. 

I have google the similar issues, but no find a better answer:-( If you are involved in this knowledge, can you explain it. Sincerely thanks！

在 2017年4月28日星期五 UTC+8上午12:39:24，Nehal J Wani写道：

[Nehal J Wani]

The official documentation says, "The Single Dynamic Library (SDL)
automatically links interface, threading, and computational libraries
and thus simplifies linking".

If I understand correctly, mkl_rt.so uses dlopen() to dynamically open
the required shared libraries (libmkl_core.so, libiomp5.so,
libmkl_intel_thread.so, etc). So, what you see in the output for
LD_DEBUG is consistent with what the documentation says.

I solved my problem by linking my program against mkl_rt.so only and
let it handle the requirement for other shared objects dynamically.

Cheers!

Nehal J Wani

> https://groups.google.com/a/continuum.io/d/msgid/conda/40cc32d9-8001-47db-8ec2-cb0e76455fe6%40continuum.io.

[cy yan]

Here, Intel Official people makes a comment :  https://software.intel.com/en-us/forums/intel-math-kernel-library/topic/733207

Best !

在 2017年4月27日星期四 UTC+8下午4:07:49，cy yan写道：