Help on building Julia with Intel MKL on Windows?

[Zhong Pan]

Anybody knows how to build Julia with Intel MKL on Windows? 

I found the article below but it is for Linux.

https://software.intel.com/en-us/articles/julia-with-intel-mkl-for-improved-performance

Thanks!

[Chris Rackauckas]

You just do what it says here: https://github.com/JuliaLang/julia. Then you can replace a lot of the functions using VML.jl

[Zhong Pan]

Thanks! I am still a bit confused about this part:

To build Julia for Windows, this page says I need to use MinGW compiler either under MSYS2 or Cygwin: https://github.com/JuliaLang/julia/blob/master/README.windows.md

However, to build Julia with MKL BLAS and LAPACK libraries, the first link you referenced says I need to use Intel Compiler 15 or above. The example given is for POSIX system.

So, can I directly use the Windows version of Intel Compiler to compile Julia with the options turned on to use MKL, and forget about MinGW completely?

Thanks for leading me to VML.jl. I assume this module is independent of MKL BLAS and LAPACK, so even if I didn't link Julia with MKL, as long as I have MKL binary installed, this would work?

[Tony Kelman]

Intel compilers on windows are MSVC style, which our build system is not really set up to handle. There is experimental partial support (search for "MSVC support tracking issue" if you're interested) but it would really require rewriting the build system to use cmake to work smoothly.

You can build Julia from source with mingw but direct the build system to use an mkl library instead of building openblas from source. At one point mkl on windows didn't export gfortran style naming conventions for blas and lapack functions, but recent versions do last I checked. Could even be able to switch from an existing Julia binary by rebuilding the system image.

[Zhong Pan]

Thank you Tony for the information. I still had some problem.

I tried building Julia (release-0.5) using MinGW following the instructions here: 

https://github.com/JuliaLang/julia/blob/master/README.windows.md

The only difference is, after creating the Make.user file, I edited it using a text editor and added the options to use MKL, so the complete Make.user now looks like:

override CMAKE=/C/CMake/bin/cmake.exe

USE_INTEL_MKL = 1

USE_INTEL_MKL_FFT = 1

USE_INTEL_LIBM = 1

The addition of the above lines is per build instructions about MKL at https://github.com/JuliaLang/julia, except that I removed "USEICC = 1" and "USEIFC = 1" since I am not using Intel compiler. But I do have Intel MKL with the free Community License installed on my computer at "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl"

After I started the build with "make -j 3", the build would run for a while then stop with 

make: *** [Makefile:81: julia-deps] Error 2

I went through the messages and found this to be the most suspicious for a root cause:

c:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/../lib/gcc/x86_64-w64-mingw32/6.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lmkl_rt

So I realized that MinGW was not able to find the MKL libraries. I tried the following 2:

(1) Copied all the 17 .lib files from "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\lib\intel64_win" to "C:\mkllib" (so the path does not contain a space), then I did

$ export MKLLIB=/C/mkllib

(2) Copied all the 17 .lib files to "C:\Users\zpan\Documents\GitHub\julia\usr\lib"

But I was still getting the same Error 2 in the end, and I could still find the error message saying "cannot find -lmkl_rt".  Well, mkl_rt.lib was indeed part of the 17 files, which are:

mkl_blas95_ilp64.lib

mkl_blas95_lp64.lib

mkl_core.lib

mkl_core_dll.lib

mkl_intel_ilp64.lib

mkl_intel_ilp64_dll.li

mkl_intel_lp64.lib

mkl_intel_lp64_dll.lib

mkl_intel_thread.lib

mkl_intel_thread_dll.l

mkl_lapack95_ilp64.lib

mkl_lapack95_lp64.lib

mkl_rt.lib

mkl_sequential.lib

mkl_sequential_dll.lib

mkl_tbb_thread.lib

mkl_tbb_thread_dll.lib

I am currently trying the command "make install" to build it without parallel option - this is very slow but I can see the messages in proper order.

Did I miss something to let the linker see the .lib files? Any help greatly appreciated! 

[Zhong Pan]

OK my non-parallel build bumped into a different problem. It seems the CMAKE_C_COMPILER PATH was not understood. 

I am not sure what is the correct one, so I tried a few variations:

C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe

/C/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc

/C/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe

For example, I tried modifying CMakeCache.txt by modifying this line:

CMAKE_C_COMPILER:UNINITIALIZED=C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc

into

CMAKE_C_COMPILER:UNINITIALIZED=C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe

(and also tried the other 2 variations)

Or creating a new line:

CMAKE_C_COMPILER=C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe

(and also tried the other 2 variations)

Problem is everytime I run "make" or "make install", CMakeCache.txt is overwritten.

I also tried the environmental variable way:

$ export CC=/C/Users/zpan/Documents/GitHub/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc

(and also tried the other 2 variations)

None of the solutions worked. So if I do "make" or "make install" I am stuck at this error message. 

I am not experienced in building projects with MinGW, so maybe the problem is a trivial one. But, for me it's tough. :-) Any help appreciated!

See full error message below:

zpan@WSPWork MSYS /c/users/zpan/documents/github/julia

$ make

-- The C compiler identification is GNU 6.2.0

CMake Error at CMakeLists.txt:2 (project):

  The CMAKE_C_COMPILER:

    C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc

  is not a full path to an existing compiler tool.

  Tell CMake where to find the compiler by setting either the environment

  variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to

  the compiler, or to the compiler name if it is in the PATH.

-- Configuring incomplete, errors occurred!

See also "C:/Users/zpan/Documents/GitHub/julia/deps/build/mbedtls-2.3.0/CMakeFiles/CMakeOutput.log".

make[1]: *** [/c/users/zpan/documents/github/julia/deps/mbedtls.mk:40: build/mbedtls-2.3.0/Makefile] Error 1

[Tony Kelman]

maybe better to just temporarily add gcc's location to your working path for the duration of the make (don't leave it there longer though).

I think that linker error is from arpack. Try making a copy of mkl_rt.dll called libmkl_rt.dll and see if that helps. libtool isn't able to link against dlls that don't have a lib prefix unless you have a corresponding .dll.a import library (maybe copying and renaming one od the .lib files to libmkl_rt.dll.a could also work?) because being annoying is what libtool does.

[Zhong Pan]

I realize I may be documenting a bunch of build issues that more experienced programmer may overcome more easily. Anyway since I started, please bear with me. :-)

So I solved the problem of incorrect CMAKE_C_COMPILER. What I did was I found the make file that was causing Error 1:

/c/users/zpan/documents/github/julia/deps/mbedtls.mk

I opened it, scrolled to line 40, and inserted a line before line 38:

MBEDTLS_OPTS += -DCMAKE_C_COMPILER="/C/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe"

After saving this and trying "make install" again, the compiler seems to be found, but another issue popped up, which I cannot solve yet. 

The error message:

zpan@WSPWork MSYS /c/users/zpan/documents/github/julia

$ make install

-- The C compiler identification is GNU 6.2.0

-- Check for working C compiler: C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe

-- Check for working C compiler: C:/users/zpan/documents/github/julia/usr/x86_64-w64-mingw32/sys-root/mingw/bin/gcc.exe -- works

-- Detecting C compiler ABI info

-- Detecting C compiler ABI info - done

-- Detecting C compile features

-- Detecting C compile features - done

-- Found Perl: C:/msys64/usr/bin/perl.exe (found version "5.22.1")

Can't open perl script "scripts/config.pl": No such file or directory

CMake Error at tests/CMakeLists.txt:122 (message):

  Could not create symbolic link for:

  C:/Users/zpan/Documents/GitHub/julia/deps/srccache/mbedtls-2.3.0-gpl/tests/data_files

  --> Invalid switch - "Users".

-- Configuring incomplete, errors occurred!

See also "C:/Users/zpan/Documents/GitHub/julia/deps/build/mbedtls-2.3.0/CMakeFiles/CMakeOutput.log".

make[2]: *** [/c/users/zpan/documents/github/julia/deps/mbedtls.mk:41: build/mbedtls-2.3.0/Makefile] Error 1

make[1]: *** [Makefile:81: julia-deps] Error 2

make: *** [Makefile:331: install] Error 2

[Zhong Pan]

Tony,

Thanks for your reply. I did solve the CMAKE_C_COMPILER issue - see my previous post (just 1 min after yours). However, it's not by appending to $PATH though - I had tried adding the gcc path to $PATH and the same error still showed up.

Thanks for the hint about adding "lib" prefix. I don't have .dll files though - the MKL library consists of 17 .lib files (the file list I gave missed a few letters at the end for some files so they become .l or .li - my apologies). 

I will try adding the prefix anyway... After other road blocks are removed. Please see my previous post for details.

[Zhong Pan]

The "Can't open perl script "scripts/config.pl": No such file or directory" error has been solved. See: https://github.com/ARMmbed/mbedtls/issues/541

What I did was to open the CMakeList.txt file under "C:\Users\zpan\Documents\GitHub\julia\deps\srccache\mbedtls-2.3.0-gpl", and replaced ${PERL_EXECUTABLE} with ${CMAKE_CURRENT_SOURCE_DIR} on line 34.

 

Trying to solve the "Could not create symbolic link" issue.

[Zhong Pan]

Just to report my progress and ask for help again if anybody succeeded in building Julia 0.4 or 0.5 on Windows with MKL.

I backed off a little and tried building release-0.5 on Windows following the standard instructions below. That is, without any MKL related configurations.

https://github.com/JuliaLang/julia/blob/master/README.windows.md

But the build failed with Error 2 (some dependency failed to build). 

Since my main focus is to get MKL to work rather than trying out 0.5, I tried building release-0.4 first following the same instructions (no MKL). And that was a success!

Encouraged, I added the following lines to Make.user and copied the 17 .lib files from the MKL library to julia/usr/bin and julia/usr/lib and C:\mkllib

USE_INTEL_MKL = 1

USE_INTEL_MKL_FFT = 1

USE_INTEL_LIBM = 1

I also did this, though I know it probably won't matter:

export MKLLIB=/c/mkllib

"make -j 4" failed with the expected error:

...

configure: error: Cannot find BLAS libraries

...

make: *** [Makefile:51: julia-deps] Error 2

I know CMake probably won't be able to figure out Windows style libraries by itself, but I don't know how to make this work. Could anyone help? 

P.S. the 17 MKL .lib files are listed below:

mkl_blas95_ilp64.lib

mkl_blas95_lp64.lib

mkl_core.lib

mkl_core_dll.lib

mkl_intel_ilp64.lib

mkl_intel_ilp64_dll.lib

mkl_intel_lp64.lib

mkl_intel_lp64_dll.lib

mkl_intel_thread.lib

mkl_intel_thread_dll.lib

[Zhong Pan]

BTW I did try Tony's idea of adding "lib" as prefix to each .lib filename. That didn't solve the problem.

[Tony Kelman]

Which configure file was this from? Julia doesn't need the .lib files, it needs the dll, and I assure you there are dlls in MKL. However some of the dependencies may need the .lib files, and/or you can try copying them to a .dll.a file name if that helps libtool or configure work better.

[Zhong Pan]

Thanks, Tony, with your help I think I am pretty close.

Yes you are right, I searched for *.dll and found them in a folder that I didn't notice before.

Here's what I did last based on your comments to move things forward a few steps more:

(1) Copied all the 17 .lib files from "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\lib\intel64_win" to "C:\Users\zpan\Documents\GitHub\julia\usr\x86_64-w64-mingw32\sys-root\mingw\lib". Changed them from .lib files to .a files and added prefix "lib". So now the 17 files look like:

libmkl_blas95_ilp64.a

libmkl_blas95_lp64.a

libmkl_core.a

libmkl_core_dll.a

libmkl_intel_ilp64.a

libmkl_intel_ilp64_dll.a

libmkl_intel_lp64.a

libmkl_intel_lp64_dll.a

libmkl_intel_thread.a

libmkl_intel_thread_dll.a

libmkl_lapack95_ilp64.a

libmkl_lapack95_lp64.a

libmkl_rt.a

libmkl_sequential.a

libmkl_sequential_dll.a

libmkl_tbb_thread.a

libmkl_tbb_thread_dll.a

(2) Copied all the .dll files from "C:\Program Files (x86)\IntelSWTools\compilers_and_libraries_2017.0.109\windows\redist\intel64_win\mkl" to "C:\Users\zpan\Documents\GitHub\julia\usr\x86_64-w64-mingw32\sys-root\mingw\bin". This includes the folder "33". Here are all the files that got copied:

1033/mkl_msg.dll

libimalloc.dll

mkl_avx.dll

mkl_avx2.dll

mkl_avx512.dll

mkl_avx512_mic.dll

mkl_core.dll

mkl_def.dll

mkl_intel_thread.dll

mkl_mc.dll

mkl_mc3.dll

mkl_rt.dll

mkl_sequential.dll

mkl_tbb_thread.dll

mkl_vml_avx.dll

mkl_vml_avx2.dll

mkl_vml_avx512.dll

mkl_vml_avx512_mic.dll

mkl_vml_cmpt.dll

mkl_vml_def.dll

mkl_vml_mc.dll

mkl_vml_mc2.dll

mkl_vml_mc3.dll

(3) "make -j 4" again. This time, there was no "Cannot find BLAS libraries" error anymore. The build went on for a while until here:

floatfuncs.jl

Please submit a bug report with steps to reproduce this fault, and any error messages that follow (in their entirety). Thanks.

Exception: EXCEPTION_ACCESS_VIOLATION at 0x0 -- unknown function (ip: 0000000000000000)

unknown function (ip: 0000000000000000)

*** This error is usually fixed by running `make clean`. If the error persists, try `make cleanall`. ***

make[1]: *** [Makefile:200: /c/users/zpan/Documents/GitHub/julia/usr/lib/julia/sys.o] Error 1

make: *** [Makefile:69: julia-sysimg-release] Error 2

I tried both "make clean" and "make cleanall", and the same problem persisted. I also tried renaming the .dll and .a files in a couple ways in vain. 

Again, any help appreciated!! Hope I am close to success. When I am done I will repeat the process and document it here for others to see.

[Tony Kelman]

Something's broken, you'll probably have to run that in gdb and try to get a backtrace. I don't think the .lib files are full-fledged static libraries (how big are they?), I suspect they are import libraries that go along with the corresponding dll's. So the mingw naming convention from them would be libmkl_rt.dll.a for mkl_rt.lib. The main thing you want to do is make sure the blas and lapack library ccall's are getting pointed to the mkl_rt.dll.

[Zhong Pan]

Hi Tony thanks for your reply. Sorry I got distracted by other things.

Sorry tracing down an access violation using gdb is beyond my skills. I'll see if I can understand the make files first as I never used them (yes, you are talking to a .NET, Matlab and Python "programmer" who usually doesn't build C++ projects :-)).

As to the file sizes, I think the attached shows the whole picture the best. 

Thanks again for your help. It's very important for me to make Intel MKL work with Julia on Windows before I can promote Julia to my colleagues - they are mostly Matlab users while a few started using Anaconda Python. They take it for granted that MKL is there. Yes Matlab costs $$$ but it's company money and IT handles the license server etc., so for engineers what work with reasonable effort is the most important. I'd love to see them pick up Julia instead of Python so we start the move from Matlab to open source on the right track early on (not to say Python is not good - it's great! But I think Julia is better).

[Zhong Pan]

BTW I have re-installed Intel MKL in a folder without any spaces in the path name, so you may notice the location changed. Not sure if it helps, just trying.

[Zhong Pan]

I completely restarted the build process from fresh (git clone). In addition to the renaming and copying .lib and .dll files, I also did some many configuration of environmental variables before the build. 

These are mostly based on what C:\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\bin\mklvars.bat would have done.

export MKLROOT=/c/IntelSWTools/compilers_and_libraries_2017.0.109/windows/mkl

export MKL_TARGET_ARCH=intel64

export REDIST=/c/IntelSWTools/compilers_and_libraries_2017.0.109/windows/redist

export MKL_LP64_ILP64=lp64

export LIB=/c/IntelSWTools/compilers_and_libraries_2017.0.109/windows/tbb/lib/intel64/vc_mt:/C/IntelSWTools/compilers_and_libraries_2017.0.109/windows/mkl/lib/intel64:/C/IntelSWTools/compilers_and_libraries_2017.0.109/windows/compiler/lib/intel64

export INCLUDE=/C/IntelSWTools/compilers_and_libraries_2017.0.109/windows/mkl/include

Now I am stuck at a different place:

Making install in SVD

Making install in SYM

libtool: compile:  gcc -m64 -DTIME_WITH_SYS_TIME=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_LOCALE_H=1 -DHAVE_WCHAR_H=1 -DHAVE_STDARG=1 -DHAVE_SYS_TIME_H=1 -DHAVE_STDINT_H=1 -DHAVE_VA_COPY=1 -DHAVE_SETLOCALE=1 -DHAVE_GETTIMEOFDAY=1 -DHAVE_LONG_LONG=1 -DHAVE_INTMAX_T=1 -DMPFR_HAVE_INTMAX_MAX=1 -DMPFR_HAVE_FESETROUND=1 -DHAVE_DENORMS=1 -DHAVE_ROUND=1 -DHAVE_TRUNC=1 -DHAVE_FLOOR=1 -DHAVE_CEIL=1 -DHAVE_NEARBYINT=1 -DHAVE_LDOUBLE_IEEE_EXT_LITTLE=1 -DLT_OBJDIR=\".libs/\" -DHAVE_ATTRIBUTE_MODE=1 -I. -I../src -I../src -DSRCDIR=\".\" -I/c/users/zpan/Documents/GitHub/julia/usr/include -DNPRINTF_L -DNPRINTF_T -DNPRINTF_J -MT random2.lo -MD -MP -MF .deps/random2.Tpo -c random2.c  -DDLL_EXPORT -DPIC -o .libs/random2.o

mv: cannot stat '/c/users/zpan/Documents/GitHub/julia/usr/bin/libarpack-2.dll': No such file or directory

make[1]: *** [Makefile:1331: /c/users/zpan/Documents/GitHub/julia/usr/bin/libarpack.dll] Error 1

make[1]: *** Waiting for unfinished jobs....

I cannot find libarpack-2.dll or libarpack.dll anywhere.

BTW, I bumped into a folder under MKL installation: C:\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\lib\intel64_win_mic

The files inside seems to be Linux-like library files, but I don't know how they should be used:

zpan@WSPWORK C:\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\lib\intel64_win_mic

> dir

 Volume in drive C is Windows7_OS

 Volume Serial Number is 4131-0400

 Directory of C:\IntelSWTools\compilers_and_libraries_2017.0.109\windows\mkl\lib\intel64_win_mic

09/16/2016  15:02    <DIR>          .

09/16/2016  15:02    <DIR>          ..

08/01/2016  18:36        19,309,845 libmkl_ao_worker.so

08/01/2016  18:30         2,119,960 libmkl_blas95_ilp64.a

08/01/2016  18:30         2,147,752 libmkl_blas95_lp64.a

08/01/2016  18:34       153,164,198 libmkl_core.a

08/01/2016  18:36       122,863,997 libmkl_core.so

08/01/2016  18:34        24,628,804 libmkl_intel_ilp64.a

08/01/2016  18:36        10,850,199 libmkl_intel_ilp64.so

08/01/2016  18:34        26,277,006 libmkl_intel_lp64.a

08/01/2016  18:36        12,012,951 libmkl_intel_lp64.so

08/01/2016  18:34        52,875,578 libmkl_intel_thread.a

08/01/2016  18:36        40,692,525 libmkl_intel_thread.so

08/01/2016  18:32        25,226,176 libmkl_lapack95_ilp64.a

08/01/2016  18:31        25,654,272 libmkl_lapack95_lp64.a

08/01/2016  18:36         5,128,544 libmkl_rt.so

08/01/2016  18:34        30,265,120 libmkl_sequential.a

08/01/2016  18:36        24,301,227 libmkl_sequential.so

08/01/2016  18:34        32,158,354 libmkl_tbb_thread.a

08/01/2016  18:36        25,593,814 libmkl_tbb_thread.so

09/16/2016  15:02    <DIR>          locale

              18 File(s)    635,270,322 bytes

               3 Dir(s)  31,684,468,736 bytes free

[Tony Kelman]

mic probably means xeon phi, so that's likely for cross compiling to a xeon phi device from a windows host. Based on the sizes of the .lib files, I take what I said back, those probably are static libraries. I did this many months ago and have a Make.user somewhere.

Before you spend too much time on this, how sure are you that MKL will make a major difference over openblas on your application? openblas is close in performance to MKl on many, but not quite all operations. In some cases it may even be faster. Open source Julia is much better supported against openblas, especially on Windows. There are avenues for commercial supported Julia as well.

[Zhong Pan]

I saw benchmarks showing impressive speedup using Anaconda + MKL vs. open source library. For example, https://github.com/ContinuumIO/mkl-optimizations-benchmarks

For my application, a representative problem would be split-step Fourier method (https://en.wikipedia.org/wiki/Split-step_method). This would involve multiple arrays (e.g. 8), each containing tens of thousands to hundreds of thousands of complex values, going through FFT - array calculation - inverse FFT - array calculation... repeated many many times. There could be dozens of parameter combinations to run the calculation on, which could be parallelized. Currently the calculation is done in Matlab with some use of GPU, which is reasonably fast but not fast enough. Python would be slow for this situation as each step depends on previous step in a nonlinear way so the loop cannot be vectorized. I was hoping to convert the Matlab project into Julia to gain significant speedup (fast Julia base + MKL + GPU) on a single 6-core machine. If then I can get a few machines to work together to test different parameter sets that'd be even better - realistic limitation is we have to stick with Windows and won't have a real cluster.

Based on the instructions, building Julia with MKL seems to be easy on Linux. I didn't realize it could be this hard for Windows, at least for me.

Commercial distribution of Julia could be an option if the license fee is low (people got spoiled by Anaconda). However, I don't see one that supports MKL yet. 

[Tony Kelman]

That benchmark doesn't say what they were using for "normal backends" - was it openblas or atlas or the reference blas, which set of kernels, were they using multithreading, etc. The open source Julia build will almost certainly have faster FFT's than SciPy without MKL, since Julia uses FFTW but SciPy uses a slower non-GPL-licensed implementation. Note that we had a bug on our buildbots regarding optimization flags in some dependencies so you may want to wait until 0.4.7 binaries or test with 0.5.0-rc4 or newer for comparing FFT performance.

Julia Computing offers custom solutions that aren't necessarily listed on the products page of our website yet, you can inquire at the contact address there.

[Zhong Pan]

Thanks for the information. I will install 0.5.0-rc4 binary and try a small benchmark among Julia, Matlab (forgot which version I have probably 2013), also Anaconda Python 2.7 with and without MKL. I will avoid loops in doing so.

Will drop an email regarding commercial solution over at Julia Computing.