Hello,
I'm developing a project using BigDL 2.1.0 and Spark 3.1.2 on a desktop with Ubuntu Linux 22.04 installed. The network trains smoothly.
My desktop has 16 cores (8 real cores, I suppose). I'd like to use the MKL library to boost the training performance but I am not sure whether my code has utilized MKL effectively or not.
Here are my questions:
1. How do I know that the MKL lib has been used in my project or not?
I am using Scala and SBT to handle dependencies, two related libraries include:
"com.intel.analytics.bigdl" % "bigdl-dllib-spark_3.1.2" % "2.1.0",
"com.intel.analytics.bigdl.core.native.mkl" % "mkl-java-x86_64-linux" % "2.0.0"
Is the second lib redundant or is it necessary for MKL to work?
When the project starts, some logs read:
24-03-14 15:58:32 [main] INFO Engine$:121 - Auto detect executor number and executor cores number
24-03-14 15:58:32 [main] INFO Engine$:123 - Executor number is 1 and executor cores number is 8
24-03-14 15:58:33 [main] INFO ThreadPool$:95 - Set mkl threads to 1 on thread 1
2. It seems to me that the MKL lib is not used in my project. I tried to install the "intel-mkl" packages from Ubuntu repo. But I am not sure if this Ubuntu package is necessary or not.
3. Do we need to export some environment variables in the system for the MKL to work?
4. Is there any test program to know that we are effectively using MKL in our program?
In summary, I think that if there is documentation for MKL installation and usage with BigDL then it is very useful. I have not found such a document yet.
Thank you very much for your info.
Best,
Phuong Le-Hong