java.lang.NoClassDefFoundError: com/intel/analytics/bigdl/utils/Engine

[Huijie Lu]

Hi there,

I'm trying to run a scala program using bigdl on yarn cluster mode, but it gives me the following error:

22/05/04 21:37:36 ERROR ApplicationMaster: User class threw exception: java.lang.NoClassDefFoundError: com/intel/analytics/bigdl/utils/Engine$ java.lang.NoClassDefFoundError: com/intel/analytics/bigdl/utils/Engine$    at com.conversantmedia.ds.recommender.RecommenderDL$.<init>(RecommenderDL.scala:7)    at com.conversantmedia.ds.recommender.RecommenderDL$.<clinit>(RecommenderDL.scala)    at com.conversantmedia.ds.recommender.RecommenderDL.main(RecommenderDL.scala)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) Caused by: java.lang.ClassNotFoundException: com.intel.analytics.bigdl.utils.Engine$    at java.net.URLClassLoader.findClass(URLClassLoader.java:387)    at java.lang.ClassLoader.loadClass(ClassLoader.java:418)    at java.lang.ClassLoader.loadClass(ClassLoader.java:351)    ... 8 more

Any idea about what caused this and how to fix it?

Thank you!

Huijie 

[Qiyuan Gong]

Hi Huijie,

     Thank you for feedback! Also, sorry for the inconvenience. :)

     According to error message, it seems that "com/intel/analytics/bigdl/utils/Engine" can not be found. In most case, it is caused by bigdl related jars cannot be found in classpath.
     
     A quick fixes for this issue:
     1. Using bigdl submit instead of spark submit. This script will add bigdl jars into classpath automatically. https://github.com/intel-analytics/BigDL/blob/main/scripts/spark-submit-with-bigdl

     2. Or add bigdl related jars into classpath in your submit command. https://spark.apache.org/docs/latest/running-on-yarn.html#adding-other-jars 
     3. Or package all dependencies (including bigdl jars) into xxxx-with-dependency.jar.  Just like https://stackoverflow.com/questions/574594/how-can-i-create-an-executable-jar-with-dependencies-using-maven 

      Let's me know if there is any problem. 

Have a nice day!

Qiyuan

[Huijie Lu]

Hi Qiyuan,

Thank you for your reply!

But I still have trouble running the job. I have an existing scala project called com.conversantmedia.ds.recommender. I created a new object called RecommenderDL under the package as com.conversantmedia.ds.recommender.RecommenderDL, where I want to apply the BigDL libraries to build a deep learning model. 

I'm using the sbt management tool. The BigDL library has been imported in build.sbt as 

libraryDependencies += "com.intel.analytics.bigdl" % "bigdl-SPARK_2.4" % "0.12.2" % "provided"

Then I did "sbt clean assembly" to generate a jar file called recommender_system.jar, which should include all the bigdl dependencies, right? Why do you say I should add the bigdl jars into class path? Should there be an independent jar file for bigdl? 

Thank you!

Huijie 

[Qiyuan Gong]

Hi Huijie,

     Sorry for late reply.

     In maven/sbt, "provided" means this dependency will not be packaged into assembled/fat jar.  (https://www.baeldung.com/scala/sbt-fat-jar)

     In that case, this dependency should be prepared (e.g., default classpath) in container or other runtime env. :)

     

     Please change or remove "provided" in build.sbt.

Have a nice day!

Qiyuan

[Huijie Lu]

I removed "provided" from the library dependency, and it works now. Thank you very much!