A problem of initializing Graph in Spark program

[Dingfei Zhang]

Hi,

I'm new to scala-graph and scala. I used scala to write below simple Spark codes, and tried to build a graph using scala-graph library.

package org.apache.spark.examples
import org.apache.spark.SparkContext._
import org.apache.spark.{SparkConf, SparkContext}
import scalax.collection.Graph
import scalax.collection.GraphEdge.DiEdge
 
object Test{
  def main(args: Array[String]) {
    val sparkConf = new SparkConf().setAppName("Test")
    val ctx = new SparkContext(sparkConf)
 
    val g = Graph[String, DiEdge]("string")
 
    ctx.stop()
  }
}

when running the code on Spark platform (the compiling is ok), I met below errors.

$ spark-submit --master spark://localhost.localdomain:7077   --class org.apache.spark.examples.Test  ~/../test/bin/Test.jar
Exception in thread "main" java.lang.NoClassDefFoundError: scalax/collection/generic/GraphCoreCompanion
        at org.apache.spark.examples.RouteChecker.main(RouteChecker.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:328)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: scalax.collection.generic.GraphCoreCompanion
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
        ... 8 more

But when I commented out the graph initialization code  "val g = Graph[String, DiEdge]("string") " , the compiling and running are both OK.

Does anyone know how to solve this problem? thank you very much.

Best Regards,

Dingfei 

[Peter Empen]

Did you also try to comment out spark-related code?

Please make sure to refer to the right Scala version and dependencies.

[Dingfei Zhang]

hi Peter,

thanks for your reply. yes, this problem turns out to be a incorrect Scala version issue. After updating my Scala, this program works fine. thanks.

Best regards,

Dingfei

在 2014年11月19日星期三UTC+8上午4时33分40秒，Peter Empen写道：