Error after including Alluxio client into Spark's classpath

[Dima Fadeyev]

Hello, everyone,

I'm trying to setup Alluxio 1.0.0 with Spark 1.6.1. After adding this line to spark-env.sh

export SPARK_CLASSPATH=/opt/alluxio/core/client/target/alluxio-core-client-1.0.0-jar-with-dependencies.jar:$SPARK_CLASSPATH

I'm getting this exception when launching any Spark job: 

spark-submit  --class org.apache.spark.examples.SparkPi --master yarn /opt/spark/lib/spark-examples-1.6.1-hadoop2.7.2.jar 10
16/03/19 16:25:59 ERROR actor.ActorSystemImpl: Uncaught fatal error from thread [sparkDriverActorSystem-akka.remote.default-remote-dispatcher-8] shutting down ActorSystem [sparkDriverActorSystem]
java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker signature: (Ljava/util/concurrent/Executor;)Lorg/jboss/netty/channel/socket/nio/AbstractNioWorker;) Wrong return type in function
    at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:283)
    at akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:240)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
    at scala.util.Try$.apply(Try.scala:161)
    at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
    at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
    at akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
    at scala.util.Success.flatMap(Try.scala:200)
    at akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
    at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:711)
    at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:703)
    at scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
    at scala.collection.Iterator$class.foreach(Iterator.scala:727)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
    at akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:703)
    at akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:491)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:467)
    at akka.remote.EndpointManager.aroundReceive(Remoting.scala:394)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
    at akka.actor.ActorCell.invoke(ActorCell.scala:487)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
    at akka.dispatch.Mailbox.run(Mailbox.scala:220)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Does anyone know what is the cause of this issue? Please, help me resolve it.

[Jiří Šimša]

Hi Dima,

Can you try to build Alluxio client jar with -Pspark and see if that resolves the issue? (I believe you are encountering the same problem as here).

Best,

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jiří Šimša

Alluxio, Inc.