java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client when runing Spark on Yarn Sample

[Zhitao Yan]

Hi, All,

I want to run Spark on our Hadoop 2.0 Cluster(YARN). After building use source code, I follow the instruction from http://spark.incubator.apache.org/docs/latest/running-on-yarn.html . But when I run:

 SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar     ./spark-class org.apache.spark.deploy.yarn.Client       --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar       --class org.apache.spark.examples.SparkPi       --args yarn-standalone       --num-workers 3       --master-memory 4g       --worker-memory 2g       --worker-cores 1

I got following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client

Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client

        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)

        at java.security.AccessController.doPrivileged(Native Method)

        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)

        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)

Could not find the main class: org.apache.spark.deploy.yarn.Client.  Program will exit.

I checked my directory and can find the spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar file. 

[10.10.0.141:hadoop@lxnamenode02:/home/hadoop/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3]$ ll

total 173412

drwxrwxr-x 3 hadoop hadoop     4096 Dec 18 18:45 cache

-rw-rw-r-- 1 hadoop hadoop 89756622 Dec 18 19:29 spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar

-rw-rw-r-- 1 hadoop hadoop 87809175 Sep 17 16:40 spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar

[10.10.0.141:hadoop@lxnamenode02:/home/hadoop/spark-0.8.0-incubating-bin-cdh4/assembly/target/scala-2.9.3]$

So what's the possible reason that I can not run the sample? 

[Zhitao Yan]

The problem was resolved after checking /bin/compute-classpath.sh. There are 2 jars under ./assembly/target/scala-2.9.3, one is spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar , the other is: spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar , in compute-classpath.sh, we can see:

CLASSPATH="$SPARK_CLASSPATH:$FWDIR/conf"

if [ -f "$FWDIR/RELEASE" ]; then

  ASSEMBLY_JAR=`ls "$FWDIR"/jars/spark-assembly*.jar`

else

  ASSEMBLY_JAR=`ls "$FWDIR"/assembly/target/scala-$SCALA_VERSION/spark-assembly*hadoop*.jar`

fi

it generate wrong CLASSPATH which has a tab between the 2 jars.  I moved the spark-assembly_2.9.3-0.8.0-incubating-hadoop2.0.0-mr1-cdh4.2.0.jar and it can skip the exception. But I got a new exception now:

Exception in thread "main" java.lang.reflect.UndeclaredThrowableException

        at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)

        at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:149)

        at org.apache.hadoop.yarn.client.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:173)

        at org.apache.spark.deploy.yarn.Client.logClusterResourceDetails(Client.scala:76)

        at org.apache.spark.deploy.yarn.Client.run(Client.scala:53)

        at org.apache.spark.deploy.yarn.Client$.main(Client.scala:328)

        at org.apache.spark.deploy.yarn.Client.main(Client.scala)

Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "lxdatanode02/10.10.0.144"; destination host is: "lxjournalnode":8032; 

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:212)

        at $Proxy7.getClusterMetrics(Unknown Source)

        at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:146)

        ... 5 more

Caused by: java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "lxdatanode02/10.10.0.144"; destination host is: "lxjournalnode":8032; 

        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:761)

        at org.apache.hadoop.ipc.Client.call(Client.java:1239)

        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202)

        ... 7 more

Caused by: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status

        at com.google.protobuf.UninitializedMessageException.asInvalidProtocolBufferException(UninitializedMessageException.java:81)

        at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.buildParsed(RpcPayloadHeaderProtos.java:1094)

        at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto$Builder.access$1300(RpcPayloadHeaderProtos.java:1028)

        at org.apache.hadoop.ipc.protobuf.RpcPayloadHeaderProtos$RpcResponseHeaderProto.parseDelimitedFrom(RpcPayloadHeaderProtos.java:986)

        at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:946)

        at org.apache.hadoop.ipc.Client$Connection.run(Client.java:844)

在 2013年12月18日星期三UTC+8下午9时48分40秒，Zhitao Yan写道：

[Matei Zaharia]

It’s unfortunate that the launch script doesn’t warn you about this, but this happened because you built Spark twice, for different Hadoop versions, without doing “sbt clean” in-between. Just delete the old JAR by hand.

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Zhitao Yan]

Thanks for the reply. The second issue also has been resolved by rebuild use the same version as our envrionment:

Hadoop 2.0.0-cdh4.3.0

在 2013年12月20日星期五UTC+8上午2时36分12秒，Matei Zaharia写道：