Errors with Spark-0.8.1 and hadoop-yarn 2.2.0

[Izhar ul Hassan]

Hi,

I have a 3 node installation of hadoop 2.2.0 with yarn. I have installed spark-0.8.1 with support for spark enabled. I get the following errors when trying to run the examples:

SPARK_JAR=./assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop2.0.5-alpha.jar \
    ./spark-class org.apache.spark.deploy.yarn.Client \
      --jar examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar \
      --class org.apache.spark.examples.SparkPi \
      --args yarn-standalone \
      --num-workers 3 \
      --master-memory 4g \
      --worker-memory 2g \
      --worker-cores 1

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/deploy/yarn/Client
Caused by: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.Client
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
Could not find the main class: org.apache.spark.deploy.yarn.Client. Program will exit.

spark-0.8.0 with hadooop 2.0.5-alpha works fine.

/Izhar

[Patrick Wendell]

Could you write down the exact commands you are using to build the
spark assembly jar? 2.0.5-alpha is not the same as 2.2... (but in the
subject you listed 2.2) so I'm a bit confused which version you are
targeting here.

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Izhar ul Hassan]

Hi,

Sorry that was my mistake. I corrected it.