Running spark job on YARN

[Som Satapathy]

Hi All,

I am trying to run spark jobs over YARN on a single node setup. I am referring to http://www.spark-project.org/docs/0.7.2/running-on-yarn.html for that. 

I am passing 'standalone' as my master URL. But strangely my spark job always shows as 'RUNNING' in my YARN resourcemanager console. When I dig into the log, I see the following error:

13/06/20 19:01:45 WARN spark.SparkContext: Master standalone does not match expected format, parsing as Mesos URL
Failed to load native Mesos library from .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Exception in thread "Thread-3" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:154)
Caused by: java.lang.UnsatisfiedLinkError: no mesos in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758)
	at java.lang.Runtime.loadLibrary0(Runtime.java:823)
	at java.lang.System.loadLibrary(System.java:1045)
	at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:46)
	at spark.SparkContext.<init>(SparkContext.scala:201)
	at spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:38)
	at spark.examples.JavaSparkPi.main(JavaSparkPi.java:21)
	... 5 more

When I browsed through the user group for similar errors, I found an email mentioning about the need to set MESOS_NATIVE_LIBRARY path in the spark-env.sh. Does this mean I need to install mesos as well to run spark jobs on YARN? 

Looking forward for your inputs.

Thanks,

Som

[Som Satapathy]

I found an email in the user group mentioning about the need to set 

MESOS_NATIVE_LIBRARY path in the spark-env.sh.

Does this mean I need to install mesos as well to run spark jobs on YARN? 

Thanks,

Som

[Mridul Muralidharan]

Yarn support works fine only in master (and will be in 0.8.*) - with
secure hadoop integration currently submitted by Tom as a PR.
0.7 (and earlier) have some support which is pretty buggy ...


Regards,
Mridul

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Som Satapathy]

I am using 0.8.0 version of spark, but it's been a smooth experience so far running a spark job over yarn (yarn version is cdh4.3.0). Still stuck with the same problem.

Would love to hear inputs from folks who have managed to get spark going over yarn.

Thanks,

Som

[Som Satapathy]

I am using 0.8.0 version of spark, but it's *not* been a smooth experience so far running a spark job over yarn (yarn version is cdh4.3.0). Still stuck with the same problem.

[Mridul Muralidharan]

Did you use the instructions for running spark on yarn from 0.8 ?
Then you should not be seeing this error : mesos should not be
initialized at all in that case : since you need to specify
'yarn-standalone'.

If it is a documentation issue, that would need to be resolved ofcourse.

- Mridul

[Som Satapathy]

Hi Mridul,

Thanks for the input. I did try running the job setting the master as 'yarn-standalone'. Though the job seems to have started fine, it never goes through completely. I get the following error when I try to run the SparkPi example over yarn:

Note:

my yarn version: cdh4.3.0

spark version: 0.8.0

scala version: 2.9.3

13/06/21 15:39:51 INFO yarn.ApplicationMaster: All workers have launched.
13/06/21 15:39:51 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 60000
13/06/21 15:39:51 INFO yarn.YarnAllocationHandler: rsrcRequest ... host : *, numContainers : 0, p = 1, capability: memory: 1408
13/06/21 15:39:51 INFO yarn.WorkerRunnable: Prepared Local resources Map(app.jar -> resource {, scheme: "hdfs", port: -1, file: "/user/somsatapathy/spark/7app.jar", }, size: 359513, timestamp: 1371854387434, type: FILE, visibility: APPLICATION, , spark.jar -> resource {, scheme: "hdfs", port: -1, file: "/user/somsatapathy/spark/7spark.jar", }, size: 63823475, timestamp: 1371854387388, type: FILE, visibility: APPLICATION, )
13/06/21 15:39:51 INFO yarn.WorkerRunnable: Prepared Local resources Map(app.jar -> resource {, scheme: "hdfs", port: -1, file: "/user/somsatapathy/spark/7app.jar", }, size: 359513, timestamp: 1371854387434, type: FILE, visibility: APPLICATION, , spark.jar -> resource {, scheme: "hdfs", port: -1, file: "/user/somsatapathy/spark/7spark.jar", }, size: 63823475, timestamp: 1371854387388, type: FILE, visibility: APPLICATION, )
13/06/21 15:39:51 INFO yarn.WorkerRunnable: Setting up worker with commands: List(java  -server  -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m  spark.executor.StandaloneExecutorBackend akka://sp...@10.58.17.191:56529/user/StandaloneScheduler 1 somsatapathy-MacBookPro1 1 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr)
13/06/21 15:39:51 INFO yarn.WorkerRunnable: Setting up worker with commands: List(java  -server  -XX:OnOutOfMemoryError='kill %p' -Xms1024m -Xmx1024m  spark.executor.StandaloneExecutorBackend akka://sp...@10.58.17.191:56529/user/StandaloneScheduler 2 somsatapathy-MacBookPro1 1 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr)
13/06/21 15:39:54 INFO cluster.StandaloneSchedulerBackend: Registered executor: Actor[akka://sparkExecutor@somsatapathy-MacBookPro1:56543/user/Executor] with ID 1
13/06/21 15:39:54 INFO cluster.StandaloneSchedulerBackend: Registered executor: Actor[akka://sparkExecutor@somsatapathy-MacBookPro1:56542/user/Executor] with ID 2
13/06/21 15:39:54 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager somsatapathy-MacBookPro1:56548 with 647.7 MB RAM
13/06/21 15:39:54 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager somsatapathy-MacBookPro1:56549 with 647.7 MB RAM
13/06/21 15:39:54 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
Exception in thread "Thread-2" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:154)
Caused by: java.lang.IncompatibleClassChangeError: class spark.InnerClosureFinder has interface org.objectweb.asm.ClassVisitor as super class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
	at spark.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:69)
	at spark.ClosureCleaner$.clean(ClosureCleaner.scala:89)
	at spark.SparkContext.clean(SparkContext.scala:729)
	at spark.RDD.map(RDD.scala:233)
	at spark.examples.SparkPi$.main(SparkPi.scala:18)
	at spark.examples.SparkPi.main(SparkPi.scala)
	... 5 more

This clearly seems to be some sort of incompatibility issue. But it's not clear what's causing this.

Any help here?

Thanks,

Som

[Mridul Muralidharan]

Ah ! This is not an issue related to running on yarn if I understand
right - but a spark issue itself.
Matei, I saw some PR about ClosureCleaner and ASM changes - something
related to that ?

Thanks,
Mridul

[Matei Zaharia]

Yes, this was a bug in the build process that I've recently fixed. It was pulling in two versions of ASM. Try a new, clean build of the master branch.

Matei

[Som Satapathy]

Thanks guys!

Som

[3504...@qq.com]

Hi,Som! I have same problem as yours recently,and I dont know how solve it ,can you give me some sugesstions?Thanks!

在 2013年6月25日星期二UTC+8上午6时34分08秒，Som Satapathy写道：