Running spark job on YARN

[Som Satapathy]

Hi All,

I am trying to run spark jobs over YARN on a single node setup. I am referring to http://www.spark-project.org/docs/0.7.2/running-on-yarn.html for that. 

I am passing 'standalone' as my master URL. But strangely my spark job always shows as 'RUNNING' in my YARN resourcemanager console. When I dig into the log, I see the following error:

13/06/20 19:01:45 WARN spark.SparkContext: Master standalone does not match expected format, parsing as Mesos URL
Failed to load native Mesos library from .:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
Exception in thread "Thread-3" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:154)
Caused by: java.lang.UnsatisfiedLinkError: no mesos in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758)
	at java.lang.Runtime.loadLibrary0(Runtime.java:823)
	at java.lang.System.loadLibrary(System.java:1045)
	at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:46)
	at spark.SparkContext.<init>(SparkContext.scala:201)
	at spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:38)
	at spark.examples.JavaSparkPi.main(JavaSparkPi.java:21)
	... 5 more

When I browsed through the user group for similar errors,I found an email mentioning about

 the need to set MESOS_NATIVE_LIBRARY path in the spark-env.sh. 

Does this mean I need to install mesos as well to run spark jobs on YARN? 

Looking forward for your inputs.

Thanks,

Som

[Gupta]

this problem is solved or not because i am also facing this same error.. plz help me.. am new as shark & spark user..
thanks in advance..

[Jey Kottalam]

Hi Gupta,

Could you please clarify which error you're encountering, including
the exact error messages and log output?

Thanks,
-Jey

> --
> You received this message because you are subscribed to the Google Groups
> "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spark-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Gupta]

i installed shark 0.7.0 on spark 0.7.2 & successful shark job on stand alone but when i run on cluster then i am facing following error..

$ bin/shark

Starting the Shark Command Line Client

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

Logging initialized using configuration in jar:file:/usr/local/hive/hive-0.10.0/lib/hive-common-0.10.0.jar!/hive-log4j.properties

Hive history file=/tmp/hduser/hive_job_log_hduser_201309251019_715397053.txt

7.808: [Full GC 380492K->18954K(2027264K), 0.1543770 secs]

Failed to load native Mesos library at /usr/lib/mesos-0.10.0/libmesos.so

Exception in thread "main" java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/mesos-0.10.0/libmesos.so

        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1828)

        at java.lang.Runtime.load0(Runtime.java:792)

        at java.lang.System.load(System.java:1059)

        at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:38)

        at spark.SparkContext.<init>(SparkContext.scala:171)

        at shark.SharkEnv$.init(SharkEnv.scala:33)

        at shark.SharkCliDriver.<init>(SharkCliDriver.scala:237)

        at shark.SharkCliDriver$.main(SharkCliDriver.scala:137)

        at shark.SharkCliDriver.main(SharkCliDriver.scala)

is it mean that i have to install mesos library.

[Jey Kottalam]

Hi Gupta,

It's strange that Spark is trying to load the Mesos library, which is
not needed under YARN. Could you please check that the
"MESOS_NATIVE_LIBRARY" is not set in your "conf/spark-env.sh", and
that you are passing "yarn-standalone" as the Spark master URL?

-Jey

[Rohit Gupta]

thanks for your help. 

yesterday's problem solved & now the issue is .. Shark +hive-0.10
when i run using hive-0.10.0 then i am facing this error..

Starting the Shark Command Line Client

WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.

Logging initialized using configuration in jar:file:/usr/local/hive/hive-0.10.0/lib/hive-common-0.10.0.jar!/hive-log4j.properties

Hive history file=/tmp/hduser/hive_job_log_hduser_201309261806_1317596497.txt

7.223: [Full GC 335689K->18741K(2027264K), 0.1125730 secs]

8.661: [Full GC 120304K->23477K(2027264K), 0.1263000 secs]

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.hive.cli.CliDriver.getCommandCompletor()Ljline/Completor;

        at shark.SharkCliDriver$.main(SharkCliDriver.scala:160)

        at shark.SharkCliDriver.main(SharkCliDriver.scala)

[INFO] [09/26/2013 18:06:52.265] [spray-io-worker-0] [IoWorker] IoWorker thread 'spray-io-worker-0' stopped

& it run fine with using hive-0.9.0....
plz help me.

You received this message because you are subscribed to a topic in the Google Groups "Spark Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-users/buW-UYfda4M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spark-users...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Thanks and Regards,
Rohit Kumar Gupta | Software Engineer
BDI Systems | www.bdisys.com

M: +91 7416746541 | rohit...@bdisys.com