How can zeppelin run on yarn-cluster

[LIU Xiaobing]

Hi all,

    I heard that zeppelin cann't run on yarn-cluster. Is it true? Is there anybody to have run zeppelin on yarn-cluster successfully if it can? And how?

Regards

[Roman Zykov]

We launched it in  yarn client mode

export MASTER=yarn-client

[Ophir Cohen]

Did you manage to run on Yarn with Spark 1.2?
I couldn't build it for Hadoop 2.4 and Spark 1.2...

[Roman Zykov]

It works with 1.1.1. But I saw in this forum that Zeppelin doesn't work with 1.2 at this time

[Ophir Cohen]

Ok Thanks, I'll try it with 1.1

[goi...@gmail.com]

Ophir,

Can we chat on your experience with Zeppelin? (goi.cto at gmail.com)

Eran

[Ophir Cohen]

Sure.
Feel free: email, hangouts whatever you like.

--
You received this message because you are subscribed to a topic in the Google Groups "zeppelin-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zeppelin-developers/f-RndeowiDo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zeppelin-develo...@googlegroups.com.
To post to this group, send email to zeppelin-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[kevin...@gmail.com]

It works with 1.2, we are using zeppelin with spark1.2

Best Regards,

Kevin.

在 2015年1月24日星期六 UTC+8上午5:03:27，Roman Zykov写道：

[Ophir Cohen]

On yarn?
Can you share how did you configure it?
I keep getting the exception I posted.

--

[LIU Xiaobing]

Hi Kevin,

Could you please share the work? How did you configure it?

Regards

Xiaobing

--

You received this message because you are subscribed to a topic in the Google Groups "zeppelin-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zeppelin-developers/f-RndeowiDo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zeppelin-develo...@googlegroups.com.
To post to this group, send email to zeppelin-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Best Regards
LIU Xiaobing 刘小兵

[Ophir Cohen]

+1 for that...

[戴云晶]

The script is like this:

$ZEPPELIN_HOME/bin/zeppelin.sh \

  --master yarn-client \

  --jars ${JARS} \

  --files ${FILES} \

  --properties-file ${PROPERTY_FILE} \

  --queue ${QUEUE} \

  --driver-memory ${SPARK_DRIVER_MEMORY} \

  --driver-cores ${SPARK_DRIVER_CORE} \

  --driver-library-path ${SPARK_LIBRARY_PATH} \

  --driver-java-options "${SPARK_DRIVER_JAVA_OPTIONS}" \

  --executor-memory ${SPARK_EXECUTOR_MEMORY} \

  --executor-cores ${SPARK_EXECUTOR_CORE} \

  --num-executors $SPARK_NUM_EXECUTORS \

  $@

[Damien Corneau]

Since there is so many discussions and people interested in YARN,

It would be nice to have a working YARN example for zeppelin, that way we could have it as tutorial just like the basic one we have on the website

http://zeppelin-project.org/docs/tutorial/tutorial.html

You received this message because you are subscribed to the Google Groups "zeppelin-developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zeppelin-develo...@googlegroups.com.

[kevin...@gmail.com]

Hi, Xiaobing

There is no configuration. The only thing you should do is setting the SPARK_HOME environment variable.

And then start the job as I have described. In fact it's just a spark job which started by spark-submit, so the parameters are the same as spark-submit.

Best Regards,

Kevin.

在 2015年1月27日星期二 UTC+8下午12:23:55，LIU Xiaobing写道：

Hi Kevin,

Could you please share the work? How did you configure it?

Regards

Xiaobing

On Tue, Jan 27, 2015 at 10:50 AM, <kevin...@gmail.com> wrote:

It works with 1.2, we are using zeppelin with spark1.2

Best Regards,

Kevin.

在 2015年1月24日星期六 UTC+8上午5:03:27，Roman Zykov写道：

It works with 1.1.1. But I saw in this forum that Zeppelin doesn't work with 1.2 at this time

On Friday, January 23, 2015 at 4:45:31 PM UTC+3, Ophir Cohen wrote:

Did you manage to run on Yarn with Spark 1.2?
I couldn't build it for Hadoop 2.4 and Spark 1.2...

--
You received this message because you are subscribed to a topic in the Google Groups "zeppelin-developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zeppelin-developers/f-RndeowiDo/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zeppelin-developers+unsub...@googlegroups.com.

To post to this group, send email to zeppelin-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Charmee Patel]

Hi,

I am also trying to get Zeppelin work in yarn-client and yarn-submit mode. Looks like many people are trying to do same thing. If you can help through my errors, I can do a blog post with more info on what is needed to get Zeppelin working with Yarn.

I have CDH 5.3 . Here is what I have so far

1. Built Zeppelin on local (OSX) using 
   
1. mvn clean package -Pspark-1.2 -Dhadoop.version=2.5.0-cdh5.3.0 -Phadoop-2.4 -DskipTests -Pyarn
2. Generated a distribution package and deployed to edge node of my cluster using 
   
1. mvn clean package -P build-distr -DskipTests
3. At this point local mode works fine
4. To get Yarn mode working 
1. I set Hadoop conf dir in zeppelin-env.sh and set Yarn Master=yarn-client
2.  Set Spark_Yarn_Jar but that did not work
3. Copied my spark assembly jar in interpreter/spark directory but that did not work either
4. I took Spark Yarn Jar out and let Zeppelin use Spark that comes bundled with it
1. This seemed to work initially, but as soon as I call Spark Action (collect/count etc) I get this exception 

java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:182)

Any pointers on what else I can try?

Thanks,

Charmee