Re: "Cannot run program ’spark-submit‘ " when start livy after RPM install hue

Enrico Berti

unread,

Nov 26, 2015, 12:00:07 PM11/26/15

to Wang Bin, Hue-Users

Do you have Spark installed in your cluster?

On Thu, Nov 26, 2015 at 5:39 PM, Wang Bin <ben....@gmail.com> wrote:

Before RPM, "hue/apps/spark/" directory structure as follows:
-babel.cfg
-hueversion.py -> ../../VERSION
-hueversion.pyc
-java
-bin
-conf
-livy-defaults.conf.tmpl
-dist.xml
-livy-assembly
-livy-core
-livy-repl
-livy-server
-livy-yarn
-pom.xml
-README.rst
-target
-java-lib
-Makefile
-setup.py
-src
-spark
-spark.egg-info

After RPM and install, "hue/apps/spark/" directory structure as follows:
-babel.cfg
-ext-eggs
-hueversion.py
-hueversion.pyc
-java-lib
-livy-assembly.jar
-Makefile
-setup.py
-src
-spark
-spark.egg-info

configured /etc/profile as follows:
# set spark environment
SPARK_HOME=/home/spark/
export SPARK_HOME
export PATH=${PATH}:${SPARK_HOME}/bin

// the hue file
...
su -s /bash/shell root "cd /var/log/hue/ && nohup /usr/lib/hue/build/env/bin/hue livy_server > livy.log &"
...

when "service hue start", the livy.log as follows:
Failed to run spark-submit executable: java.io.IOException: Cannot run program "spark-submit": error=2, No such file or directory

I have tried as follows:
1, insert the congiguration into "/etc/hue/conf.empty/hue.ini"
livy.server.spark-submit = /home/spark/bin/spark-submit

The same Error as the log.

2、I want to configure the spark-submit into livy-defaults.conf, but no found the file.

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

Wang Bin

unread,

Nov 26, 2015, 6:00:19 PM11/26/15

to Enrico Berti, Hue-Users

yes, just execute "su -s /bash/shell root "cd /var/log/hue/ && nohup /usr/lib/hue/build/env/bin/hue livy_server > livy.log &"" in terminal is ok, but in hap and service hap start shows "Failed to run spark-submit executable: java.io.IOException: Cannot run program "spark-submit": error=2, No such file or directory". I don't know the reason.

Enrico Berti

unread,

Nov 26, 2015, 6:28:30 PM11/26/15

to Wang Bin, Hue-Users

spark-submit is part of Apache Spark. You need to have Spark installed on your cluster. Not Livy, not Hue, but Apache Spark. http://spark.apache.org/

Wang Bin

unread,

Nov 26, 2015, 7:33:30 PM11/26/15

to Enrico Berti, Hue-Users

yes, I have installed spark in my cluster`s every Node, and execute "su -s /bash/shell root "cd /var/log/hue/ && nohup /usr/lib/hue/build/env/bin/hue livy_server > livy.log &"" in terminal is ok, but write the statement in hap and "service hap start" shows "Failed to run spark-submit executable: java.io.IOException: Cannot run program "spark-submit": error=2, No such file or directory". I don't know the reason.

Romain Rigaux

unread,

Nov 26, 2015, 9:53:07 PM11/26/15

to Wang Bin, Enrico Berti, Hue-Users

Spark exec is not there, what happens when you type 'spark-submit' in the shell?

Wang Bin

unread,

Nov 26, 2015, 10:40:20 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 10:42:20 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 10:45:22 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 10:53:42 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 11:15:00 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 11:18:04 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 26, 2015, 11:22:49 PM11/26/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 27, 2015, 3:17:46 AM11/27/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Wang Bin

unread,

Nov 27, 2015, 8:34:10 AM11/27/15

to Hue-Users, rom...@cloudera.com, enrico...@cloudera.com

Thank you for your answers. Livy-server can start and submitting snippets by add . /etc/profile in configure file hue.

在 2015年11月27日星期五 UTC+8下午4:17:46，Wang Bin写道：

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+unsubscribe@cloudera.org.

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+unsubscribe@cloudera.org.

Enrico Berti

unread,

Nov 27, 2015, 9:04:25 AM11/27/15

to Wang Bin, Hue-Users, Romain Rigaux

awesome! glad you figured it out!

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

Wang Bin

unread,

Nov 28, 2015, 6:25:41 AM11/28/15

to Romain Rigaux, Enrico Berti, Hue-Users

#spark-submit

shows the Usage and Options of spark-submit.

Reply all

Reply to author

Forward