Spark in HUE with YARN mode

[Anandha Loganathan]

Hi,

I am trying to run Spark in YARN mode using HUE.   I am able to run Spark, PySpark in local mode. Right now I am trying to run in cluster mode either interactive or submit mode.

I followed the steps through this document to configure.   http://gethue.com/how-to-use-the-livy-spark-rest-job-server-for-interactive-spark-2-2/

I am unable to submit it using yarn mode.  Here are  my configuration. 

HUE - 3.11

Livy - .livy-server-0.2.0

in HUE.ini, my configurations are 

 I am running livy on the same host. 

  # Host address of the Livy Server.

  ## livy_server_host=localhost

  # Port of the Livy Server.

  ## livy_server_port=8998

  # Configure Livy to start in local 'process' mode, or 'yarn' workers.

   livy_server_session_kind=yarn

  # Host of the Sql Server

   sql_server_host=usw2prdpmn01

  # Port of the Sql Server

  ## sql_server_port=10000

 

The question is I am unable submit the spark job to the cluster. I don't see any session is running in YARN scheduler.  The configuration setup page is 1 year old and I am not sure what is mis configuration I am making to run it in YARN mode.  

Any help  or pointers is highly appreciated.

Thanks

Anand