Livy with spark Mesos

[raghav....@gmail.com]

HI all,

I am using Spark on a DCOS cluster. This uses mesos as manager. I was wondering if there any way to integrate livy with mesos.

-pR

[Saisai Shao]

Hi, 

Livy doesn't bind to any specific cluster manager, you could configure the livy.spark.master (IIRC) to point to Mesos. Though we add some improvements for running on YARN, we don't limit users to only run on YARN. I guess there's some users who running Livy and Spark with standalone mode.

--
You received this message because you are subscribed to the Google Groups "Livy Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to livy-dev+unsubscribe@cloudera.org.

[sub kums]

My spark is running inside the docker container (In Mesos cluster ). How to expose SPARK HOME outside the container to the host machine. 

I need Livy to use spark home to start the livy server.. Can you help me ? Is any other alternative options ? 

[Saisai Shao]

Sorry I don't have a extensive knowledge of Docker, not sure how to run Spark / Livy inside the Docker.

[Prabhu Kasinathan]

My two cents:

1) You can start Livy server within Spark container and expose 8998 port. {or}

2) You can start that spark container by setting volumne to spark home and you can use outside the container.

[zeo...@gmail.com]

I have never tried to run it on mesos but there are a number of livy features (session recovery, SparkApp state management) that assume the use of yarn framework. So even if you manage to get it working it is likely to have a much reduced feature set compared to livy+spark+yarn.

[Saisai Shao]

Yes, the similar functionalities could also be achieved using Mesos, because we're not super familiar with Mesos, that's why currently running on yarn has some advantages.

--