Genie with Mesos Clusters

[Yogesh]

Hi,
I am trying to use Genie to run Spark jobs on DC/OS or Mesos clusters. I would like to know if Genie supports DC/OS or Mesos out of the box. If not, what would it entail to add that support. I would be great if anyone can share their experience / point to resources of running Genie with DC/OS or Mesos clusters.
Thanks,
Yogesh.

[Tom Gianos]

Yogesh,

While we haven't run Genie with Mesos clusters at Netflix yet looking at the documentation for Mesos it doesn't appear it would be that hard. I looked at how spark submit works with Mesos and it looks like you just need to provide the --master argument with mesos://something or mesos://zk etc.  So to do this I could see registering your mesos clusters with setup files that export a Mesos master URL like EXPORT MESOS_MASTER_URL=mesos://myMesosCluster/api/v1/etc/etc. Then within your command configuration you can set your path to use this like --master ${MESOS_MASTER_URL}. In this way your spark-submit would work on any number of mesos clusters registered with Genie. We do something similar with our presto clusters internally.

Let me know if that helps.

Tom