Running Spark jobs via Marathon

277 views
Skip to first unread message

B. Svenjord

unread,
Aug 22, 2016, 4:26:00 AM8/22/16
to marathon-framework
I've got a Marathon job running for the following:
./bin/spark-class org.apache.spark.deploy.mesos.MesosClusterDispatcher

However, following that I'd like to now be able to run individual Spark jobs as separate Marathon jobs with the command:
./bin/spark-submit ....

My question is:
how can I call spark-submit from a Mesos executor of the binaries are not installed on it? 
(Note: I'm aware that http://spark.apache.org/docs/latest/running-on-mesos.html#connecting-spark-to-mesos also recommends installing Spark on all the Mesos slaves, but is that the only option?)

Any guidance is much appreciated.

Durga swaroop Perla

unread,
Sep 3, 2016, 10:00:02 AM9/3/16
to marathon-framework
If you're looking to have Mesos manage your spark cluster, then that is the only way. You need to have spark binaries on all the mesos agents but If you just want a simple standalone cluster then you don't need to do that. 
Reply all
Reply to author
Forward
0 new messages