Spark Job Server

[Evan Chan]

Hey guys,

As some of you know, here at Ooyala we are working on a general-purpose REST Job Server for Spark, and are very much intending to contribute it back to Spark. 

We would like to share about what we have done so far, and gather feedback as far as its architecture and design.   In addition, questions such as

- should it be contributed back as a separate project of spark itself, or would it be better if we just open sourced it independently?

Would it be better to share the design / overview here as a email?  Should I create a ticket?  

Overall, the job server is intended to help with running jobs

- for standalone, or Mesos, or YARN modes

- for isolated jobs or for jobs sharing cached datasets in a context

- job statuses and history are persisted and queryable

Looking forward to the discussion,

Evan

[Matei Zaharia]

Hey Evan,

It might be easiest to open a JIRA for this if you already have a design doc. JIRA makes it easy to attach files and to track the discussion over time.

In any case though, this would be cool to have. Where to include it is hard to tell without seeing the design in more detail. It might be nice to make it part of Spark as a whole (and for example have it on by default in the standalone mode), unless it has some things that seem like they'll change quickly over time or otherwise seems hard to develop together with Spark.

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-develope...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Jason Dai]

Hi Evan,

This is very cool. We are planning to deploy Spark for multiple users and concurrent jobs, and will be very interested in such a job server. As Matei mentioned, a design doc in JIRA is a good starting point; in addition, if you have code/prototype that you can share, we are happy to hack it around :-)

Thanks,

-Jason

--

[Evan Chan]

It wouldn't be too hard for us to share the code, since we have mostly maintained it as something we plan to contribute....  but let me start with a JIRA ticket.   Also we're about to embark on a redesign, so it's a good time to share our design with the community.

-Evan

You received this message because you are subscribed to a topic in the Google Groups "Spark Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-developers/NPw9Kk-2wk4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spark-develope...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

--

Evan Chan

Staff Engineer
e...@ooyala.com  | 

[Henry Saputra]

HI Evan, +1 for this effort.

I would love to contribute and help with this feature in any way.

- Henry

--

[Evan Chan]

Hey guys,

I've created a ticket, https://spark-project.atlassian.net/browse/SPARK-818

and attached a very high level features / architecture / API document.

We are in the middle of redesigning the system and should have more documents / detailed dataflows to attach soon, but please have a look and start commenting.  

thanks,

Evan

You received this message because you are subscribed to a topic in the Google Groups "Spark Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spark-developers/NPw9Kk-2wk4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spark-develope...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Evan Chan]

I received one comment.... anybody else?   It'd be great to hear feedback on overall architecture, whether this could be a spark project itself, etc.

I will start a separate email discussion on a particular aspect of job server that I'd like feedback from.

thanks,

Evan

[MLnick]

Hi Evan

This sounds like something very useful and something that I need right now. I have a workflow where my front-end servers will periodically kick off jobs on the compute cluster in a scheduled manner (and possibly low-latency jobs in the future).

The basic architecture looks good overall. 

I started a basic version of the same idea just for my own narrow use case and ran into issues that appear to be conflicts between Jetty / Netty versions in the web server (I am using Scalatra) and Spark.

How did you approach this? Did you just use the same versions as Spark? Spray? etc etc?

Any idea when this might be ready to release?

Thanks

Nick

[Evan Chan]

Hi Nick,

I responded on the ticket itself, but we are planning to start contributing it back in bits and pieces starting in about a week, so we can get feedback.

-Evan

[Henry Saputra]

HI Evan, will you guys be doing it in separate branch/repo or will be as new module in Spark? 

Would love to try and help around when we can.

- Henry

[Evan Chan]

Hi Henry, we are planning for it to be in a separate repo.

-Evan

[Evan Chan]

Sorry I meant separate module.