Spark job server and zeppelin

[goi...@gmail.com]

Hi,

I am looking into the possibility of integrating spark-jobserver as a back-end for zeppelin. 

1) did you find a need to manage spark-jobs in such a way while working with zeppelin? 

2) How do you see the monitoring process working with spark?

Eran 

[moon soo Lee]

Hi, Eran

Thanks for opening interesting topic.

In my understanding (i have no deep understanding), spark-jobserver runs spark job with their dependencies

The jobs are packaged in Jar. Then the jars are uploaded into spark-jobserver and run by REST API.

Zeppelin is more like spark-shell. It runs as a spark job, but unlike any other spark job, it does not have any prebuilt code, instead it does have scala interpreter. While zeppelin is running, a spark job is keep running and Zeppelin's scala interpreter dynamically interpret user code and send it to SparkWorkers to run.

right now, i have no good idea how spark-jobserver can be used as a Zeppelin backend.

In the other way, if zeppelin have ability to submit prebuilt spark job using spark-jobserver with scheduling, triggering support, like oozie / azkaban for map-reduce, that might be helpful for some cases.

I'm not sure if i understand your question about monitoring process working with spark. can you explain little bit more about it?

Thanks,

moon

[이호연]

I also use zeppelin and spark job server. And I think if I could change the zeppelin's SparkContext to the one of Spark Jobserver's, that would be fine to integrate them. It means I can run adhoc script as well as a prebuilt spark job in a zeppelin web server.
There's some obstacles to do that, I think. A lot of dependencies are different each other, i.e spray and akka,Spark Jobserver's default context should be ready before starting the zeppelin engine.