Speed up Spark applications in using Livy

[Youngwoo Kim]

Hi,

We have a spark application, which is written in scala and running on yarn with spark-submit. the app is launched by user's request and it takes 30+ seconds or x minutes (3 < x < 30) In long running jobs, startup time is not a problem but, in the mean time, we have a lot of jobs that should complete in 30 or 40 seconds. so, the time to initialize spark driver on yarn is too long. in our cluster, it takes 10+ seconds. I'm curious, is it possible to reuse existing spark driver or context on Livy server ? I just want to reduce the start up time in yarn mode.

Thanks,

Youngwoo

[Romain Rigaux]

This is exactly the goal of Livy, to have a long running context, you can ask your question there http://groups.google.com/a/cloudera.org/group/livy-user

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.