Azkaban Job Type Plugin for Flink

[yann....@gmail.com]

Hi all,

We want to integrate our Flink instances with our Azkaban scheduler.
For that we will have to create a custom job type plugin for Azkaban.
Has anyone already started creating something like that ? If not... I guess we will try to !

Best,

Yann

[Ameya Kanitkar]

That sounds interesting. 

No one has started working on this AFAIK.

Can you share design and use cases?

[Juho Autio]

I assume that your use case is about having an existing Flink cluster and running some jobs on it.

Just sharing how we do for reference. We use a custom job type that can launch an AWS EMR cluster and submit steps on it. This is how we also run both batch & stream Flink jobs. EMR has the limitation that there can be only one step running at a time. We install Flink in an EMR bootstrap step and it uses YARN so deployment is simple.

Cheers,

Juho

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.