Running Azkaban outside of Cluster

[V]

Can we run Azkaban outside of the cluster, on that case do we have provision to make connection to cluster?

[Swatz]

I'm running Azkaban outside the cluster.

I have launched the cluster with whirr. So I use proxy to make connection and execute my jobs.

This machine, outside the cluster has hadoop setup in pseudo-distributed mode. Though this is not necessary, you can as well use the hadoop-{$version} tarball.

While running your command, make sure you use --config with the details of the cluster.

so, it goes like this

type=command

command=hadoop --config /home/ubuntu/.whirr/workflow/ jar your_jar.jar ... [if you have pseudo-distributed mode setup]

command=<path_to_tarball>/bin/hadoop --config /home/ubuntu/.whirr/workflow/ jar your_jar.jar ... [running from the tarball]

Hope this helps.

Thanks.

[yu.ch...@gmail.com]

It is very possible you run azkaban instance anywhere there is an ACL to the production cluster you want to run jobs on.

In Azkaban2, each job type has its own classloader. You can config a specific type with configurations of a specific cluster. For example, you can give hadoop conf directory in the class loader classpath of say pig, then it should be talking to the namenode and job trackers specified in the conf files. 

You need ACL from azkaban machine to your cluster, of course.