Re: [azkaban] Outside of cluster

[Felix GV]

Yes, that's what we're doing, but the Azkaban host must have the relevant hadoop packages installed as well as the correct hadoop configs, so that it can spawn jobs on the cluster... obviously, in order to spawn the jobs, the JobTracker should also be reachable from the Azkaban host.

--
Felix

On Mon, Mar 25, 2013 at 7:10 PM, V <siv.v...@gmail.com> wrote:

Can we run Azkaban out of cluster ?

--
You received this message because you are subscribed to the Google Groups "azkaban" group.
To unsubscribe from this group and stop receiving emails from it, send an email to azkaban-dev...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[siv.v...@gmail.com]

Thanks Felix!
I'm able to set it up now.
 
I'm setting it up in Ubuntu, where able to run Azkaban through script, if I deploy the war file in Tomcat container I'm facing below issue.

Mar 29, 2013 2:37:58 AM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class azkaban.web.AzkabanServletContextListener
java.lang.NullPointerException
    at azkaban.app.AzkabanApp.getLastExecutionId(AzkabanApp.java:411)
    at azkaban.app.AzkabanApp.<init>(AzkabanApp.java:158)
    at azkaban.web.AzkabanServletContextListener.contextInitialized(AzkabanServletContextListener.java:63)
    at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4779)
    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5273)
    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
    at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:895)
    at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:871)
    at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:615)
    at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:958)
    at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1599)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:679)

[Felix GV]

Cool, glad you could get it going :)

I haven't tried using the war at all, so I can't really comment on your other issue. I guess someone from LinkedIn may be able to help...!

Good luck :) !

--
Felix

[Richard Park]

It's good to see that Azkaban is being used in a variety of settings. Nothing we've done would prevent wars, except with servlet context. A singleton of Azkaban app needs to be instantiated which is not done.

However, we've de-emphasized this to give focus to more contained solutions as with Jetty. Do you need Azkaban to run in Tomcat or Glassfish etc? If so, we can try to run it under these scenarios to make sure that the Azkaban war content is accessible.

I'm in the SF Bay Area. It would be good to see how others deploy/run Azkaban so we can get a better idea of what we should target next. If any are willing, we (Chenjie and I) may be able to go to you to see what other things we should focus on.

Thanks,

-Richard

[siv.v...@gmail.com]

I followed the other route - starting the azkaban-server.sh using a cron in ubuntu under my username. I'm able to maintain this system always up and running now.

I have one another usecase - setting up azkaban in Windows, I'm trying to run the sample Java jobs in Azkaban, I'm facing below issues.

Job Name: JavaTest

Type: java

Path: \JavaTest

Dependencies: None

Key	Value
job.class	com.azk.ExampleJob
working.dir	\JavaTest
method.run	run
type	java

03-04-2013 22:11:12 UTC JavaTest INFO - HADOOP_HOME not set, using default hadoop config.
03-04-2013 22:11:12 UTC JavaTest INFO - 1 commands to execute.
03-04-2013 22:11:12 UTC JavaTest ERROR - Fatal error occurred while running job 'JavaTest':
java.lang.RuntimeException: Failed to create temp property file 
	at azkaban.jobs.AbstractProcessJob.createFlattenedPropsFile(AbstractProcessJob.java:177)
	at azkaban.jobs.AbstractProcessJob.initPropsFiles(AbstractProcessJob.java:106)
	at azkaban.jobs.builtin.ProcessJob.run(ProcessJob.java:64)
	at azkaban.app.LoggingJob.run(LoggingJob.java:106)
	at azkaban.flow.IndividualJobExecutableFlow$1.run(IndividualJobExecutableFlow.java:180)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: The system cannot find the path specified
	at java.io.WinNTFileSystem.createFileExclusively(Native Method)
	at java.io.File.checkAndCreate(Unknown Source)
	at java.io.File.createTempFile(Unknown Source)
	at azkaban.jobs.AbstractProcessJob.createFlattenedPropsFile(AbstractProcessJob.java:174)
	... 5 more

[Richard Park]

We haven't tested under Windows yet. For the most part, a unix based file system is expected. I could imagine that we're setting a few paths incorrectly.

---

From: azkab...@googlegroups.com [azkab...@googlegroups.com] on behalf of siv.v...@gmail.com [siv.v...@gmail.com]
Sent: Wednesday, April 03, 2013 3:20 PM
To: azkab...@googlegroups.com
Subject: Re: [azkaban] Outside of cluster