Handling Scheduled Jobs in High Availability Topology

[Chris Cerda]

Hey, so I have setup Rundeck in the following manner:


rundeck.url.com -> apache instances(2 of them) -> rundeck instances(2 of them).

We are using an oracle database on the backend for the job and project configurations, along with having a shared disk that holds the local configurations on the server.  How do we handled scheduled jobs, b/c as it is now, if I schedule a job, the job runs from only that one server.  That is good, but how do I handle it if a user schedules it and they happen to be on 01, but then what happens if 01 is down, and 02 is up.  I need 02 to run the schedule.  And at the same time, I only want the job to kick off from one server if they are both up.

How do we handle having scheduled jobs only ever running from one server.  Frankly, doesn't matter which server the job kicks off from, or 01 or 02 server, but we definitely can't have it run twice.

[Chris Cerda]

So it seems that if I have it scheduled and then restart both servers, it'll be scheduled to run on both, but on one of the servers it'll throw the below stack trace.  I assume that is because the job is set to not allow multiple executions at once. 

Also, so I have 2 servers with 1 scheduled job, that runs on both at the given interval.  Now let's say I want to delete or modify that schedule.  When I go to my load balancer url, let's say I hit the 01 Instance, and delete the job schedule.  That's fine, it doesn't run on 01 any more, however the job continues to run on 02.



2016-07-07 13:51:00,442 ERROR ExecutionJob - Unable to start Job execution: Job "Test Local Job" [8980e311-50d6-4066-83f9-bdb276fdbc82] is currently being executed (execution [[6561]])
rundeck.services.ExecutionServiceException: Job "Test Local Job" [8980e311-50d6-4066-83f9-bdb276fdbc82] is currently being executed (execution [[6561]])
        at rundeck.services.ExecutionService.createExecution(ExecutionService.groovy:1616)
        at rundeck.quartzjobs.ExecutionJob.initialize(ExecutionJob.groovy:265)
        at rundeck.quartzjobs.ExecutionJob.execute_internal(ExecutionJob.groovy:95)
        at rundeck.quartzjobs.ExecutionJob$_execute_closure1.doCall(ExecutionJob.groovy:74)
        at com.codahale.metrics.Timer.time(Timer.java:99)
        at rundeck.quartzjobs.ExecutionJob.execute(ExecutionJob.groovy:73)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
2016-07-07 13:52:00,001 INFO  GrailsServiceInjectorJobListener - services injected for job

[Greg Schueler]

Hi Chris,

you need to enable "rundeck.clusterMode.enabled=true" in rundeck-config.properties, and specify a UUID for each server (`rundeck.server.uuid` in framework.properties)

Then scheduled jobs are bound to only one server, and will not run on other servers.  You can transfer the schedule ownership from one server to another via an API call, or by modifying/uploading the job to another cluster node.

-- 
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/62654a9a-2658-40bb-95c5-9c22db60d379%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Cerda]

This seemed to work in terms that I see a job only running on the server that it was created on. 

Is there a way to trigger an automatic failover of the scheduled job if the node it was scheduled on, goes down? 

To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discuss+unsub...@googlegroups.com.