rolling tasks for multiple clusters?

[Doug OLeary]

Hi;

I've read the docs on rolling updates and that sounds very close to what I'm looking for; however, I need to execute rolling reboots for multiple clusters preferably simultaneously.

More specifically, after patching oracle clusters, i need to:

1.  Execute a task to relocate oracle services on one node.

2.  Reboot that node.

3.  Execute a task to relocate oracle services back to that node.

Those tasks should happen on only one node of a cluster at a time and the rolling updates documented by ansible work well for one cluster.  I have occasions where I'm patching 3 or 4 clusters though.  

With my limited ansible knowledge, the only thing I'm coming up with is a rolling update across all nodes or separate playbooks for each cluster both of which are grossly inefficient.  3 clusters should be run as 3 simultaneous rolling updates.  the separate playbooks for each cluster just sounds wrong.  

Can someone point me in the right direction?  I'm happy to do the reading but my google-foo must be weak as I'm not finding the concept that I'm so obviously missing.

Thanks

Doug O'Leary

[J Hawkesworth]

I wound up having multiple playbooks and using something else to run them simultaneously.

I've used gnu parallels in some cases and jenkins jobs in others.

parallels is nice because you can kick off a lot of jobs simultaneously with a single command line.

However its not great for viewing progress as the output from each playbook isn't returned until its completed (iirc)

Jenkins jobs give better visibility of what's going on, but become tedious to set up via the ui if you have more than a few of them.

Other tools to orchestrate multiple playbooks are available (tower/awx, semaphore, vespene, rundeck) but I haven't spent enough time trying any of them yet to see if they offer any advantage.

That said it might be possible to do what you want in a single playbook, but you might need to get creative with your inventory.

Be aware that you can target multiple host groups in a single play

hosts: webservers:middleware:batchmachines

If you can address all the machines you need to manipulate then you might be able to set serial: 1 and perform rolling changes that way.

Hope the above is useful to you.

Jon

[Doug OLeary]

Well, that's not the answer I was looking for! :)  

Thanks for the reply.  In case it's not clear, I'm very new to ansible and thought my need for ansible tower was a couple of months if not a year or more away.  In the meantime, I'll look into those tools.  I use pssh for remote parallel executions; haven't had the need for local parallel execution.

Thanks again.  I appreciate it.

Doug

[J Hawkesworth]

If you have understood 'serial' and host groups then I think you know what ansible will do for you in terms of rolling updates.

There's a good discussion here regarding parallelizing ansible playbooks: https://github.com/ansible/proposals/issues/31

I should have mentioned there are some tricks you can use to run multiple playbooks discussed in Brian Coca's talk here: https://www.slideshare.net/bcoca/more-tips-n-tricks (linked from above discussion).

I'm not saying what I've done is the right way but it has turned out to be important to be able to isolate where problems lie when you are trying to update a system made up of a lot of different components.

Jon