Continuous patching for large build pool

[bglowacki]

I'm working on a project to perform windows patching across a large pool of 450 Windows build agents. The build agents need to be patched monthly. Once a month, each agent needs to be offlined as a build agent, patched using bigfix, reboot - and then brought back online as a build agent. I have written a series of python scripts launched from ansible to do this - and it works pretty well but is not optimal.

Currently, the project works against 5 build agents at a time to reduce the impact associated with greatly reducing build resources at any given time. To do this, I utilize the serial: 5 option on my playbook plays - which works to limit the patching to 5 agents at a time. However, all agents must be complete before ansible moves on to the next group of 5 agents. Some of our build agents get involved with activities that can take as long as 8 hours to offline, but much more often they take less than 20 minutes. Since all 5 agents must be finished prior to the play continuing on to the next set of agents, this greatly increases the length of time to apply these patches. I can't move on unless the blocking 8 hour agent is finished! 

What I would like is to be able to specify 5 agents "at a time," meaning that, when one of the 5 agents finishes with the patching play, it will automatically start the play against another agent - always keeping the maximum number of agents getting patched at any give time to be 5. That way, if one agent is blocked for 8 hours, the remaining 4 slots can be used to cycle through the remaining build agents - and so on. 

Is there some combination of the serial/forks options that would provide me this kind of capability - or am I asking for a new feature enhancement?

[Kai Stian Olstad]

On 05.10.2018 22:59, bglowacki wrote:
> What I would like is to be able to specify 5 agents "at a time,"
> meaning
> that, when one of the 5 agents finishes with the patching play, it will
> automatically start the play against another agent - always keeping the
> maximum number of agents getting patched at any give time to be 5. That
> way, if one agent is blocked for 8 hours, the remaining 4 slots can be
> used
> to cycle through the remaining build agents - and so on.
>
> Is there some combination of the serial/forks options that would
> provide me
> this kind of capability - or am I asking for a new feature enhancement?

This is not possible with Ansible itself, you have strategy: free, but
it still wait for all hosts in the serial batch to finished before
moving to next set of hosts.

What you can do is use parallel or xargs to accomplish this.
This will run 5 ansible-playbook in parallel, each ansible-playbook is
limited to one host.

ansible-playbook playbook.yml --list-hosts | sed -e '1,/hosts/d' -e
's/^ *//' | xargs -P5 -n1 ansible-playbook playbook.yml --limit

This would only work well if you don't need to type in passwords.

--
Kai Stian Olstad

[bglowacki]

Brilliant! Works perfectly! Thank you so much. :)

[chenchireddy guvvala]

Use the serial keyword in Playbook, Like below

- name: test play
  hosts: webservers
  serial: 3

Reference: https://docs.ansible.com/ansible/2.6/user_guide/playbooks_delegation.html

Thanks.

[bglowacki]

Unfortunately, in my experience the serial keyword results in 3 concurrent plays (using your example), but all plays must be complete before moving on to the next 3. :(

[bglowacki]

I have opened a request for an integrated enhancement with Ansible: https://github.com/ansible/ansible/issues/46649

However, Kai Stian Olstad solution is a good work around, in the meantime.