Ansible performing poorly on scale setups

[nds.ap...@gmail.com]

Hi,

we are using Ansible playbooks to build traffic tools and have around 500 hosts.

For e.g:

We build a stream file to map the interfaces of all the hosts and then trigger traffic on respective hosts participating. This stream file is around 100 MB.

So just this line in one of the playbook:

  vars_files:
    - "{{ playbook_dir }}/{{ streams_file_name }}"
 

Is taking almost 1-2 hrs of time to run.

===========================================

Read vars_file '{{ playbook_dir }}/{{ streams_file_name }}'
<172.16.0.161> ESTABLISH SSH CONNECTION FOR USER: root
<172.16.0.195> SSH: EXEC sshpass -d74 ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'User="root"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/3dd37d3787 172.16.0.195 '/bin/sh -c '"'"'echo ~root && sleep 0'"'"''
<172.16.0.241> (0, b'/root\n', b'')
====================================================

Has not gone yet to the main part of the traffic start process.

Have used mitogen too and its not helping us in any way.

Have configured everything that can be done on the configuration part.

Fork currently is being set to 64 considering the RAM that we have (64GB) and also increased the CPU cores too but does not help in reducing the time.

Any pointers in working on scale.

Was under the impression that Ansible would help for scale but does not look to be promising.

Regards
Aprameya

[steve missoh]

I dont think ansible has ever been designed to perform at scale.

Chef or Puppet are better at that because they are agent based.

If you already tried mitogen and ssh pipelining and even increased the forks, I suggest you take a look at the new paradigm of execution environments that comes with Ansible Automation Platform 2.

Regards. 

[Sandip Bhattacharya]

On 01.10.21 23:11, nds.ap...@gmail.com wrote:
> We build a stream file to map the interfaces of all the hosts and then trigger traffic on respective hosts participating. This stream file is around 100 MB.
>
> So just this line in one of the playbook:
>
>   vars_files:
>     - "{{ playbook_dir }}/{{ streams_file_name }}"
> Is taking almost 1-2 hrs of time to run.

I believe there are other vars plugins that you can use? They may work better than the files module. Or write your own.
https://docs.ansible.com/ansible/latest/plugins/vars.html

Alternatively, instead of using vars plugins, you maybe able to use lookup plugins for fetching data, and use faster stores to randomly access data instead of loading it all into memory at the beginning.

- Sandip

--
https://blog.sandipb.net
https://twitter.com/sandipb