FORKS --- not working as expected and also discrepecies in playbook profile task prints

[nds.ap...@gmail.com]

Hi,

We have about 500 hosts and we were testing the ssh connections that run in parallel at a given point before running the actual playbooks as we were hitting bottlenecks.

Ansible version being used: 2.10.5

Its a simple playbook which runs "ls" command on all the hosts.

QUERY 1:

-----------

Set the forks to 200 but i see the increment happening at max 64 ssh sessions/second using a watch command: (RAM 64 gb, 8 cores)

I was under the impression that Forks when set would indicate that many hosts the ssh connections would be created in parallel but thats not the case.

I should be seeing 200 ssh connection at once.

Can you please let me know if i am missing anything here? (pipelining, controlpath everything is set)

QUERY 2:

-------------

Also tried with the mitogen and non mitogen strategy and see the following:

strategy: free
Monday 18 October 2021  00:25:15 -0700 (0:00:05.505)       0:00:17.968 ********
===============================================================================
Run traffic from sources ------------------------------------------------------------------------------------------------------------ 5.51s

Here it says plabook took 5.51 seconds but if you see the time above its 17 minutes

with Mitogen

Monday 18 October 2021  00:26:29 -0700 (0:00:12.428)       0:00:12.529 ********
===============================================================================
Run traffic from sources ----------------------------------------------------------------------------------------------------------- 12.43s
Here the time almost matches.

Why this discrepancies seen for strategy set time does not match? Any reason?

what time should be considered here?

Regards
Aprameya

[Matt Martz]

200 forks will likely not be achievable.  Although we attempt to spawn that many forks based on your setting, you will create too much CPU contention on the single core that manages all of the forks, that you are likely to never see that many.

Somewhere between 50-75 is often the max that most people will ever see.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/26ca4254-f5fe-4a4a-884a-4a2c47aa4c1cn%40googlegroups.com.

--

Matt Martz
@sivel
sivel.net

[nds.ap...@gmail.com]

Hi Matt,

Thanks for the reply. We have 8 cores. Should help in achieving the same?

When we do it with a python script (forking processes) 200 ssh sessions are achievable. Its not working when used through ansible. Is there a way?

Can you also please comment on the second query too?

Regards
Aprameya

[Matt Martz]

The problem isn't really with forking, or with how many cores you have.  The problem is more related to the fact that a single process that is bound to a single core is responsible for spawning forks, monitoring the processes, and handling responses back from all of the forks.

As a result, that single process can reach CPU contention easily with higher fork counts. 

As for why the callback reports inconsistent results is likely due to the fact it was never tested with the free strategy, and not written in a way that would allow it to properly track the timing of the tasks.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/3b55da89-2621-41f7-8f92-fe4ee4101176n%40googlegroups.com.

[nds.ap...@gmail.com]

Hi Matt,

In one of the Tech Talk in youtube from a solution architect of RedHat i saw this slide:

Ref: https://www.youtube.com/watch?v=4kfH0RHrFV8 (running ansible at scale)

Ansible configuration:

4GB RAM 100 Forks

8 GB RAM 200 Forks

16 GM RAM 400 Forks

32 GB RAM 800 Forks

But with our system 64 GB RAM i dont see any where close to this kind of configuration. Infact we only see 30-32 Forks at max happening.

Regards
Aprameya

[Matt Martz]

Those numbers are completely unrealistic.  Memory is hardly the limiting factor. A single fork can often consume 1G of memory itself. 100 forks per 4GB of memory seems to only be the allocation necessary to actually load Python (40MB) itself without doing anything.

The limit is generally the computing power of a single core of your CPU, and your 30 effective forks is what I would consider typical performance.

I could talk about this topic for days. But you are getting roughly what I'd expect.

Maybe watch this presentation from AnsibleFest 2019:

https://www.ansible.com/running-ansible-at-scale-atl-2019

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/3cb212c9-9806-4485-8d19-a615afe0740bn%40googlegroups.com.