ansible cmd run takes over 40s per server

[Filip Slunecko]

Hi,

I installed a fresh Ansible from git hub on CentOS 6.3 and run a simple command takes over 40s with CPU 100% for Ansible.

This is a line from cProfile(http://pastebin.com/mX4qXsiN).

 11523   40.093    0.003   54.893    0.005 hostkeys.py:181(lookup)

It seems to that Ansible is doing a lot of lookups.
The same version works on my box without an issue(Fedora 20).

Command:
ansible/bin/ansible XXX -u root --private-key /root/.ssh/id_rsa -i inventory/ -m shell -a echo test -vvv

I ran Ansible on CentOS 6.3 boxes and I've never seen like this before.

Thanks

[Michael DeHaan]

This is pretty unusual.

This is a function inside the paramiko source code that does not (at least obviously does not) do anything like DNS lookups.

I'm at a bit of a loss, perhaps you can debug further?

Fedora 20 has a new enough SSH so the default connection type is "ssh" not paramiko there, curious if you have the same with "-c paramiko" on that box (and using the same key?)

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1f88ea77-ec8a-47ce-b97e-a9ac10c7fcc0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Filip Slunecko]

ansible XX -c paramiko -u root --private-key .ssh/root_id -i inventory/ -m shell -a echo test

it's fast on F20.

We have a big /etc/hosts file (1527 hosts), but I copied the host file to my box and it doesn't make a difference.

[Filip Slunecko]

I deployed on different box (the same version) and it works just fine.
Not sure what is the issue on the first one.

[Adam Heath]

nscd turned off? ldap in use?

[Michael DeHaan]

Paramiko's host file handling has been noted to be a performance problem in some cases where host files get large.

We haven't exactly tracked down *why*.

(I really really wish RHEL/CentOS would update their OpenSSH... can't wait for RHEL 7 -- and less and less folks will need paramiko)

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-project+unsubscribe@googlegroups.com.
To post to this group, send email to ansible-project@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/5323AEF4.8040403%40brainfood.com.

[Filip Slunecko]

So the reason is that my known_hosts file has over 5k records.
I think that Paramiko is for looping for each host in known_hosts and /etc/hosts file.
In my case it's around 800k checks.

On Saturday, March 15, 2014 7:20:06 AM UTC-7, Michael DeHaan wrote:

Paramiko's host file handling has been noted to be a performance problem in some cases where host files get large.

We haven't exactly tracked down *why*.

(I really really wish RHEL/CentOS would update their OpenSSH... can't wait for RHEL 7 -- and less and less folks will need paramiko)

On Fri, Mar 14, 2014 at 9:37 PM, Adam Heath <ad...@brainfood.com> wrote:

nscd turned off?  ldap in use?

On 03/14/2014 07:36 PM, Filip Slunecko wrote:

I deployed on different box (the same version) and it works just fine.
Not sure what is the issue on the first one.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

[Michael DeHaan]

Paramiko's processing of known_hosts is rather poor for large host files.

I'd recommend using the SSH transport if you can.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/1b6ac1e9-3cfe-4e50-8a3d-50be108f2263%40googlegroups.com.