Ansible copy module hangs on CentOS 7

[Martin Ilchev]

Hi Guys,

I have a playbook to copy DNS zone files from local directory to a remote one and restart the named service. The playbook is as follows:

---

# This playbook upgrades zone data files from the control machine to all DNS servers 1 machine at a time

- name: Upgrade zone data files. Reload named-chroot if required

  hosts: dnsservers

  gather_facts: no

  sudo: yes

  serial: 1

  tasks:

    - name: copy dns zone files from control machine to DNS servers

      copy: src=zones/ dest=/var/named owner=root group=named mode=0440

      register: namedzones

    - name: restart dns service if required

      service: name=named-chroot state=restarted

      when: namedzones.changed

I test run the playbook as follows:

ansible-playbook -vv dns-copy-zones.yml --ask-become-pass --check

The output is:

SUDO password: 

PLAY [Upgrade zone data files. Reload named-chroot if required] *** 

TASK: [copy dns zone files from control machine to DNS servers] ***

fatal: [192.168.16.195] => ssh connection error while waiting for sudo password prompt

FATAL: all hosts have already failed -- aborting                                                                                                                                               

PLAY RECAP ******************************************************************** 

           to retry, use: --limit @/home/martin/dns-copy-zones.retry

192.168.16.195             : ok=0    changed=0    unreachable=1    failed=0

When I increase the verbosity I can see 5-6 files are copied then copy hangs locally on transferring the next file and eventually times out.

On the remote machine if I tail /var/log/secure I can see all the files up to the last one going through.

The machine I run the playbook from and the destination machine are both CentOS 7. The version is ansible 1.9.2.

Below is the more verbose output I have highlighted what I think are the important messages. I am only including the last file to go through successfully and the next one to hang (let me know if you require the full output):

ON CONTROL MACHINE:

<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=zasgrvyvetzjqebxkfgjmaeohbealypu] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r /var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [ -d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.empty && exit 0; (python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1(); afile = open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c '"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile = open("'"'"'"'"'"'"'"'"'/var/named/named.empty'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo '"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.empty)'"'"''

<192.168.16.195> EXEC ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/martin/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 192.168.16.195 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=bnbotucjqsepetntcsddztjuluwdvzam] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-bnbotucjqsepetntcsddztjuluwdvzam; rc=flag; [ -r /var/named/named.localhost ] || rc=2; [ -f /var/named/named.localhost ] || rc=1; [ -d /var/named/named.localhost ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc} "/var/named/named.localhost && exit 0; (python -c '"'"'"'"'"'"'"'"'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1(); afile = open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (python -c '"'"'"'"'"'"'"'"'import sha; BLOCKSIZE = 65536; hasher = sha.sha(); afile = open("'"'"'"'"'"'"'"'"'/var/named/named.localhost'"'"'"'"'"'"'"'"'", "rb") buf = afile.read(BLOCKSIZE) while len(buf) > 0: hasher.update(buf) buf = afile.read(BLOCKSIZE) afile.close() print(hasher.hexdigest())'"'"'"'"'"'"'"'"' 2>/dev/null) || (echo '"'"'"'"'"'"'"'"'0 '"'"'"'"'"'"'"'"'/var/named/named.localhost)'"'"''

ON THE DESTINATION MACHINE (output in /var/log/secure):

Jul 16 12:48:00 dominus sudo: martin : TTY=pts/2 ; PWD=/home/martin ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-zasgrvyvetzjqebxkfgjmaeohbealypu; rc=flag; [ -r /var/named/named.empty ] || rc=2; [ -f /var/named/named.empty ] || rc=1; [ -d /var/named/named.empty ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc}  "/var/named/named.empty && exit 0; (python -c 'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();

afile = open("'/var/named/named.empty'", "rb")

buf = afile.read(BLOCKSIZE)

while len(buf) > 0:

        hasher.update(buf)

        buf = afile.read(BLOCKSIZE)

afile.close()

print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha; BLOCKSIZE = 65536; hasher = sha.sha();

afile = open("'/var/named/named.empty'", "rb")

buf = afile.read(BLOCKSIZE)

while len(buf) > 0:

        hasher.update(buf)

        buf = afile.read(BLOCKSIZE)

afile.close()

print(hasher.hexdigest())' 2>/dev/null) || (echo '0  '/var/named/named.empty)

Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): conversation failed

Jul 16 12:51:05 dominus sudo: pam_unix(sudo:auth): auth could not identify password for [martin]

Jul 16 12:51:05 dominus sudo: martin : 1 incorrect password attempt ; TTY=pts/0 ; PWD=/home/martin ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-akpnodnhiuvxfpmhtrqwonulsepeazkv; rc=flag; [ -r /var/named/db.sub1.testdns1.com ] || rc=2; [ -f /var/named/db.sub1.testdns1.com ] || rc=1; [ -d /var/named/db.sub1.testdns1.com ] && rc=3; python -V 2>/dev/null || rc=4; [ x"$rc" != "xflag" ] && echo "${rc}  "/var/named/db.sub1.testdns1.com && exit 0; (python -c 'import hashlib; BLOCKSIZE = 65536; hasher = hashlib.sha1();

afile = open("'/var/named/db.sub1.testdns1.com'", "rb")

buf = afile.read(BLOCKSIZE)

while len(buf) > 0:

        hasher.update(buf)

        buf = afile.read(BLOCKSIZE)

afile.close()

print(hasher.hexdigest())' 2>/dev/null) || (python -c 'import sha; BLOCKSIZE = 65536; hasher = sha.sha();

afile = open("'/var/named/db.sub1.testdns1.com'", "rb")

buf = afile.read(BLOCKSIZE)

while len(buf) > 0:

        hasher.update(buf)

        buf =

Jul 16 12:51:05 dominus sudo: martin : (command continued) afile.read(BLOCKSIZE)

afile.close()

print(hasher.hexdigest())' 2>/dev/null) || (echo '0  '/var/named/db.sub1.testdns1.com)

Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): conversation failed

Jul 16 12:53:25 dominus sudo: pam_unix(sudo:auth): auth could not identify password for [martin]

Jul 16 12:54:25 dominus sshd[21162]: Received disconnect from 192.168.16.55: 11: disconnected by user

Jul 16 12:54:25 dominus sshd[21159]: pam_unix(sshd:session): session closed for user martin

Apologies for the massive post however let me know if you require more information.

Kind Regards,

Martin

[Martin Ilchev]

After a lot of digging I was able to find the source of the problem. There was an sshd_config.j2 template that was taken from Debian based system. In it the default settings for Subsystem is different than CentOS

# Debian

Subsystem sftp /usr/lib/openssh/sftp-server

#Centos

Subsystem sftp /usr/libexec/openssh/sftp-server

so lib vs libexec after restoring the config file back to normal the playbook completes without errors.

Hopefully this will save someone else a couple of days.

M