The ansible process abruptly stops running

[Ivan Nunes]

Hello,

I have a "playbook" that performs tasks on 15 servers. But sometimes the process simply does not stop running. The process is abruptly stopped.

The playbook runs the night shift by cron. I have no idea how to debug this problem. The ansible logs does not help much. Sometimes the log is stopped in the current task.

Processes on the host machine:

sudo ps aux |grep -i ansible

ansible    639  0.0  0.0   4448   800 ?        Ss   Nov19   0:00 /bin/sh -c /usr/local/bin/run_ansible.sh pdv.yml cod
ansible    640  0.0  0.1  12432  3052 ?        S    Nov19   0:00 /bin/bash /usr/local/bin/run_ansible.sh pdv.yml cod
ansible    650  0.0  1.3  97936 26756 ?        S    Nov19   0:03 python /usr/bin/ansible-playbook -i hosts.cod -l pdvcod pdv.yml --extra-vars shutdown=True
root      1782  0.1  0.3 105632  6400 ?        Ss   14:52   0:00 sshd: ansible [priv]
ansible   1834  0.0  0.1 105632  3248 ?        S    14:52   0:00 sshd: ansible@pts/1 
ansible   1835  0.5  0.2  22604  5300 pts/1    Ss   14:52   0:00 -bash
ansible   1858  0.0  0.0  11724  1672 pts/1    S+   14:53   0:00 grep --color=auto -i ansible
ansible  14575  0.0  0.1  44584  3448 ?        Ss   Nov19   0:02 ssh: /home/ansible/.ansible/cp/ansible-ssh-172.16.84.77-22-ansible [mux]
ansible  21564  0.0  1.2 802440 25384 ?        Sl   Nov19   0:00 python /usr/bin/ansible-playbook -i hosts.cod -l pdvcod pdv.yml --extra-vars shutdown=True
ansible  21572  0.0  1.1  97648 22948 ?        S    Nov19   0:18 python /usr/bin/ansible-playbook -i hosts.cod -l pdvcod pdv.yml --extra-vars shutdown=True
ansible  21573  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21575  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21577  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21578  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21580  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21581  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21582  0.0  0.0      0     0 ?        Z    Nov19   0:00 [python] <defunct>
ansible  21640  0.0  0.0  12768  1876 ?        S    Nov19   0:00 sshpass -d4 ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o ConnectTimeout=10 172.16.84.77 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=dpdtgnihdmtnmcoyihilkcsgroyhwvrv] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-dpdtgnihdmtnmcoyihilkcsgroyhwvrv; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command; rm -rf /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/ >/dev/null 2>&1'"'"''
ansible  21641  0.0  0.2  44016  4752 pts/5    Ss+  Nov19   0:00 ssh -C -tt -v -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/home/ansible/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o GSSAPIAuthentication=no -o PubkeyAuthentication=no -o ConnectTimeout=10 172.16.84.77 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=dpdtgnihdmtnmcoyihilkcsgroyhwvrv] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-dpdtgnihdmtnmcoyihilkcsgroyhwvrv; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command; rm -rf /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/ >/dev/null 2>&1'"'"''
root     32588  0.0  0.3 105632  6448 ?        Ss   11:52   0:00 sshd: ansible [priv]
ansible  32640  0.0  0.1 105768  3840 ?        S    11:52   0:00 sshd: ansible@pts/0 

Processes on client servers:

sudo ps aux |grep -i ansible

root      6338  0.0  0.3  10040  3304 ?        Ss   Nov19   0:01 sshd: ansible [priv]
ansible   6343  0.0  0.2  10180  2224 ?        S    Nov19   0:08 sshd: ansible@pts/0 
ansible   7131  0.0  0.0   2232   496 pts/0    Ss+  Nov19   0:00 /bin/sh -c sudo -k && sudo -H -S -p "[sudo via ansible, key=dpdtgnihdmtnmcoyihilkcsgroyhwvrv] password: " -u root /bin/sh -c 'echo BECOME-SUCCESS-dpdtgnihdmtnmcoyihilkcsgroyhwvrv; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command; rm -rf /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/ >/dev/null 2>&1'
root      7133  0.0  0.1   5732  1656 pts/0    S+   Nov19   0:00 sudo -H -S -p [sudo via ansible, key=dpdtgnihdmtnmcoyihilkcsgroyhwvrv] password:  -u root /bin/sh -c echo BECOME-SUCCESS-dpdtgnihdmtnmcoyihilkcsgroyhwvrv; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command; rm -rf /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/ >/dev/null 2>&1
root      7134  0.0  0.0   2232   288 pts/0    S+   Nov19   0:00 /bin/sh -c echo BECOME-SUCCESS-dpdtgnihdmtnmcoyihilkcsgroyhwvrv; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command; rm -rf /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/ >/dev/null 2>&1
root      7135  0.0  0.6  12544  6084 pts/0    S+   Nov19   0:09 /usr/bin/python /home/ansible/.ansible/tmp/ansible-tmp-1447979713.35-99473851710552/command

ansible-playbook 1.9.2

Best regards.