ansible ssh issue

[William Muriithi]

Hello,

This is not an ansible issue to be frank but I am hoping someone here have seen it before and can offer some guidance.  Note, I can manually ssh to the system, but ansible can't.

I have a system that was previously working fine as a managed host.  For the last two days though, ansible isn't able to manage it.  Ansible connection fails at setup stage.  I have the ssh logs for both the client and the server, but can't seem to glean the root cause from it. Hoping a second pair of eyes might pick what I am missing

Below are the client log and server logs consecutively.  Does anything stand out from this logs?

Client:

TASK [setup] *******************************************************************

Using module file /usr/lib/python2.6/site-packages/ansible/modules/core/system/setup.py

<arsenic.eng.example.com> ESTABLISH SSH CONNECTION FOR USER: root

<arsenic.eng.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r arsenic.eng.example.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''

<arsenic.eng.example.com> ESTABLISH SSH CONNECTION FOR USER: root

<arsenic.eng.example.com> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/tmp/ansible-ssh-%h-%p-%r arsenic.eng.example.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /root/.ansible/tmp/ansible-tmp-1492557862.94-75489720546518 `" && echo ansible-tmp-1492557862.94-75489720546518="` echo /root/.ansible/tmp/ansible-tmp-1492557862.94-75489720546518 `" ) && sleep 0'"'"''

fatal: [arsenic.eng.example.com]: UNREACHABLE! => {

    "changed": false,

    "msg": "Failed to connect to the host via ssh: OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\nControl socket connect(/tmp/ansible-ssh-arsenic.eng.example.com-22-root): Connection refused\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to arsenic.eng.example.com [192.168.20.154] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 192.168.20.154 port 22: Connection refused\r\nssh: connect to host arsenic.eng.example.com port 22: Connection refused\r\n",

    "unreachable": true

}

        to retry, use: --limit @/etc/ansible/site.retry

PLAY RECAP *********************************************************************

arsenic.eng.example.com : ok=0    changed=0    unreachable=1    failed=0

SSH Server:

debug2: fd 7 setting O_NONBLOCK

debug1: server_init_dispatch_20

debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768

debug1: input_session_request

debug1: channel 0: new [server-session]

debug2: session_new: allocate (allocated 0 max 10)

debug3: session_unused: session id 0 unused

debug1: session_new: session 0

debug1: session_open: channel 0

debug1: session_open: session 0: link with channel 0

debug1: server_input_channel_open: confirm session

debug1: server_input_global_request: rtype no-more-...@openssh.com want_reply 0

debug3: Wrote 48 bytes for a total of 2261

debug1: server_input_channel_req: channel 0 request env reply 0

debug1: session_by_channel: session 0 channel 0

debug1: session_input_channel_req: session 0 req env

debug2: Setting env 0: LANG=en_US.UTF-8

debug1: server_input_channel_req: channel 0 request exec reply 1

debug1: session_by_channel: session 0 channel 0

debug1: session_input_channel_req: session 0 req exec

debug2: fd 3 setting TCP_NODELAY

debug2: fd 10 setting O_NONBLOCK

debug2: fd 9 setting O_NONBLOCK

debug2: fd 12 setting O_NONBLOCK

debug3: Wrote 80 bytes for a total of 2341

debug2: channel 0: read 41 from efd 12

debug2: channel 0: rwin 2097152 elen 41 euse 1

debug2: channel 0: sent ext data 41

debug3: Wrote 80 bytes for a total of 2421

debug2: channel 0: read 34 from efd 12

debug2: channel 0: rwin 2097111 elen 34 euse 1

debug2: channel 0: sent ext data 34

debug3: Wrote 64 bytes for a total of 2485

debug2: channel 0: read 337 from efd 12

debug2: channel 0: rwin 2097077 elen 337 euse 1

debug2: channel 0: sent ext data 337

debug2: channel 0: read 175 from efd 12

debug3: Wrote 224 bytes for a total of 2709

debug2: channel 0: rwin 2096740 elen 175 euse 1

debug2: channel 0: sent ext data 175

debug3: Wrote 112 bytes for a total of 2821

debug2: channel 0: read 45 from efd 12

debug2: channel 0: rwin 2096565 elen 45 euse 1

debug2: channel 0: sent ext data 45

debug3: Wrote 80 bytes for a total of 2901

debug3: Wrote 48 bytes for a total of 2949

debug1: Received SIGCHLD.

debug1: session_by_pid: pid 930

debug1: session_exit_message: session 0 channel 0 pid 930

debug2: channel 0: request exit-status confirm 0

debug1: session_exit_message: release channel 0

debug2: channel 0: write failed

debug2: channel 0: close_write

debug2: channel 0: send eow

debug2: channel 0: output open -> closed

debug2: channel 0: read<=0 rfd 10 len 0

debug2: channel 0: read failed

debug2: channel 0: close_read

debug2: channel 0: input open -> drain

debug2: channel 0: read 0 from efd 12

debug2: channel 0: closing read-efd 12

debug2: channel 0: ibuf empty

debug2: channel 0: send eof

debug2: channel 0: input drain -> closed

debug2: channel 0: send close

debug2: notify_done: reading

debug3: Wrote 160 bytes for a total of 3109

debug3: channel 0: will not send data after close

debug2: channel 0: rcvd close

Received disconnect from 192.168.20.221: 11: disconnected by user

debug1: do_cleanup

debug1: PAM: cleanup

debug1: PAM: closing session

debug1: PAM: deleting credentials

debug3: PAM: sshpam_thread_cleanup entering

                                                           [FAILED]

[root@arsenic ~]#

Thank in advance

Regards,

William