Unable to establish SSH connection using Ansible

[cmacrae]

I'm working on a role to deploy my Ansible setup (sort of a 'master node' configuration - where I execute Ansible from). 

Issue Description

I've encountered a rather strange issue: I'm able to SSH as the user I'm running Ansible as to systems within my infrastructure using key pair authentication, however; when using ansible from the shell, it seems it fails SSH connection. 

Debugging this reveals (what I feel is the relevant message):

debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /opt/ansible/.ssh/id_rsa
debug3: sign_and_send_pubkey: RSA SHA256:GOyUuuByt5CVOiBABZy+O2AKtfBzbmOIvp7Fj6A2ljA
debug2: we sent a publickey packet, wait for reply

Following this, it attempts several other private key locations, then drops to keyboard-interactive, then fails.

What makes this strange is the last line:

debug2: we sent a publickey packet, wait for reply


As it doesn't seem there is any response - yet I'm able to SSH in fine from the shell, as that same user, with that private key:

[ansible@test ~/etc]$ ssh bunny.cmacr.ae -l root -i /opt/ansible/.ssh/id_rsa
[root@bunny ~]#

The above is exhibited when trying to execute any functions - testing using 'ping' and 'setup' like so:

ansible bunny.cmacr.ae -m ping

ansible bunny.cmacr.ae -m setup

I've also tried explicitly passing the 'remote_user' value as an argument, just in case it was not evaluating this from my inventory variables (I have it set for all systems as 'root'), like so:

ansible -e remote_user=root bunny.cmacr.ae -m setup

But I'm afraid it's the same results.

Environment Details

I'm running Ansible from Git, version 2.0.0:

[ansible@test ~/etc]$ ansible --version
ansible 2.0.0 (devel 42cfacf83b) last updated 2015/07/28 20:45:45 (GMT +000)
  lib/ansible/modules/core: (detached HEAD a46b3a4dc4) last updated 2015/07/28 20:45:51 (GMT +000)
  lib/ansible/modules/extras: (detached HEAD e30d8b84fe) last updated 2015/07/28 20:45:56 (GMT +000)
  v1/ansible/modules/core: (detached HEAD f8d8af17cd) last updated 2015/07/28 20:46:00 (GMT +000)
  v1/ansible/modules/extras: (detached HEAD 495ad450e5) last updated 2015/07/28 20:46:04 (GMT +000)
  config file = /opt/ansible/etc/ansible.cfg
  configured module search path = /opt/ansible/git/lib/ansible

This is running in a SmartOS zone (base64 14.3.0) - I've been operating from the same OS environment using Ansible 1.9.1 fine for quite some time.

I have 'remote_user' set to 'root' for all systems within my inventory.

Full Debug Output

Here's the full debug log from running 'ansible -vvvv bunny.cmacr.ae -m setup':

Using /opt/ansible/etc/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<bunny.cmacr.ae> ESTABLISH SSH CONNECTION FOR USER: ansible
<bunny.cmacr.ae> EXEC ssh -C -tt -vvv -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/tmp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 bunny.cmacr.ae mkdir -p "$HOME/.ansible/tmp/ansible-tmp-1438166197.14-198606753001666" && chmod a+rx "$HOME/.ansible/tmp/ansible-tmp-1438166197.14-198606753001666" && echo "$HOME/.ansible/tmp/ansible-tmp-1438166197.14-198606753001666"
bunny.cmacr.ae | FAILED! => {
    "failed": true,
    "msg": "ERROR! SSH encountered an unknown error. The output was:
OpenSSH_6.9p1, OpenSSL 1.0.2d 9 Jul 2015
debug1: Reading configuration data /opt/local/etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket \"/tmp/ansible-ssh-bunny.cmacr.ae-22-ansible\" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to bunny.cmacr.ae [10.19.3.19] port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: fd 4 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 10000 ms remain after connect
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /opt/ansible/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version Sun_SSH_1.5
debug1: no match: Sun_SSH_1.5
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to bunny.cmacr.ae:22 as 'ansible'
debug3: hostkeys_foreach: reading file \"/opt/ansible/.ssh/known_hosts\"
debug3: record_hostkey: found key type RSA in file /opt/ansible/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from bunny.cmacr.ae
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-...@openssh.com,ssh-rsa-...@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve255...@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-...@openssh.com,ssh-rsa-...@openssh.com,ssh-rsa,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ecdsa-sha2-nis...@openssh.com,ssh-ed2551...@openssh.com,ssh-dss-...@openssh.com,ssh-dss-...@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnda...@lysator.liu.se
debug2: kex_parse_kexinit: chacha20...@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes12...@openssh.com,aes25...@openssh.com,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijnda...@lysator.liu.se
debug2: kex_parse_kexinit: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-m...@openssh.com,hmac-ripe...@openssh.com,hmac-sha...@openssh.com,hmac-md...@openssh.com,hmac-md5,hmac-ripemd160,hmac-ri...@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-...@openssh.com,umac-1...@openssh.com,hmac-sha...@openssh.com,hmac-sha...@openssh.com,hmac-s...@openssh.com,uma...@openssh.com,umac...@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-m...@openssh.com,hmac-ripe...@openssh.com,hmac-sha...@openssh.com,hmac-md...@openssh.com,hmac-md5,hmac-ripemd160,hmac-ri...@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zl...@openssh.com,zlib,none
debug2: kex_parse_kexinit: zl...@openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour128,arcfour256,arcfour
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: none,zlib
debug2: kex_parse_kexinit: af-ZA,ar-AE,ar-BH,ar-DZ,ar-EG,ar-IQ,ar-JO,ar-KW,ar-LB,ar-LY,ar-MA,ar-OM,ar-QA,ar-SA,ar-TN,ar-YE,as-IN,az-AZ,be-BY,bg-BG,bn-BD,bn-IN,bo-CN,bo-IN,bs-BA,ca-ES,cs-CZ,da-DK,de-AT,de-BE,de-CH,de-DE,de-LI,de-LU,el-CY,el-GR,en-AU,en-BW,en-BZ,en-CA,en-GB,en-HK,en-IE,en-IN,en-JM,en-MH,en-MT,en-NA,en-NZ,en-PH,en-PK,en-SG,en-TT,en-US,en-ZA,en-ZW,es-AR,es-BO,es-CL,es-CO,es-CR,es-DO,es-EC,es-ES,es-GQ,es-GT,es-HN,es-MX,es-NI,es-PA,es-PE,es-PR,es-PY,es-SV,es-US,es-UY,es-VE,et-EE,fi-FI,fr-BE,fr-CA,fr-CF,fr-CH,fr-FR,fr-GN,fr-LU,fr-MC,fr-MG,fr-ML,fr-NE,fr-SN,ga-IE,gu-IN,he-IL,hi-IN,hr-HR,hu-HU,hy-AM,id-ID,ii-CN,is-IS,it-CH,it-IT,ja-JP,ka-GE,kk-KZ,km-KH,kn-IN,ko-KR,lt-LT,lv-LV,mk-MK,ml-IN,mn-CN,mn-MN,mr-IN,ms-MY,mt-MT,nb-NO,ne-IN,ne-NP,nl-BE,nl-NL,nn-NO,or-IN,pa-IN,pa-PK,pl-PL,pt-BR,pt-GW,pt-MZ,pt-PT,ro-MD,ro-RO,ru-MD,ru-RU,ru-UA,sa-IN,si-LK,sk-SK,sl-SI,sq-AL,sr-BA,sr-ME,sr-RS,sv-FI,sv-SE,ta-IN,ta-LK,te-IN,th-TH,tr-TR,ug-CN,uk-UA,ur-IN,ur-PK,vi-VN,zh-CN,zh-HK,zh-MO,zh-SG,i-default,zh-TW
debug2: kex_parse_kexinit: af-ZA,ar-AE,ar-BH,ar-DZ,ar-EG,ar-IQ,ar-JO,ar-KW,ar-LB,ar-LY,ar-MA,ar-OM,ar-QA,ar-SA,ar-TN,ar-YE,as-IN,az-AZ,be-BY,bg-BG,bn-BD,bn-IN,bo-CN,bo-IN,bs-BA,ca-ES,cs-CZ,da-DK,de-AT,de-BE,de-CH,de-DE,de-LI,de-LU,el-CY,el-GR,en-AU,en-BW,en-BZ,en-CA,en-GB,en-HK,en-IE,en-IN,en-JM,en-MH,en-MT,en-NA,en-NZ,en-PH,en-PK,en-SG,en-TT,en-US,en-ZA,en-ZW,es-AR,es-BO,es-CL,es-CO,es-CR,es-DO,es-EC,es-ES,es-GQ,es-GT,es-HN,es-MX,es-NI,es-PA,es-PE,es-PR,es-PY,es-SV,es-US,es-UY,es-VE,et-EE,fi-FI,fr-BE,fr-CA,fr-CF,fr-CH,fr-FR,fr-GN,fr-LU,fr-MC,fr-MG,fr-ML,fr-NE,fr-SN,ga-IE,gu-IN,he-IL,hi-IN,hr-HR,hu-HU,hy-AM,id-ID,ii-CN,is-IS,it-CH,it-IT,ja-JP,ka-GE,kk-KZ,km-KH,kn-IN,ko-KR,lt-LT,lv-LV,mk-MK,ml-IN,mn-CN,mn-MN,mr-IN,ms-MY,mt-MT,nb-NO,ne-IN,ne-NP,nl-BE,nl-NL,nn-NO,or-IN,pa-IN,pa-PK,pl-PL,pt-BR,pt-GW,pt-MZ,pt-PT,ro-MD,ro-RO,ru-MD,ru-RU,ru-UA,sa-IN,si-LK,sk-SK,sl-SI,sq-AL,sr-BA,sr-ME,sr-RS,sv-FI,sv-SE,ta-IN,ta-LK,te-IN,th-TH,tr-TR,ug-CN,uk-UA,ur-IN,ur-PK,vi-VN,zh-CN,zh-HK,zh-MO,zh-SG,i-default,zh-TW
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: kex: server->client aes128-ctr hmac-sha1 zlib
debug1: kex: client->server aes128-ctr hmac-sha1 zlib
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<7680<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 2017/4095
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:kQ55qytyt1/NNW3vi0cKIBRRSFYiElZxOya2t1UACok
debug3: hostkeys_foreach: reading file \"/opt/ansible/.ssh/known_hosts\"
debug3: record_hostkey: found key type RSA in file /opt/ansible/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from bunny.cmacr.ae
debug3: hostkeys_foreach: reading file \"/opt/ansible/.ssh/known_hosts\"
debug3: record_hostkey: found key type RSA in file /opt/ansible/.ssh/known_hosts:3
debug3: load_hostkeys: loaded 1 keys from 10.19.3.19
debug1: Host 'bunny.cmacr.ae' is known and matches the RSA host key.
debug1: Found key in /opt/ansible/.ssh/known_hosts:3
debug2: bits set: 2038/4095
debug2: set_newkeys: mode 1
debug1: Enabling compression at level 6.
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /opt/ansible/.ssh/id_rsa (0),
debug2: key: /opt/ansible/.ssh/id_dsa (0),
debug2: key: /opt/ansible/.ssh/id_ecdsa (0),
debug2: key: /opt/ansible/.ssh/id_ed25519 (0),
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /opt/ansible/.ssh/id_rsa
debug3: sign_and_send_pubkey: RSA SHA256:GOyUuuByt5CVOiBABZy+O2AKtfBzbmOIvp7Fj6A2ljA
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /opt/ansible/.ssh/id_dsa
debug3: no such identity: /opt/ansible/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /opt/ansible/.ssh/id_ecdsa
debug3: no such identity: /opt/ansible/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /opt/ansible/.ssh/id_ed25519
debug3: no such identity: /opt/ansible/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,keyboard-interactive)."}

Last Note

Thanks in advance for any help on this! Quite curious as to what's going on.
If I can provide any more detail, please; let me know.

[Andrea Tartaglia]

Hi,

From the full debug you can see ansible is using the “ansible” user to try the login. 

Have you tried setting remote_user in the ansible.cfg [ /opt/ansible/etc/ansible.cfg in your case ]? 

Ref. https://docs.ansible.com/ansible/intro_configuration.html#remote-user

-

A.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/95729739-b5e6-421c-8d29-61384bcf1b3e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Markham]

The problem looks to be that you're trying to ssh as user 'ansible' when running the ansible command, instead of user 'root'. Using '-e remote_user=root' sets a variable, not a configuration option.

Try specifying remote_user=root in your ansible.cfg, setting environment variable $ANSIBLE_REMOTE_USER=root, or using ansible_ssh_user in your inventory.

[cmacrae]

Ah! Not sure how I didn't catch that in the debug output!
Thank you very much, after setting this in ansible.cfg, this now works.

Cheers, I appreciate the help :)

[cmacrae]

Cheers Paul! This is now sorted - really appreciate the help - I'm not sure how I missed that in the debug output...