Module ping: One host fails with "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh"

[Jörg Kastning]

Hello there,

before executing the following commands I started a ssh-agent bash and added my private key.

Running the command "ansible all -m ping" checks succeeded but one. For one host I got the following error:

host1.example.com | UNREACHABLE! => {
    "changed": false,
    "msg": "SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
    "unreachable": true
}

But when I run the following command in the same shell, the connection is successful:

ssh root@host1.example.com

The section in my inventory file looks like:

[host-group]
host[1:2].example.com
host-test.example.com

Only host1 fails, the other two are checkt successfully.

I use ansible 2.1.0.0  from EPEL-Repo with RHEL 7.2.

Could someone explain to me what's wrong here?

Kind regards,
Joerg

[Alexey Vazhnov]

Did ansible use login root for SSH connection? I think Ansible try to login with your username, not root. Use «ansible_ssh_user» in inventory file, or «--user» in ansible arguments, or «User» in ~/.ssh/config.

[Jörg Kastning]

Yes, ansible use login root for SSH connection. But why should ansible try a different user? I run the command against a group of three hosts and only one fails. The three hosts are from one staging area and configured in the same way (same user, keys, packages).

[Kai Stian Olstad]

On 19. juli 2016 15:19, Jörg Kastning wrote:
> Yes, ansible use login root for SSH connection. But why should ansible try
> a different user?

There are many places user configuration can be done, anyway, have you
tried to add -vvvv to the command to get a more verbose output?

--
Kai Stian Olstad

[Jörg Kastning]

I did not specify a user anywhere. Whether in ssh_config nor in
Inventory nor with the command. Here is the output with -vvvv:
>ansible host-group -vvvv -m ping
Using /etc/ansible/ansible.cfg as config file
Loaded callback minimal of type stdout, v2.0
<host1.example.com> ESTABLISH SSH CONNECTION FOR USER: None
<host1.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r host1.example.com
'/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" && echo
ansible-tmp-1468943476.87-148558126071189="` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" ) &&
sleep 0'"'"''
<host2.example.com> ESTABLISH SSH CONNECTION FOR USER: None
<host2.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r host2.example.com
'/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.87-38319674786550 `" && echo
ansible-tmp-1468943476.87-38319674786550="` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.87-38319674786550 `" ) &&
sleep 0'"'"''
<host-test.example.com> ESTABLISH SSH CONNECTION FOR USER: None
<host-test.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r host-test.example.com
'/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.88-266972946311694 `" && echo
ansible-tmp-1468943476.88-266972946311694="` echo
$HOME/.ansible/tmp/ansible-tmp-1468943476.88-266972946311694 `" ) &&
sleep 0'"'"''
<host1.example.com> PUT /tmp/tmpO6LVmh TO
/root/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189/ping

<host1.example.com> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[host1.example.com]'

<host2.example.com> PUT /tmp/tmpPCIRrD TO
/root/.ansible/tmp/ansible-tmp-1468943476.87-38319674786550/ping

<host2.example.com> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[host2.example.com]'

<host-test.example.com> PUT /tmp/tmp2vmkei TO
/root/.ansible/tmp/ansible-tmp-1468943476.88-266972946311694/ping

<host-test.example.com> SSH: EXEC sftp -b - -C -vvv -o
ControlMaster=auto -o ControlPersist=60s -o
KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r
'[host-test.example.com]'

host1.example.com | UNREACHABLE! => {
"changed": false,
"msg": "SSH Error: data could not be sent to the remote host. Make
sure this host can be reached over ssh",
"unreachable": true
}

<host2.example.com> ESTABLISH SSH CONNECTION FOR USER: None
<host2.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt host2.example.com
'/bin/sh -c '"'"'LANG=de_DE.UTF-8 LC_ALL=de_DE.UTF-8
LC_MESSAGES=de_DE.UTF-8 /usr/bin/python
/root/.ansible/tmp/ansible-tmp-1468943476.87-38319674786550/ping; rm -rf
"/root/.ansible/tmp/ansible-tmp-1468943476.87-38319674786550/" >
/dev/null 2>&1 && sleep 0'"'"''
<host-test.example.com> ESTABLISH SSH CONNECTION FOR USER: None
<host-test.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
ControlPersist=60s -o KbdInteractiveAuthentication=no -o
PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
PasswordAuthentication=no -o ConnectTimeout=10 -o
ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt
host-test.example.com '/bin/sh -c '"'"'LANG=de_DE.UTF-8
LC_ALL=de_DE.UTF-8 LC_MESSAGES=de_DE.UTF-8 /usr/bin/python
/root/.ansible/tmp/ansible-tmp-1468943476.88-266972946311694/ping; rm
-rf "/root/.ansible/tmp/ansible-tmp-1468943476.88-266972946311694/" >
/dev/null 2>&1 && sleep 0'"'"''
host2.example.com | SUCCESS => {
"changed": false,
"invocation": {
"module_args": {
"data": null
},
"module_name": "ping"
},
"ping": "pong"
}
host-test.example.com | SUCCESS => {
"changed": false,
"invocation": {
"module_args": {
"data": null
},
"module_name": "ping"
},
"ping": "pong"
}
[2016.07.19 17:51:17] ~/ansible

I'm somewhat curious about the following line:
ESTABLISH SSH CONNECTION FOR USER: None

I would expect ansible to establish a connection for root. Shouldn't
that be the default behaviour?

[Dick Davies]

On 19 July 2016 at 17:11, Jörg Kastning <joerg.k...@gmail.com> wrote:

> I'm somewhat curious about the following line:
> ESTABLISH SSH CONNECTION FOR USER: None
>
> I would expect ansible to establish a connection for root. Shouldn't
> that be the default behaviour?

No, there isn't a default behaviour. You need to tell it what user to use
(set ansible_ssh_user somewhere).

[Kai Stian Olstad]

Actually it is, it will use the same user as you are logged in as on the
Ansible Control Machine.

--
Kai Stian Olstad

[Kai Stian Olstad]

On 19. juli 2016 18:11, Jörg Kastning wrote:
> <host1.example.com> ESTABLISH SSH CONNECTION FOR USER: None
> <host1.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
> PasswordAuthentication=no -o ConnectTimeout=10 -o
> ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r host1.example.com
> '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
> $HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" && echo
> ansible-tmp-1468943476.87-148558126071189="` echo
> $HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" ) &&
> sleep 0'"'"''

Here Ansible is doing the ssh and setting up the run.
Since it doesn't fail here, the ssh is working.

> <host1.example.com> PUT /tmp/tmpO6LVmh TO
> /root/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189/ping
> <host1.example.com> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o
> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
> PasswordAuthentication=no -o ConnectTimeout=10 -o
> ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[host1.example.com]'
>

> host1.example.com | UNREACHABLE! => {
> "changed": false,
> "msg": "SSH Error: data could not be sent to the remote host. Make
> sure this host can be reached over ssh",
> "unreachable": true
> }

But when Ansible tries to copy over the files it need to run the ping
command it fails, so something is denying sftp to host1.

You could check that ssh is working with the raw module like this:
ansible test1.example.com -m raw -a "hostname"

> I'm somewhat curious about the following line:
> ESTABLISH SSH CONNECTION FOR USER: None

I guess this means that no user is set in Ansible, and that the logged
in user is being used.

> I would expect ansible to establish a connection for root. Shouldn't
> that be the default behaviour?

The default behavior is the user you are logged in as.

--
Kai Stian Olstad

[rup]

We also have a similar issue when the ssh times out.  We've been having that occur occasionally lately, randomly to servers, and are considering setting up a config file to override the 10 second timeout default.  Just a thought anyway.

[Dick Davies]

Sorry Kai, you're quite right.

On 19 July 2016 at 18:40, Kai Stian Olstad

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/578E65F7.3010608%40olstad.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Werner Flamme]

Kai Stian Olstad [19.07.2016 19:55]:

> On 19. juli 2016 18:11, Jörg Kastning wrote:
>> <host1.example.com> ESTABLISH SSH CONNECTION FOR USER: None
>> <host1.example.com> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o
>> ControlPersist=60s -o KbdInteractiveAuthentication=no -o
>> PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o
>> PasswordAuthentication=no -o ConnectTimeout=10 -o
>> ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r host1.example.com
>> '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo
>> $HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" && echo
>> ansible-tmp-1468943476.87-148558126071189="` echo
>> $HOME/.ansible/tmp/ansible-tmp-1468943476.87-148558126071189 `" ) &&
>> sleep 0'"'"''
>
> Here Ansible is doing the ssh and setting up the run.
> Since it doesn't fail here, the ssh is working.

Whenever this problem occurred on one of my hosts, the "Subsystem sftp"
line in /etc/ssh/sshd_config on that host points to an non-existent
sftp-server binary.

HTH,
Werner

--

[Jörg Kastning]

Hi,

At first, thank you all for your help. :-)

When I run the command Kai Stian Olstad suggested I got the following:

ansible host1.example.com -m raw -a "hostname"
host1.example.com | SUCCESS | rc=0 >>
host1

And we have the same situation as Werner. The Subsystem sftp line in /etc/ssh/sshd_config on host1 points do a non-existent sftp-server binary.

Thanks again all of you, for your help.

Regards,
Joerg

[jorge.iglesi...@bbva.com]

Hi! 

I have the same problem but i don´t know the solution.

ssh found, scp found, but the ping said that "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh"

This is the output of  ansible -vvvv  all -m ping: 

Loaded callback minimal of type stdout, v2.0

<192.168.69.190> ESTABLISH SSH CONNECTION FOR USER: None
<192.168.69.190> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.168.69.190 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1488791696.73-239421557112032 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1488791696.73-239421557112032 )" )'
<192.168.69.104> ESTABLISH SSH CONNECTION FOR USER: None
<192.168.69.104> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.168.69.104 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1488791696.74-88043441209635 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1488791696.74-88043441209635 )" )'
<192.168.69.190> PUT /tmp/tmpuh7eyZ TO /root/.ansible/tmp/ansible-tmp-1488791696.73-239421557112032/ping
<192.168.69.190> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[192.168.69.190]'
<192.168.69.190> ESTABLISH SSH CONNECTION FOR USER: None
<192.168.69.190> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt 192.168.69.190 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /root/.ansible/tmp/ansible-tmp-1488791696.73-239421557112032/ping; rm -rf "/root/.ansible/tmp/ansible-tmp-1488791696.73-239421557112032/" > /dev/null 2>&1'
192.168.69.190 | SUCCESS => {

    "changed": false,
    "invocation": {
        "module_args": {
            "data": null
        },
        "module_name": "ping"
    },
    "ping": "pong"
}

<192.168.69.104> PUT /tmp/tmpfmBWoS TO /root/.ansible/tmp/ansible-tmp-1488791696.74-88043441209635/ping
<192.168.69.104> SSH: EXEC sftp -b - -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[192.168.69.104]'
192.168.69.104 | UNREACHABLE! => {
    "changed": false,
    "msg": "ERROR! SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh",
    "unreachable": true
}

TY!

Jorge

"Este mensaje está dirigido de manera exclusiva a su destinatario y puede contener información privada y confidencial. No lo reenvíe, copie o distribuya a terceros que no deban conocer su contenido. En caso de haberlo recibido por error,  rogamos lo notifique al remitente y proceda a su borrado, así como al de cualquier documento que pudiera adjuntarse.

 Por favor tenga en cuenta que los correos enviados vía Internet no permiten garantizar la confidencialidad de los mensajes ni su transmisión de forma íntegra.

 Las opiniones expresadas en el presente correo pertenecen únicamente al remitente y no representan necesariamente la opinión del Grupo BBVA."

 "This message is intended exclusively for the adressee and may contain privileged and confidential information. Please, do not disseminate, copy or distribute it to third parties who should not receive it. In case you have received it by mistake, please inform the sender and delete the message and attachments from your system.

 Please keep in mind that e-mails sent by Internet do not allow to guarantee neither the confidentiality or the integrity of the messages sent."

[Jörg Kastning]

Hi,

cloud you conncet to the host using ssh from a terminal without using the ansible module? Have you checked that the subsystem sftp line in /etc/ssh/sshd_config on your host points to an existent sftp-server binary?

Regards,
Joerg

[balubhai555]

Hello,Jorg,

 

I got the same error. At the time, I did the below and then ssh connection got success and data successfully sent thru the ssh to remote host.

In ansible Control machine execute these commands.

root>ssh-add   //it adds  the identity files to ssh-agent ...i.e add the ~/.ssh/id_rsa.pub file ssh-agent

root>eval "$(ssh-agent)"   // after that will start the ssh-agent to start the ssh connection and execute this for refreshment of hosts...

root>"ansible all -m ping -c ssh -u <<remote-host-username>>"

then the error resolved.  

 Thank you.