Problem connecting to host behind jump server when host set dynamically

[Linus Sunde]

I have a problem where a playbook using add_host fails unless the host (which sits behind a jump server) has already been connected to by running another playbook.

Can anyone explain to me why the following happens, and hopefully how to fix it?

I have two playbooks and a hosts file:

dynamic-host.yml:

- hosts: localhost

  tasks:

    - name: add remote host

      add_host: name=remote groups=hosts

- hosts: hosts

  name: hello world

  tasks:

    - shell: echo "hello world!"

static-host.yml:

- hosts: remote

  name: hello world

  tasks:

    - shell: echo "hello world!"

hosts:

[remote]

remote ansible_host=10.0.0.1 ansible_port=22 ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q 10.1.0.1"'

If I run the dynamic playbook, it fails:

ansible-playbook dynamic-host.yml -i hosts -vvvv

...

fatal: [remote]: UNREACHABLE! => {"changed": false, "msg": "ERROR! SSH encountered an unknown error. The output was:\nOpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/lsunde/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/lsunde/.ansible/cp/ansible-ssh-10.0.0.1-22-lsunde\" does not exist\r\ndebug2: resolving \"10.0.0.1\" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to 10.0.0.1 [10.0.0.1] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 10.0.0.1 port 22: Connection timed out\r\nssh: connect to host 10.0.0.1 port 22: Connection timed out\r\n", "unreachable": true}

PLAY RECAP *********************************************************************

localhost                  : ok=2    changed=1    unreachable=0    failed=0   

remote                     : ok=0    changed=0    unreachable=1    failed=0   

If I then run the static playbook, it succeeds:

ansible-playbook static-host.yml -i hosts -vvvv

...

PLAY RECAP *********************************************************************

remote                     : ok=2    changed=1    unreachable=0    failed=0   

If t then run the dynamic playbook again (after running the static one), it also succeeds:

ansible-playbook dynamic-host.yml -i hosts -vvvv

...

PLAY RECAP *********************************************************************

localhost                  : ok=2    changed=1    unreachable=0    failed=0   

remote                     : ok=2    changed=1    unreachable=0    failed=0   

Any ideas why this is happening?

[Johannes Kastl]

On 06.02.17 15:34 Linus Sunde wrote:

> debug1: Control socket
> \"/home/lsunde/.ansible/cp/ansible-ssh-10.0.0.1-22-lsunde\" does
> not exist

I am not sure if this is related, but apparently the socket, by which
the multiplexing of the ssh connection should work, does not exist.

Maybe try this in ansible.cfg and see if it fixes your problem:

[ssh_connection]
control_path = %(directory)s/%%C

Johannes

[Linus Sunde]

/home/lsunde/temp/ansible-dynamic-host/ansible.cfg:

[defaults]

timeout = 60

[ssh_connection] 

control_path = %(directory)s/%%C 

output:

$ ansible-playbook dynamic-host.yml -i hosts -vvvv

Using /home/lsunde/temp/ansible-dynamic-host/ansible.cfg as config file

...

TASK [setup] *******************************************************************

<10.0.0.1> ESTABLISH SSH CONNECTION FOR USER: None

<10.0.0.1> SSH: EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o ControlPath=/home/lsunde/.ansible/cp/%C -tt 10.0.0.1 '( umask 22 && mkdir -p "$( echo $HOME/.ansible/tmp/ansible-tmp-1486395620.34-123337012460348 )" && echo "$( echo $HOME/.ansible/tmp/ansible-tmp-1486395620.34-123337012460348 )" )'

fatal: [remote]: UNREACHABLE! => {"changed": false, "msg": "ERROR! SSH encountered an unknown error. The output was:\nOpenSSH_7.2p2 Ubuntu-4ubuntu2.1, OpenSSL 1.0.2g  1 Mar 2016\r\ndebug1: Reading configuration data /home/lsunde/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/home/lsunde/.ansible/cp/c6f5791e0e44a3064cdd690f21bcbf0f354fc3a4\" does not exist\r\ndebug2: resolving \"10.0.0.1\" port 22\r\ndebug2: ssh_connect_direct: needpriv 0\r\ndebug1: Connecting to 10.0.0.1 [10.0.0.1] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: connect to address 10.0.0.1 port 22: Connection timed out\r\nssh: connect to host 10.0.0.1 port 22: Connection timed out\r\n", "unreachable": true}

[Johannes Kastl]

On 06.02.17 15:34 Linus Sunde wrote:

> I have two playbooks and a hosts file:

> Any ideas why this is happening?

Second guess:
By adding the hosts via add_hosts you overwrite your definition in the
hosts file. And thus none of your options are set anymore. Note that
this is just in memory, so your hosts file wont get changed.

If you call the static play, it uses the hosts file with parameters.

Solution:

A) Move the parameters to the host_vars for this host.

B) Call add_hosts with the parameters.

Johannes

[Linus Sunde]

Ah! That seems to be it. The following works:

ansible-playbook dynamic-host2.yml -i /dev/null -vvvv

dynamic-host2.yml:

- hosts: localhost

  tasks:

    - name: add remote host

      add_host: 

        name: remote

        ansible_host: 10.0.0.1

        ansible_port: 22

        ansible_ssh_common_args: '-o ProxyCommand="ssh -o ForwardAgent=yes -p 22022 -W %h:%p -q 10.1.0.1"'

- hosts: remote

  name: hello world

  tasks:

    - shell: echo "hello world!"