Ansible bug - IP of servers change to 127.0.0.1 mid way through playbook

[Michael Bushey]

⇒ uname -a

Linux caine 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux

⇒ ansible-playbook --version

ansible-playbook 1.7.2

From playbook:

149     - name: PHP | Copy over php-fpm.conf

150       copy: src=/ansible/files/etc-php5/fpm/php-fpm.conf dest=/etc/php5/fpm/ mode=0644 owner=root

151       tags: php

152 

153     - name: PHP | Copy over fpm-php pools

154       synchronize: src=/ansible/files/etc-php5/fpm/pool.d dest=/etc/php5/fpm/ recursive=yes delete=yes

155       tags: php

⇒ ansible-playbook -i ../hosts web_config.yml -t php

PLAY [Ansible config for web servers] ***************************************** 

GATHERING FACTS *************************************************************** 

ok: [web4]

ok: [web5]

ok: [web2]

ok: [web1]

TASK: [PHP | Copy over php-fpm.conf] ****************************************** 

ok: [web4]

ok: [web5]

ok: [web1]

ok: [web2]

TASK: [PHP | Copy over fpm-php pools] ***************************************** 

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Are you sure you want to continue connecting (yes/no)? no

fatal: [web1] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Are you sure you want to continue connecting (yes/no)? no

fatal: [web2] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Are you sure you want to continue connecting (yes/no)? no

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Are you sure you want to continue connecting (yes/no)? no

fatal: [web5] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

fatal: [web4] => SSH encountered an unknown error during the connection. We recommend you re-run the command using -vvvv, which will enable SSH debugging output to help diagnose the issue

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 

           to retry, use: --limit @/home/michael/web_config.retry

web1                        : ok=2    changed=0    unreachable=1    failed=0   

web2                        : ok=2    changed=0    unreachable=1    failed=0   

web4                        : ok=2    changed=0    unreachable=1    failed=0   

web5                        : ok=2    changed=0    unreachable=1    failed=0

Using -vvvv just verifies that magically the IP of the servers are now all 127.0.0.1:

TASK: [PHP | Copy over fpm-php pools] ***************************************** 

<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/michael/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', '127.0.0.1', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1426710670.95-25632361611852 && echo $HOME/.ansible/tmp/ansible-tmp-1426710670.95-25632361611852'"]

<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/michael/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=root', '-o', 'ConnectTimeout=10', '127.0.0.1', "/bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1426710670.96-173902150969631 && echo $HOME/.ansible/tmp/ansible-tmp-1426710670.96-173902150969631'"]

The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.

ECDSA key fingerprint is e8:71:64:d2:53:f9:46:ed:21:7a:a9:61:dd:8c:48:0b.

Any ideas? This script has worked for months. The only thing different is I have been trying to get linode support working and ran "sudo pip install linode-python"

[Brian Coca]

It looks like your name resolution has changed, can you show us your
ansible hosts file, your /etc/resolv.conf and your /etc/hosts ?




--
Brian Coca

[Michael Bushey]

using -vvvv I get this:

....

TASK: [PHP | Copy over php-fpm.conf] ****************************************** 

<w02> ESTABLISH CONNECTION FOR USER: root

<w01> ESTABLISH CONNECTION FOR USER: root

...

<w04> ESTABLISH CONNECTION FOR USER: root

<w05> ESTABLISH CONNECTION FOR USER: root

...

TASK: [PHP | Copy over fpm-php pools] ***************************************** 

<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

<127.0.0.1> ESTABLISH CONNECTION FOR USER: root

...

<127.0.0.1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto'...

[ hangs here ]

[Brian Coca]

do you have delegate_to or connection: local set on that task?

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/6bf392f5-c4b4-4fa4-8e3d-ca9acd6b360e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Brian Coca

[Michael Bushey]

Hi Brian.

From hosts:

...

[webservers]

web[01:02]

web[04:05]

...

resolv.conf contains the nameserver of my router.

from etc/hosts:

45.56.XXX.XXX   web1

104.237.XXX.XXX web2

104.239.XXX.XXX web4

104.130.XXX.XXX web5

Neither the ansible hosts file or /etc/hosts are changing mid run of the ansible script in the same place every time. This is definitely a bug in Ansible. Thanks for your input. :)

[Michael Bushey]

Hi Brian. :)

> do you have delegate_to or connection: local set on that task? 

No. Here's the  top of web_config.yml

  1 ---

  2 - name: Ansible config for web servers

  3   hosts: webservers

  4   user: root

  5   sudo: no

  6   vars_files:

  7     - password.yml

  8     - hosts.yml

  9   vars:

...

 23   tasks:

...

[Brian Coca]

but on the task that is hitting localhost, do you have a
delegate_to/connection: local there?


--
Brian Coca

[Michael Bushey]

> but on the task that is hitting localhost, do you have a delegate_to/connection: local there?

No, that's the bug I'm reporting. If I comment out 

153       #    - name: PHP | Copy over fpm-php pools

154       #      synchronize: src=/obsidian/ansible/files/etc-php5/fpm/pool.d dest=/etc/php5/fpm/ recursive=yes delete=yes

155       #      tags: php

Then the rest of the playbook works correctly. The synchronize command specifically is using 127.0.0.1 instead of the machine IPs.

[Brian Coca]

By default synchronize runs as a local action to copy files from the
ansible host to the remote server, do you have localhost or 127.0.0.1
defined in your hosts file and connection variables set?




--
Brian Coca

[Michael Bushey]

Thank you Brian!!!! Following some advice on using the linode module I added this to the ansible hosts file:

 20 [local]

 21 localhost

This broke only the synchronize command. Strange.