delegate_to does not work for array of hosts passed in to role

[Barry Kaplan]

I posted this as a bug (https://github.com/ansible/ansible/issues/11102). But it was closed saying it was a discussion. Not sure why, seems like a clear bug to me. But I'm open to be dissuaded. 

----

Ansible 1.9.1 running from OSX.

in playbook

- role: mongodb
  mongodb_replica_set_name: "{{mongodb_rs_name}}"
  mongodb_replica_set_hosts: "{{groups.mongod_rs_member}}"

mongodb role

- set_fact:
    mongodb_candidate_master: "{{mongodb_replica_set_hosts[0]}}"
  run_once: true

- shell: echo "HELLO groups.mongod_rs_member +++++++++++++++++"
  delegate_to: "{{groups.mongod_rs_member[0]}}"
  run_once: true

- shell: echo "HELLO mongodb_candidate_master +++++++++++++++++"
  delegate_to: "{{mongodb_candidate_master}}"
  run_once: true

- shell: echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"
  delegate_to: "{{mongodb_replica_set_hosts[0]}}"
  run_once: true

output

TASK: [mongodb | shell echo "HELLO groups.mongod_rs_member +++++++++++++++++"] ***
<10.0.139.56> ESTABLISH CONNECTION FOR USER: ubuntu
<10.0.139.56> REMOTE_MODULE command echo "HELLO groups.mongod_rs_member +++++++++++++++++" #USE_SHELL
<10.0.139.56> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 10.0.139.56 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=enrekuxbmqjtjdgsxvlukuvqlhnudrkw] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-enrekuxbmqjtjdgsxvlukuvqlhnudrkw; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
changed: [10.0.139.56 -> 10.0.139.56] => {"changed": true, "cmd": "echo \"HELLO groups.mongod_rs_member +++++++++++++++++\"", "delta": "0:00:00.005216", "end": "2015-05-31 04:13:32.378158", "rc": 0, "start": "2015-05-31 04:13:32.372942", "stderr": "", "stdout": "HELLO groups.mongod_rs_member +++++++++++++++++", "warnings": []}

TASK: [mongodb | shell echo "HELLO mongodb_candidate_master +++++++++++++++++"] ***
<10.0.139.56> ESTABLISH CONNECTION FOR USER: ubuntu
<10.0.139.56> REMOTE_MODULE command echo "HELLO mongodb_candidate_master +++++++++++++++++" #USE_SHELL
<10.0.139.56> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 10.0.139.56 /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=xnkqbskjckeetqglyftpndfusnsyhoyq] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-xnkqbskjckeetqglyftpndfusnsyhoyq; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
changed: [10.0.139.56 -> 10.0.139.56] => {"changed": true, "cmd": "echo \"HELLO mongodb_candidate_master +++++++++++++++++\"", "delta": "0:00:00.005059", "end": "2015-05-31 04:13:33.277391", "rc": 0, "start": "2015-05-31 04:13:33.272332", "stderr": "", "stdout": "HELLO mongodb_candidate_master +++++++++++++++++", "warnings": []}

TASK: [mongodb | shell echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"] ***
<{> ESTABLISH CONNECTION FOR USER: ubuntu
<{> REMOTE_MODULE command echo "HELLO mongodb_replica_set_hosts +++++++++++++++++" #USE_SHELL
<{> EXEC ssh -C -vvv -o ControlMaster=auto -o ControlPersist=30m -o ControlPath="/Users/bkaplan/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 { /bin/sh -c 'sudo -k && sudo -H -S -p "[sudo via ansible, key=mqgojxehsqcpahfvfcdimgoedvygtvvb] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-mqgojxehsqcpahfvfcdimgoedvygtvvb; LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python'"'"''
fatal: [10.0.139.56 -> {] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh

[Brian Coca]

From the output of the 3rd task it seems that
mongodb_replica_set_hosts[0] is being interpreted as '{' which, of
course, does not resolve. This does not seem to be an array issue but
of too many nested templating attempts.


--
Brian Coca

[Barry Kaplan]

What do you think would be causing the nested templating? Something deep in variable resolution?

The third echo is the direct version of first setting the value to a fact, which works.

[Brian Coca]

not sure what 'works' means in this context

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/a747482f-5ed6-4ac2-8dc3-752a0f9adf9c%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Brian Coca

[Barry Kaplan]

On Tuesday, June 2, 2015 at 10:04:04 PM UTC+5:30, Brian Coca wrote:

not sure what 'works' means in this context

"works" is that the third task delegates to the same host as the second task.

 

Don't you expect the exact same output for the last two tasks? Are you seeing something that I am missing?

 

[Brian Coca]

the task fails with e 'fatal', what looks the same is the verbose
output of what will be executed

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/79164104-61fa-466c-8ced-fae92d8ba19b%40googlegroups.com.

[Barry Kaplan]

2) ssh ... -o ConnectTimeout=10 10.0.139.56 /bin/sh -c ...
3) ssh ... -o ConnectTimeout=10 {           /bin/sh -c ...

So you don't think that the ssh ip in the 3 task should not be the same as 2? 

I'm sorry Brian, but I am not all following what the miscommunication is here. 

[Brian Coca]

the ssh ip in task 3 is '{' which indicates that the [0] is applied
to the 'template string' and not to the resulting array you expect.
This makes the 3rd task fail as it cannot ssh to '{'.

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/39fb66d6-bcb3-4f05-be8e-0f3c1267f589%40googlegroups.com.

[Barry Kaplan]

On Wednesday, June 3, 2015 at 8:54:06 AM UTC+5:30, Brian Coca wrote:

the ssh ip in task 3 is  '{' which indicates that the [0] is applied
to the 'template string' and not to the resulting array you expect.
This makes the 3rd task fail as it cannot ssh to '{'.

Exactly. Is that a mistake I am making? Or a bug? I filed the bug because I don't see a mistake on my part. 

[Brian Coca]

can you show that variable in a debug during execution? it does not
look like it is being set properly.

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/ansible-project/8c07227d-d1e3-46c8-80e4-a5e78dc01e15%40googlegroups.com.

[Barry Kaplan]

Here's the new test:

- set_fact:
    mongodb_candidate_master: "{{mongodb_replica_set_hosts[0]}}"
  run_once: true

- name: var groups.mongod_rs_member[0]
  debug: var=groups.mongod_rs_member[0]

  run_once: true


- shell: echo "HELLO groups.mongod_rs_member +++++++++++++++++"
  delegate_to: "{{groups.mongod_rs_member[0]}}"
  run_once: true

- name: var mongodb_candidate_master
  debug: var=mongodb_candidate_master
  run_once: true

- shell: echo "HELLO mongodb_candidate_master +++++++++++++++++"
  delegate_to: "{{mongodb_candidate_master}}"
  run_once: true

- name: var mongodb_replica_set_hosts[0]
  debug: var=mongodb_replica_set_hosts[0]
  run_once: true


- name: msg mongodb_replica_set_hosts[0]
  debug: msg="{{mongodb_replica_set_hosts[0]}}"
  run_once: true


- debug: msg="{{mongodb_replica_set_hosts[0]}}"
  run_once: true

- shell: echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"
  delegate_to: "{{mongodb_replica_set_hosts[0]}}"
  run_once: true

And the output:

ansible❯ ansible-playbook -i inventory/aws/staging playbooks/configure/mongod.yml --skip-tags base                                                                 ops/git/master !+


PLAY [Base ansible configuration (packages)] **********************************


GATHERING FACTS ***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Base ansible configuration (configure)] *********************************


GATHERING FACTS ***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Base ansible configuration (users)] *************************************


GATHERING FACTS ***************************************************************
ok: [10.0.137.189]
ok: [10.0.139.51]
ok: [10.0.136.46]


PLAY [Configure mongodb replicaset cluster] ***********************************


GATHERING FACTS ***************************************************************
ok: [10.0.136.46]
ok: [10.0.137.189]
ok: [10.0.139.51]


TASK: [debug var=groups.mongod_rs_member] *************************************
ok: [10.0.136.46] => {
    "var": {
        "groups.mongod_rs_member": [
            "10.0.136.46",
            "10.0.139.51",
            "10.0.137.189"
        ]
    }
}
ok: [10.0.139.51] => {
    "var": {
        "groups.mongod_rs_member": [
            "10.0.136.46",
            "10.0.139.51",
            "10.0.137.189"
        ]
    }
}
ok: [10.0.137.189] => {
    "var": {
        "groups.mongod_rs_member": [
            "10.0.136.46",
            "10.0.139.51",
            "10.0.137.189"
        ]
    }
}


TASK: [mongodb | set_fact ] ***************************************************
ok: [10.0.136.46]


TASK: [mongodb | var groups.mongod_rs_member[0]] ******************************
ok: [10.0.136.46] => {
    "var": {
        "groups.mongod_rs_member[0]": "10.0.136.46"

    }
}


TASK: [mongodb | shell echo "HELLO groups.mongod_rs_member +++++++++++++++++"] ***

changed: [10.0.136.46 -> 10.0.136.46]


TASK: [mongodb | var mongodb_candidate_master] ********************************
ok: [10.0.136.46] => {
    "var": {
        "mongodb_candidate_master": "10.0.136.46"

    }
}


TASK: [mongodb | shell echo "HELLO mongodb_candidate_master +++++++++++++++++"] ***

changed: [10.0.136.46 -> 10.0.136.46]


TASK: [mongodb | var mongodb_replica_set_hosts[0]] ****************************
ok: [10.0.136.46] => {
    "var": {
        "mongodb_replica_set_hosts[0]": "10.0.136.46"
    }
}


TASK: [mongodb | msg mongodb_replica_set_hosts[0]] ****************************
ok: [10.0.136.46] => {
    "msg": "10.0.136.46"
}


TASK: [mongodb | debug msg="{"] ***********************************************
ok: [10.0.136.46] => {
    "msg": "10.0.136.46"

}


TASK: [mongodb | shell echo "HELLO mongodb_replica_set_hosts +++++++++++++++++"] ***

fatal: [10.0.136.46 -> {] => SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh


FATAL: all hosts have already failed -- aborting

The last debug is interesting. I does not have a -name and it seems to render the value as '{' just like the delegate_to.

[Barry Kaplan]

Is it still unclear Brian?

I also seem to be seeing a related problem. Consider:

- name: Provision Ops RDS instance
  hosts: localhost
  connection: local
  gather_facts: no
  tags: [control]


  tasks:
    ...
    - shell: |
        echo {{groups.consul_server[0]}} && hostname -i && curl localhost:8500/v1/catalog/services
      register: __catalog
      delegate_to: groups.consul_server[0]

Here the shell commands runs on localhost, not the host specified.  The output is

TASK: [shell echo {{groups.consul_server[0]}} && hostname -i && curl localhost:8500/v1/catalog/services
] ***
<groups.consul_server[0]> REMOTE_MODULE command echo 10.0.196.116 && hostname -i && curl localhost:8500/v1/catalog/services #USE_SHELL
<groups.consul_server[0]> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713 && echo $HOME/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713']
<groups.consul_server[0]> PUT /tmp/tmpTGBlM3 TO /home/bkaplan/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713/command
<groups.consul_server[0]> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/bkaplan/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713/command; rm -rf /home/bkaplan/.ansible/tmp/ansible-tmp-1437044596.27-54518627543713/ >/dev/null 2>&1']
failed: [localhost -> groups.consul_server[0]] => {"changed": true, "cmd": "echo 10.0.196.116 && hostname -i && curl localhost:8500/v1/catalog/services", "delta": "0:00:00.012358", "end": "2015-07-16 16:33:16.332372", "rc": 7, "start": "2015-07-16 16:33:16.320014", "warnings": []}
stderr:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8500: Connection refused
stdout: 10.0.196.116
127.0.1.1

Where  '10.0.196.116' is groups.consul_server[0]. So the variable resolves correctly, but delegate to does not seem to use it. This is not specific to shell, it has the same behavior for other modules (eg, uri).

[Brian Coca]

you need moustaches:

delegate_to: "{{groups.consul_server[0]}}"

only conditionals (when: ) do not need templating, with_ has a
haphazard support for it which we plan to deprecate.

--
Brian Coca

[Barry Kaplan]

Hmm, thought I tried that, but will check. Thanks Brian.

I wonder whether it would be better for it to be error if delegate_to does not resolve rather than silently fall back to localhost... Depending on the task fallback to localhost could do some serious damage.

[Barry Kaplan]

I see the same behavior with the moustaches:

    - name: Test delegate to
      shell: echo "{{ groups.consul_server[0] }}" && hostname -i
      register: __hostname

      delegate_to: "{{ groups.consul_server[0] }}"
    - debug: var=__hostname

TASK: [Test delegate to] *********************************************
<10.0.196.116> REMOTE_MODULE command echo "10.0.196.116" && hostname -i #USE_SHELL
<10.0.196.116> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917 && echo $HOME/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917']
<10.0.196.116> PUT /tmp/tmpDXWJFH TO /home/bkaplan/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917/command
<10.0.196.116> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /home/bkaplan/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917/command; rm -rf /home/bkaplan/.ansible/tmp/ansible-tmp-1437108695.64-26819269533917/ >/dev/null 2>&1']
changed: [localhost -> 10.0.196.116] => {"changed": true, "cmd": "echo \"10.0.196.116\" && hostname -i", "delta": "0:00:00.004458", "end": "2015-07-17 10:21:35.693136", "rc": 0, "start": "2015-07-17 10:21:35.688678", "stderr": "", "stdout": "10.0.196.116\n127.0.1.1", "warnings": []}


TASK: [debug var=__hostname] **************************************************
ok: [localhost] => {
    "var": {
        "__hostname": {
            "changed": true,
            "cmd": "echo \"10.0.196.116\" && hostname -i",
            "delta": "0:00:00.004458",
            "end": "2015-07-17 10:21:35.693136",
            "invocation": {
                "module_args": "echo \"10.0.196.116\" && hostname -i",
                "module_name": "shell"
            },
            "rc": 0,
            "start": "2015-07-17 10:21:35.688678",
            "stderr": "",
            "stdout": "10.0.196.116\n127.0.1.1",
            "stdout_lines": [
                "10.0.196.116",
                "127.0.1.1"
            ],
            "warnings": []
        }
    }
}

[Brian Coca]

well, the first difference is that you are not delegating to a host
named 'groups.consul_server[0]' but to 10.0.196.116

Sorry for not noticing before, but the delegate_to on debug is
irrelevant, debug always executes on 'the master' or machine you
execute ansible on, it does not need to connect to any machine, i'm
not sure what you are expecting there.

delegate_to on shell or any module that executes remotely goes to the
delegated server, but things that operate on 'master' like debug,
prompt, add_host, group_by make no sense to delegate.

--
Brian Coca

[Barry Kaplan]

Brian, I'm not following you at all. It was the shell command that uses the delegate_to. The debug is only showing the register from the shell.

    - name: Test delegate to
      shell: echo "{{ groups.consul_server[0] }}" && hostname -i
      register: __hostname
      delegate_to: "{{ groups.consul_server[0] }}"

    - debug: var=__hostname

"stdout_lines": [
    "10.0.196.116",  <--- echo "{{ groups.consul_server[0] }}"
    "127.0.1.1"      <--- hostname -i  
],

[Brian Coca]

ah, formatting of previous email threw me off, the only reason i can
think of for this happening is that you have connection=local for the
task (from play or command line) and the host does not have it
specified through ansible_connection.

--
Brian Coca

[Barry Kaplan]

All my hosts come from the ec2 inventory plugin. I can access the hosts in all other ways without having to explicitly specify an ansible_connection.

So then is it better to specify the target host in the play and use local_action? I kinda thought these 

two were inverses of each other.

-barry

[Barry Kaplan]

Perfect! I have removed the connection=local from the play and now all works as expected. I guess I don't fully understand why connection=local should be used. When I was first starting with ansible plays that I studied that did ec2 stuff used it, so I did too.

[Brian Coca]

The connection: local is used when you want a plugin to execute on
master that normally would execute on the target. You normally want to
do this with cloud modules or modules that use some web api. Some
modules always execute on the 'master' as they are meant to just
change the data in the play like, debug, set_fact, add_host, group_by
so these don't need any connection settings.

--
Brian Coca