Nested looping with hash/dict so I can override values

[David Reagan]

tl;dr: is there a way to make with_subelements work with dicts? Or something to that effect?

I've been trying to figure out how to run a task over a dict, that also loops over a nested dict in order to run the actual commands.

So far, I haven't figured out how to use with_items, with_dict, or with_nested to do what I want. with_subelements comes close, but keeps choking on the fact my subelement is not a list.

My playbook (would have pastebined it, but pastebin wouldn't load for me...):

---
# testing playbook
- hosts: 192.168.88.2
  vars:
    people:
      johnsmith:
        fname: john
        lname: smith
        locations:
          birthplace:
            state: id
            city: boise
          second:
            state: or
            city: portland
          third:
            state: ha
            city: honolulu
      jilljones:
        fname: jill
        lname: jones
        locations:
          birthplace:
            state: mo
            city: springfield
          second:
            state: mt
            city: fort benton
          third:
            state: id
            city: emmett
      jilljones:
        locations:
          birthplace:
            state: wa
            city: wilbur
  tasks:
    - name: testing dict
      debug: msg="{{ item.1 }}"
      with_subelements:
        - people
        - locations

The output:

$ ansible-playbook -i hosts -u vagrant --sudo tmp/testing.yml
PLAY [192.168.88.2] *********************************************************** 
GATHERING FACTS *************************************************************** 
ok: [192.168.88.2]
TASK: [testing dict] ********************************************************** 
fatal: [192.168.88.2] => the key locations should point to a list, got '{'birthplace': {'city': 'boise', 'state': 'id'}, 'second': {'city': 'portland', 'state': 'or'}, 'third': {'city': 'honolulu', 'state': 'ha'}}'
FATAL: all hosts have already failed -- aborting
PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/reagand/testing.retry
192.168.88.2               : ok=1    changed=0    unreachable=1    failed=0   

What I want to see is something like:

johnsmith:
          birthplace:
            state: id
            city: boise
          second:
            state: or
            city: portland
          third:
            state: ha
            city: honolulu
jilljones:
          birthplace:
            state: wa
            city: wilbur
          second:
            state: mt
            city: fort benton
          third:
            state: id
            city: emmett

I really need this to be a dict/hash so that hash_behaviour=merge works. For example, if I want jilljones birthplace to default to springfield, mo, in group_vars, but for a specific host it should be seattle, wa, then I would set the following in the specific host_vars file:

people:
  jilljones:
    locations:
      birthplace:
         city: seattle
         state: wa

Note, I've only been using this people hash as something to experiment on. My actual use case is a task that sets configuration settings for a web app. The app config didn't lend itself to templating. I tried that first. Now I've ended up with lininfile regex replacements, each named with their own key. Something like:

configkey:
  regex:
  value:
configkey2:
  regex:
  value:
... 

That's the equivalent of the locations hash in my testing playbook.

I also don't know how many instances of the app I'll have per server, so I can't just do a task per instance. 

Ultimately, if I make my subelement a list, http://pastebin.com/mixMTz6H (pastebin worked earlier...) it works, but it overrides all the previously set subelements of the main key. That means I have to copy all the locations from group_vars into host_vars if I want to override something on a specific host. I'd prefer to avoid that. So, is there a with_subelements that would work with a hash like I want? Or another way to get the same effect?

 

[David Reagan]

So, I took a look at creating a plugin. I copied subelements.py, and modified it to return a dict.  See https://gist.github.com/jerrac/8a16e0c1031df89621da

I think it will do what I want, I've only done some basic testing. Anyone have any feedback?

Should I modify the copyright? I'm pretty sure I should, but I'm not sure what to...

--David Reagan

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/bRBQQttsFgE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/3bf69804-8242-455f-a97d-7605e6336cf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael DeHaan]

"tl;dr: is there a way to make with_subelements work with dicts? Or something to that effect?"

Doesn't sound like you need the nested loop that subelements provides and you'd be happy with https://github.com/ansible/ansible/blob/devel/lib/ansible/runner/lookup_plugins/dict.py

That is unless you want to loop over each locations for each user, in which case, sure, you're doing the right thing.

If subelements can be made to do this unobtrusively patches would be accepted.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANo%2B_Ac0tesM2K2qcGZzOf%2BP0dvgp%2B77vH%2BdLCKSrX%3Dkf4fO0Q%40mail.gmail.com.

[Petros Moisiadis]

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgy%3Dn-FNhP5BkmKe%2BuBORkgD6QN%3DfUnVFsgTZ%3D0zfhF3iw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

My pull request for with_nested (https://github.com/ansible/ansible/pull/7278) allows for all kind of nested loops, including the simpler case that is handled by the 'with_subelements' lookup.

In David's example, where the items in the first level of the loop are actually dictionary keys, to loop over the locations for each person, you would write:

with_nested:
  - people
  - people[item.0].locations


Now, for a comparison with the 'with_subelements' lookup, consider the following dict:

users:
  - name: alice
    authorized_keys:
      - /tmp/alice/onekey.pub
      - /tmp/alice/twokey.pub
  - name: bob
    authorized_keys:
      - /tmp/bob/id_rsa.pub

To construct a loop with "with_subelements" you would write:

with_subelements:
  - users
  - authorized_keys

To construct the same loop with my version of 'with_nested' you would write:

with_nested:
  - users
  - item.0.authorized_keys

Of course, the power of my version of 'with_nested' is that it allows you to make multiple-level nested loops that cannot be constructed in another way. For example, it allows for the pattern: Loop over the hosts in a specific group, for each host access a list hostvar and run a task for each item in that list. The list hostvar could be as much deep as you want. Just add the loop levels needed to reach it.
So, let's say that you want to create a number of VMs that will be the members of a 'guests' group. You would have to connect to your virtualiazation server host, let's call it 'vmhost', and do what ever it is needed to create the VMs. After they have been created, you want to deploy the VMs with Ansible, but to be able to do that you would have first to bootstrap authorized ssh keys for 'root' user in order to pass them in the VM bootstrapping process. This is exactly a task for 'with_nested'. First, you configure your inventory, as needed: basically you create the 'guests' group and hostvars files for each host in that group  with a 'users' variable like the above. Then you construct the loop: At first level, you would loop over each host in the 'guests' group. At second level, you would loop over each user in the 'users' variable. At third level you would loop over each item in the 'authorized' list. Also add a conditional for 'root' user and ...voila:

- hosts: vmhost
  tasks:
    - name: Create authorized_keys for bootstrapping vm guests
      authorized_key: user="{{ item.1.name }}" key="{{ lookup('file', item.2) }}" path=/path/to/bootstrap/guests/{{ item.0 }}/root/.ssh/authorized_keys
      with_nested:
        - groups["guests"]
        - hostvars[item.0].users
        - item.1.authorized_keys
      when: item.1.name == "root"

[David Reagan]

Yeah... I'm confused. I found with_nested to be a bit odd in the first place, much less trying to figure out what Petros is adding. 

I ended up with https://github.com/jerrac/aspects/blob/master/ansible_plugins/lookup_plugins/subdict.py used at https://github.com/jerrac/aspects/blob/master/roles/owncloud/tasks/main.yml#L91

If I were to try and make that worth a pull request, I'd want to be able to look at sub dictionaries at any level (item.0, item.1 ... item.N). I don't think my code supports more than two levels right now. I'm not inclined to try that since python is not one of my strong points.

Ignore the rest of the project, I still have a ways to go before I think others should use it. Especially since galaxy.ansible.com now exists.

--David Reagan

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/536DF435.1070208%40yahoo.gr.

[Petros Moisiadis]

On 05/14/2014 12:03 AM, David Reagan wrote:

Yeah... I'm confused. I found with_nested to be a bit odd in the first place, much less trying to figure out what Petros is adding.

With my addition, with_nested is extended allowing you to use item.0, item.1, etc... in your expressions. The numbers 0, 1, 2, etc.. denote the loop level. 0 is for the first (outer) loop level, 1 is for one level deeper, 2 is for one more level deeper, etc... Nothing more, nothing less, but this addition is quite powerful because it gives you the ability, in each possible loop branch (think of a nested loop as a tree, see below), to reference back to the values that were evaluated in previous loop levels (nodes) in the same branch.

with_nested:
- ['a', 'b', 'c']
- ['d', 'e', 'f']
- item.0


     ,------ 'd' ----- 'a'
    /         
'a' -------- 'e' ----- 'a'
    \
     `------ 'f' ----- 'a'

     ,------ 'd' ----- 'b'
    /
'b' -------- 'e' ----- 'b'
    \
     `------ 'f' ----- 'b'

     .------ 'd' ----- 'c'
    /
'c' -------- 'e' ----- 'c'
    \
     `------ 'f' ----- 'c'
==========================
 0            1         2   <-- loop levels


Also, have you looked carefully at my other examples? What exactly don' t you understand in these?

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CANo%2B_Af59CUb7yizXtoKk9jnvbaSS5gf266fZpNZnjF4qXDnkg%40mail.gmail.com.