Iterating over complex dictionary, but only when key exists

[Dane Elwell]

Hello list,

I've got a complex dictionary I'm using to configure haproxy with,
which looks something like this (non-relevant portions have been
removed):

{
"l7config": {
"stats_password": "AirwavesEyedrops",
"frontends": [
{
"frontend_id": "6dcb7365-1b6c-4f24-a7ff-79590af2d536",
"vip": "192.0.1.200",
"service": "http",
"port": 80,
"backend_id": "49cd3849-455c-4e5c-9e13-70cda7be0397"
},
{
"frontend_id": "01f99586-c14c-4c99-99e0-cf6effac3ee8",
"vip": "192.0.1.200",
"port": 443,
"service": "http",
"backend_id": "49cd3849-455c-4e5c-9e13-70cda7be0397",
"tls": {
"pem": "-----BEGIN PRIVATE KEY-----\nMII
...snip... 7w==\n-----END CERTIFICATE-----\n",
"domain": "foobar.com"
}
}
],
"backends": [
{
...
},
...
]
}
}

As you can see, I've got two frontend configurations, one which has a
'tls' field and the other which doesn't. What I need to do is iterate
over the frontends and discover which ones have the tls field, then
write out the 'pem' certificates into a file.

I have this code in my playbook which isn't working as I think it
should, as it always seems to just skip the task:

- name: Write out SSLs to files
copy: dest=/etc/ssl/{{item.tls.domain}}.pem owner=root
group=root mode=0600 content={{item.tls.pem}}
with_items:
- l7config.frontends
when: item.tls is defined

I suspect I'm on the right track, but I'm missing something blindingly
obvious and stupid. I just need someone to point out where I'm being
daft :)

Thanks

Dane

[Dane Elwell]

Ok, so after looking into this further I don't think with_items can
look 'inside' the dictionary, and I should instead be using
with_subelements. That doesn't appear to work either though:

- name: Write out SSLs to files

copy: dest=/etc/ukfast/{{item.tls.domain}}.pem owner=root group=root
mode=0600 content={{item.tls.pem}}
with_subelements:
- l7config
- frontends
when: item.tls is defined


TASK: [layer7-lb | Write out SSLs to files] ***********************************
fatal: [192.168.122.76] => subelements lookup expects a dictionary,
got 'AirwavesEyedrops'


I'm not sure why it's pulling out the value of stats_password instead
of looking up the list 'frontends' inside the dictionary 'l7config'.

I've loaded my JSON into Python to confirm the types, too:

>>> type(a['l7config'])
<type 'dict'>
>>> type(a['l7config']['frontends'])
<type 'list'>

Regards

Dane

[Michael Hoglan]

Have you thought about breaking out some jinja expressions to manipulate the structure?  In particular the selectrattr() filter.

http://jinja.pocoo.org/docs/dev/templates/

Here is a similar example I did

- hosts: localhost

  connection: local

  vars:

    config: 

      frontend: 

        - id: 1

          value: test1

          tls: 

            - key: key1

              domain: domain1

        - id: 2

          value: test2

      backend:

        - id: 3

        - id: 4

  tasks:

    - debug: var=item

      with_items: config.frontend

    - debug: var=item

      with_items: config.frontend | selectattr("tls", "defined") | list

And then the output

(ansible)[ec2-user@ip-10-0-0-226 playbooks]$ ansible-playbook test_withitems.yml 

PLAY [localhost] ************************************************************** 

GATHERING FACTS *************************************************************** 

ok: [localhost]

TASK: [debug var=item] ******************************************************** 

ok: [localhost] => (item={'tls': [{'domain': 'domain1', 'key': 'key1'}], 'id': 1, 'value': 'test1'}) => {

    "item": {

        "id": 1, 

        "tls": [

            {

                "domain": "domain1", 

                "key": "key1"

            }

        ], 

        "value": "test1"

    }

}

ok: [localhost] => (item={'id': 2, 'value': 'test2'}) => {

    "item": {

        "id": 2, 

        "value": "test2"

    }

}

TASK: [debug var=item] ******************************************************** 

ok: [localhost] => (item={'tls': [{'domain': 'domain1', 'key': 'key1'}], 'id': 1, 'value': 'test1'}) => {

    "item": {

        "id": 1, 

        "tls": [

            {

                "domain": "domain1", 

                "key": "key1"

            }

        ], 

        "value": "test1"

    }

}

PLAY RECAP ******************************************************************** 

localhost                  : ok=3    changed=0    unreachable=0    failed=0   

That should give a list of all frontends that have a tls attribute defined.

Thanks!

Michael 

[Dane Elwell]

Hi Michael,

Thanks for the response. Your example does work for me, however
looking at your example I've managed to figure out why my original
example wasn't working:

- name: Write out SSLs to files
copy: dest=/etc/ssl/{{item.tls.domain}}.pem owner=root
group=root mode=0600 content={{item.tls.pem}}
with_items:
- l7config.frontends
when: item.tls is defined

Does not work, whereas:

- name: Write out SSLs to files
copy: dest=/etc/ssl/{{item.tls.domain}}.pem owner=root
group=root mode=0600 content={{item.tls.pem}}

with_items: l7config.frontends
when: item.tls is defined

Works fine. The with_items can't be a list itself, as Ansible doesn't
appear to 'see' inside the dictionary then, and just treats it as a
string.

Didn't know about selectattr, so that's one to remember, but now I've
got my original example working I'll stick with it as it seems more
natural to me out of the two!

Thanks

Dane

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/7d540723-1226-4574-a067-482c1c916c99%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dane Elwell]

Just for completeness in case anyone else stumbles across this issue,
my final task was:

# Use long-form for task, Ansible issue #9172

- name: Write out SSLs to files
copy:

dest: /etc/ssl/{{item.tls.domain}}.pem
owner: root
group: root
mode: 0600
content: "{{item.tls.pem}}"

with_items: l7config.frontends
when: item.tls is defined

Thanks

Dane

[Michael Hoglan]

When you do the list syntax for with_items, it would have worked if you would have put in an expression.

    - name: Write out SSLs to files 

      copy: dest=/etc/ssl/{{item.tls.domain}}.pem owner=root 

group=root mode=0600 content={{item.tls.pem}} 

      with_items: 

        - "{{ l7config.frontends }}"

      when: item.tls is defined 

Here is the output from the example I gave 

- hosts: localhost

  connection: local

  vars:

    config: 

      frontend: 

        - id: 1

          value: test1

          tls: 

            - key: key1

              domain: domain1

        - id: 2

          value: test2

      backend:

        - id: 3

        - id: 4

  tasks:

    - debug: var=item

      with_items: 

        - "{{ config.frontend }}"

      when: item.tls is defined

    - debug: var=item

      with_items: config.frontend | selectattr("tls", "defined") | list

And the result

(ansible)[ec2-user@ip-10-0-0-226 playbooks]$ ansible-playbook test_withitems.yml 

PLAY [localhost] ************************************************************** 

GATHERING FACTS *************************************************************** 

ok: [localhost]

TASK: [debug var=item] ******************************************************** 

ok: [localhost] => (item={'tls': [{'domain': 'domain1', 'key': 'key1'}], 'id': 1, 'value': 'test1'}) => {

    "item": {

        "id": 1, 

        "tls": [

            {

                "domain": "domain1", 

                "key": "key1"

            }

        ], 

        "value": "test1"

    }

}

skipping: [localhost] => (item={'id': 2, 'value': 'test2'})

TASK: [debug var=item] ******************************************************** 

ok: [localhost] => (item={'tls': [{'domain': 'domain1', 'key': 'key1'}], 'id': 1, 'value': 'test1'}) => {

    "item": {

        "id": 1, 

        "tls": [

            {

                "domain": "domain1", 

                "key": "key1"

            }

        ], 

        "value": "test1"

    }

}

PLAY RECAP ******************************************************************** 

localhost                  : ok=3    changed=0    unreachable=0    failed=0   

I believe this occurs because in the single line syntax it assumes an expression already, which is why you can specify jinja filters without wrapping it all in a jinja {{ }} expression block.  But when you do the long form / list form (don't know what the correct terminology is), it doesn't do expression blocks implicitly, you have to explicitly specify them.

Good news is, all these different ways of doing things is just different tools we can utilize depending on the situation, I find myself sometimes using jinja filters and other times using the standard with / when from ansible.

Thanks!

Michael