Complex iteration problems

[Ashley Penney]

Hi,

We've run into an issue (and a pattern really) using ansible that we've been unable to solve and I'm hoping the list can teach us the "ansible way".

The background is that we want to create a number of monitors automatically, based on certain information.  The solution we have today is a variable:

monitor_exchanges:

- name: exchange1

  demand_partners:

    - 'dp1'

    - 'dp2'

    

And the consumer:

- name: Create datadog monitors

  datadog_monitor:

    state: "present"

    type: "query alert"

    name: '{{item.0.name}}-{{item.1}} over delivery'

    query: "avg({{monitor_interval}}):100*(avg:requests.outbound.transmitted.count{sp_name:{{item.0.name}}, dp_name:{{item.1}}} - avg:demand_partner.rps{sp_name:{{item.0.name}}, dp_name:{{item.1}}})/avg:demand_partner.rps{sp_name:{{item.0.name}}, dp_name:{{item.1}}} > {{monitor_threshold}}"

    message: "test"

    api_key: "{{datadog_api_key}}"

    app_key: "{{datadog_app_key}}"

  with_subelements:

    - monitor_exchanges

    - demand_partners

  when: monitor_exchanges is defined

  

This then creates, for each exchange a monitor per demand partner.  That part is fairly easy and works fine but we wanted to externalize the 'query'/'name'/'message' part of the above thing, so that we'd have something more like:

- name: Create monitors

datadog_monitor:

  state: "present"

  type: "query alert"

  query: {{item.something.query}}

  name: {{item.something.query}}

  message: {{item.something.query}}

  

We kept trying to do this with two datastructures, the one from above and then

something like "monitors" that would be a list of hashes containing each monitor to create.  I couldn't come up with the right combination of iteration to make it work however, because I really needed to iterate over two separate things in different ways, I couldn't just list the two structures, I needed to do something more like:

with_nested:

  - with_subelements:

    - monitor_exchanges

    - demand_partners

  - monitors

  

Is there another way I can model this that makes sense?  I really just want to iterate over a list of monitors and then for each monitor iterate over the `monitor_exchanges` list to pull all the appropriate data.  

[Christian Thiemann]

The fundamental problem is that Ansible doesn't allow nested loops -- even though Ansible looks like a procedural language (it's executing commands/modules in order and it can loop over things), it really isn't. The only "simulation" of a for/while loop is the with_foobar stuff, which calls a lookup plugin that must return a simple list to iterate over. You could write your own lookup plugin that does a combination of the subelements and nested plugins, but if you want to stay in the "Ansible realm", then here's a solution that I used to solve a similar problem:

Define a helper variable that does the work of with_subelements:

monitor_exchange_demand_partners: |
  {%- set result = [] -%}
  {%- for exchange in monitor_exchanges|default([]) -%}
    {%- do result.extend(exchange.demand_partners|default([]) -%}
  {%- endfor -%}
  {{ result|unique }}

That exploits the fact that Ansible variables are Jinja templates, and that if a template evaluates into a string representation of a list or dict it will be converted to that list/dict. You'll have to add this to your ansible.cfg (Ansible variables in combination with Jinja look like a functional language, but Jinja lacks all the important tools of functional programming, so we'll use an extension that turns Jinja into a procedural language instead):

[defaults]
jinja2_extensions = jinja2.ext.do

Then you can use a simple with_nested:

- datadog_monitor: ...
  with_nested:
    - monitor_exchange_demand_partners
    - monitors

[Brian Coca]

this won't work

with_nested:
- with_subelements:
- monitor_exchanges
- demand_partners
- monitors

but this will:
with_nested:
- "{{lookup('subelements', [monitor_exchanges, demand_partners]}""
- monitors

I'm not sure that will get you what you want, but it should correctly
combine the lookups now



--
Brian Coca

[Johnny Everson]

Hi Brian,

I am trying to implement your suggestion, I am getting:

"ERROR! an unexpected type error occurred. Error was sequence item 0: expected string, tuple found" (I fixed the obvious typo in your code)

Does this ring a bell?

[Brian Coca]

are you using current devel?

> --
> You received this message because you are subscribed to the Google Groups
> "Ansible Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to ansible-proje...@googlegroups.com.
> To post to this group, send email to ansible...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/ansible-project/2620eb58-cffa-4c9a-ab52-9d2e7ea60dee%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.



--
Brian Coca

[Johnny Everson]

yes, on devel, last synced on Jun 16.

[Brian Coca]

some of the lookups are not working correctly with devel, that woudl
be the error messasge you get

> https://groups.google.com/d/msgid/ansible-project/ec78d24e-68af-428c-9596-62b7a5690b73%40googlegroups.com.