Nested loops with subelements

[Ananda Debnath]

Hi,

As others have before me, I find the syntax for nested loops somewhat confusing :(

Consider my group_vars:
ec2_specs:
  - { region: "us-east-1", ami: "ami-b66ed3de", count: 1, type: "t2.micro" }
  - { region: "us-west-1", ami: "ami-b56e64f0", count: 2, type: "t2.micro" }
  - { region: "sa-east-1", ami: "ami-9337828e", count: 1, type: "t2.micro" }

nginx_containers:
  - { name: "NGINX_A", hostport: "443" }
  - { name: "NGINX_B", hostport: "444" }
  - { name: "NGINX_B", hostport: "445" }

I'm setting up a security group for these instances like so:
- name: Setup AWS EC2 Security group
  ec2_group:
    name: microserver_security_group
    description: "Security group for microservices"
    region: "{{ item.region }}"
    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: tcp
        from_port: 443
        to_port: 445
        cidr_ip: 0.0.0.0/0
    rules_egress:
      - proto: all
        cidr_ip: 0.0.0.0/0
  with_items: ec2_specs

This works because the host ports are continuous 443-445. I need to retain the flexibility of changing this to something like 443, 1443, 2443 ... etc. One way to express this might be:

- name: Setup AWS EC2 Security group
  ec2_group:
    name: microserver_security_group
    description: "Security group for microservices"
    region: "{{ item.region }}"
    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: tcp
        from_port: "{{ subitem.hostport }}"
        to_port: "{{ subitem.hostport }}"
        cidr_ip: 0.0.0.0/0
        with_subitems: nginx_containers
    rules_egress:
      - proto: all
        cidr_ip: 0.0.0.0/0
  with_items: ec2_specs

Something like: with_XYZ at arbitrary levels with XYZ representing each item would be intuitive to me here. No doubt there are other ways of expressing this as well - where the syntax makes visual sense. As of now though, I don't know how to accomplish this short of hard-coding it in the role tasks.

A second problem arising from the same problem (or my lack of syntactic know-how) is the way I call ec2 and use its somewhat clunky output.
- name: Setup AWS EC2 Instances
  ec2:
    instance_type: "{{ item.type }}"
    image: "{{ item.ami }}"
    monitoring: no
    wait: yes
    group: microserver_security_group
    key_name: microserver_keypair
    instance_tags:
      nvtype: microserver
      Name: "MicroServer"
    count: "{{ item.count }}"
    region: "{{ item.region }}"
  with_items: ec2_specs
  register: ec2

The only way I've found of using the output of the above is the following sanity test example:

- name: Test microsites 443
  shell: chdir={{ tempdir.stdout }} /usr/bin/curl -v -s -k --key test.key --cert test.crt https://{{ item.1.public_ip }}:443
  with_subelements:
    - ec2.results
    - instances
  register: curl_out

Since the curl_test itself runs from localhost, I can't use the dynamic inventory within the site.yml file against the test role. I now have to hard-code N tests here - one for each host-port in the nginx_containers variable. I'd much rather express this as a single task with a nested loop - but I haven't been able to figure out how to.

Is this possible?

Many thanks,
Ananda

[Ananda Debnath]

I guess I could express it as a nested dict in the first place?

ec2_specs:
  - { region: "us-east-1", ami: "ami-b66ed3de", count: 1, type: "t2.micro", containers: [{ name: "NGINX_A", hostport: "443" }, { name: "NGINX_B", hostport: "444" }] }
  - etc

I suspect though that if I need to express a large number of these, it can get error-prone - so a set of M properties and a set of N sub-properties would still be a more compact and less error-prone representation of this matrix.

[Ananda Debnath]

Any recommendations?

[Michael DeHaan]

Sorry, I'm not sure what your specific question is.

I see a lot of text without a question mark and am having trouble parsing.

On Wed, Nov 5, 2014 at 6:42 PM, Ananda Debnath <ananda....@gmail.com> wrote:

Any recommendations?

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/32d2c9b7-e1f2-418a-a9ec-1af50ad72aa1%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ananda Debnath]

Thanks for looking Michael,

I was looking for an intuitive way for expressing nested variables in this use case.

In my group vars, I specify 3 ec2 region/ami specs:

ec2_specs:
  - { region: "us-east-1", ami: "ami-b66ed3de", count: 1, type: "t2.micro" }

  - { region: "us-west-1", ami: "ami-b56e64f0", count: 2, type: "t2.micro" }

  - { region: "sa-east-1", ami: "ami-9337828e", count: 1, type: "t2.micro" }

I want to spin up count instances of each ami. In each of them I want to setup N separate processes that run on different ports:
process_ports:
  - { name: "Process_A", hostport: "1443" }
  - { name: "Process_B", hostport: "2974" }
  - { name: "Process_C", hostport: "3555" }

To make sure these ports are accessible from the outside world I need to set up ec2 security groups for these instances. How can I express this by using my group vars rather than hard coding them as I have done below?

- name: Setup AWS EC2 Security group
  ec2_group:
    name: microserver_security_group
    description: "Security group for microservices"
    region: "{{ item.region }}"
    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: tcp

        from_port: 1443
        to_port: 1443
        cidr_ip: 0.0.0.0/0
      - proto: tcp
        from_port: 2754
        to_port: 2754
        cidr_ip: 0.0.0.0/0
      - proto: tcp
        from_port: 3555
        to_port: 3555
        cidr_ip: 0.0.0.0/0

    rules_egress:
      - proto: all
        cidr_ip: 0.0.0.0/0
  with_items: ec2_specs

Using nesting variables trivially doesn't work:

- name: Setup AWS EC2 Security group
  ec2_group:
    name: microserver_security_group
    description: "Security group for microservices"

    region: "{{ item[0].region }}"

    rules:
      - proto: tcp
        from_port: 22
        to_port: 22
        cidr_ip: 0.0.0.0/0
      - proto: tcp

        from_port: "{{ item[1].hostport }}"
        to_port: "{{ item[1].hostport }}"
        cidr_ip: 0.0.0.0/0

    rules_egress:
      - proto: all
        cidr_ip: 0.0.0.0/0

  with_nested:
    - ec2_specs
    - process_ports

This simply causes 3 separate runs of the same ec2_group command and I end up with a single security group in each region where only the last hostport is exposed (each run overwriting the previous run).

[Dan Vaida]

Hello Ananda,

Is your number of processes (containers in your case) static? What about the ports? do they have a particular generation rule that matters to you?