Populating an EC2 Security Group rules (using ec2_group modules) with IPs (Pingdom probe servers) gathered dynamically at playbook runtime

[Jaime Gago]

Hey there,

I'm trying to write a playbook that gets the latest Pingdom probe servers IPs and add updates an EC2 Security groups rules with those IPs, but I'm failing are iterating the IPs in the rule and only the latest IPs is added (I'm replacing instead of appending). I opened an ticket on github (1) but because I hadn't detailed out the whole use case it got closed without really answering the issue; so I thought I'd post here see what others are thinking. 

I'm not sure whether I'm trying to hard to fit this into a playbook as I have this working via a script, now of course I could call the script itself but that IMHO would defeat the purpose of using Ansible in the first place.

I understand why the playbook fails to append the rules but I haven't been able to figure out a way around other than modifying the ec2_group module itself.

J.

(1) https://github.com/ansible/ansible/issues/7584

[andreub]

Hi Jaime,

I had the exact same problem. ec2_group modules recreates the rules every time you use the it, so if you're running it in a loop, it will create the group the rule for the last item only.

A workaround that I implemented is to generate a var.yml with a var defined with the rules out of a template, and then source it dynamically:

---

  - name: Create rules

    sudo: False

    local_action:

      module: template src=sg_rules.j2 dest=./roles/postgres-server/vars/rules.yml

  - name: Load vars

    sudo: False

    include_vars: rules.yml

  - name: Open ports for DB clients

    sudo: False

    local_action:

      module: ec2_group

      aws_access_key: "{{ ofertia_s3_access_key }}"

      aws_secret_key: "{{ ofertia_s3_secret_key }}"

      name: "{{ aws_sg }}"

      description: "{{ aws_sg }} group"

      region: "{{ aws_region }}"

      rules: "{{ security_rules }}"

Where my j2 template is something like:

---

security_rules:

{% for trusted_host in trusted_hosts %}

  -

    proto: tcp

    from_port: 22

    to_port: 22

    cidr_ip: {{ trusted_host.ip }}/32

  -

    proto: icmp

    from_port: -1

    to_port: -1

    cidr_ip: {{ trusted_host.ip }}/32

{% endfor %}

Andreub

[Jaime Gago]

Nice work around, thanks a lot for sharing, for the record I am doing something similar in other AWS oriented playbooks (e.g. to create mappings of  the AWS random subnets IDs to the actual subnet CIDR I've defined in my var files) but I didn't think about using it here. It seems the template/include var is a common trick for AWS playbooks but maybe that's just my experience, meh.

Thanks again!

J.

[Le Van]

Many thanks for this tip.

It's clever

[Steven Truong]

Thanks for the tips.  I tried this and this worked but to only some extents.  

What do I meant by that and here are the steps that you can repeat to see the potential issue of this ec2_group module.

1. I started out with a vars yml file that has about 9 different IP addresses/32

---
aws_vpc_id: vpc-...
aws_secret_key: zUxxx....xxx
aws_access_key: Axx....xxx
sg_group_name_ssh: ext-ssh-prod

allowed_ssh_hosts:
 - 

 - 

 - 

 - 

 - 

 - 

 - 

 -

 - 

2. Created the modules tasks/main.yml

- name: generate rules
template: src=security_rules.j2 dest={{ ansible_path }}/roles/aws_sg_ext_ssh/vars/ext_ssh_prod.rules
  when: aws_vpc_id == 'vpc-....'

- name: load vars
include_vars: ext_ssh_prod.rules
  when: aws_vpc_id == 'vpc-.......'

- name: ssh access rules
ec2_group:
name: "{{ sg_group_name_ssh | mandatory }}"
description: Allow ssh access from outside of AWS
vpc_id: "{{ aws_vpc_id | mandatory }}"
region: us-east-1
aws_secret_key: "{{ aws_secret_key | mandatory }}"
aws_access_key: "{{ aws_access_key | mandatory }}"
purge_rules: true
rules: "{{ ext_ssh_prod_rules }}"
  when: aws_vpc_id == 'vpc-.....'

3. Created the template:

ext_ssh_prod_rules:
{% for host in allowed_ssh_hosts %}

-
   proto: tcp
   from_port: 22
   to_port: 22

   cidr_ip: {{ host }}
{% endfor %}

4. Created the playbook

- hosts: localhost
  vars_files:
    - vars/vpc_prod_east.yml
  roles:
    - aws_sg_ext_ssh

5. Applied the playbook and things worked as expected.  I saw 9 rules created in the security group

6. Added 8.8.8.8/32 to the end of the ext_ssh_prod_rules

7. Applied the playbook again and a rule was added for 8.8.8.8/32

8. Removed 8.8.8.8/32 from ext_ssh_prod_rules

9. Applied the playbook again and now the rule for 8.8.8.8/32 was not there but 3 more rules also were not there either.  So for some reasons the 3 rules got deleted.

I checked the intermediate roles/vars/ext_ssh_prod.rules every single time and the output has always been correct with either 9 or 10 entries (when 8.8.8.8/32 was addeded).

So in order for us to use this reliably I must apply the playbook again TWICE after removing an IP address.  

I consider this as a bug and will look at the src codes to see what is the situation there.

Cheers and Ansible rocks.

[senorsmile]

Has anyone made any further progress on this yet?

[Kimmo Koskinen]

I ran into this problem, and found a workaround by creating a jinja2 filter module: https://gist.github.com/viesti/1febe79938c09cc29501

- Kimmo