Idiomatic EC2 project structure

[Dominic Bou-Samra]

Hi all,

I've been playing with Ansible as a solution to our deployment infrastructure. Our stack is:

• Dozens of "transporters" running on EC2.
• Dozens of "detectors" running on EC2
• Production and staging environments
• Multiple regions (atm US East and Sydney)

What I require from Ansible is:

• A dynamic inventory, MIXED IN with a static group declaration. I would like to be able to refer to my "Australian transporters in production", as easily as possible. I would like no IP's/hostnames stored locally (i.e, use EC2.py as a dynamic inventory).
• Ability to provision new EC2 instances in any region, in any environment.
• As little duplication as possible.

I am struggling with trying to lay my project out so that I can achieve these goals. 

Could anyone provide me with some suggestions or example layouts, ideally with sample command? Let me know if you need more information and what I have so far

[Lorin Hochstein]

Hi Dominic:

To support mixing static with dynamic inventory, have the "hostfile" entry in your ansible.cfg point to a directory instead of a file. For example, my ansible.cfg contains:

[defaults]

hostfile = inventory

And my inventory directory looks like this:

inventory/hosts

inventory/ec2.py

inventory/ec2.ini

The inventory/hosts file is a static Ansible inventory file. The inventory/ec2.py and inventory/ec2.ini are the dynamic inventory parts. You can edit ec2.ini to specify which regions you want, the example ec2.ini file that ships with ansible is pretty clear on how to do this.

I gave a talk on this topic a few months back, the slides may not mean too much without me talking (they're pretty sparse) but here they are:

http://go-talks.appspot.com/github.com/lorin/camp-devops-talk/talk.slide

You can provision instances using the ec2 module. I recommend that you use tags when you do provisioning because ec2.py will automatically create groups based on tags. One gotcha here is that ec2.py will cache by default, so if you want to launch an instance and then configure it immediately after it comes up, you'll need to disable caching before in ec2.ini (set cache_max_age=0).

If you want to do "Australian transporters in production", I don't think you can actually define a new group as an intersection of groups, so you'd have to specify the hosts by explicitly doing the intersection of groups:

hosts: australia:&transports:&production

(where "australia", "transports" and "production" are all groups that I have assumed you have created via tags).

[Dominic Bou-Samra]

Hi Lorin,

Thanks for replying. 

So the above presentation is exactly the solution we had arrived at in our Ansible spikes. I wish I had seen that 1 week ago :P. 

You are right about the intersection of groups - it's a shame you cannot declare hosts like (excuse any syntax issues - code looks funny on a phone):

[staging]

[staging:children]

tag_env_staging

[transporters]

[transporters:children]

tag_type_transporter

[staging_transporters]

[staging_transporters:children]

staging

transporters

And then in your playbook, declare:

hosts: staging_transporters

But perhaps it's better to pass in variables like:

// configure all staging transporters in us-east-1. Staging inventory declares group_var with staging param

ansible-playbook -i staging -e zone=us-east-1 transporter.yml 

With transporter.yml looking like:

- name: Config transporters

  hosts: {{environment}}:&{{zone}}:&transporters

What do you think of that?

[Michael DeHaan]

"You are right about the intersection of groups - it's a shame you cannot declare hosts like (excuse any syntax issues - code looks funny on a phone):"

Can you elaborate on what you think is missing?  ":children" to define child groups is how that part works, so I'm missing the understanding of what part you find lacking there.

Thanks! 

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/fb2e7d05-0664-445a-81a9-64c6b2a8ab0e%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Lorin Hochstein]

On Thu, Aug 14, 2014 at 7:42 AM, Michael DeHaan <mic...@ansible.com> wrote:

"You are right about the intersection of groups - it's a shame you cannot declare hosts like (excuse any syntax issues - code looks funny on a phone):"

Can you elaborate on what you think is missing?  ":children" to define child groups is how that part works, so I'm missing the understanding of what part you find lacking there.

I can create a new a group as a union of other groups (groups of groups), but AFAIK there's no way to create a new group as the intersection of other groups.

For instance, if I have a group named "atlanta" and a group named "webservers", there's no way to define a group called "atlanta_webservers" that refers to the intersection of those two groups. You have to do "hosts: atlanta:&webservers" in plays.

Intersections would be really useful because I want to tag my EC2 instances like:

type=webserver

env=production

And then define a "production_webservers" group as the intersection of these. 

Lorin

 

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/Ohx3KEYwZdw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.

To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgym88h0GsehetMbygfPGwzvKJTkgmNs0fabhmCdgN%2BD8w%40mail.gmail.com.

[Michael DeHaan]

"I can create a new a group as a union of other groups (groups of groups), but AFAIK there's no way to create a new group as the intersection of other groups."

Shouldn't be needed in the inventory file, as this is something you can address using the host spec:

- hosts: alpha:&beta

or also

- hosts: alpha

with

ansible-playbook foo.yml --limit beta

Either of these could be used to apply tags.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACkJihNU1xdY75QXkub2yVBR19XsWy7zXJBsbzhu4uWu1GMH4A%40mail.gmail.com.

[Dominic Bou-Samra]

Perhaps I am doing it wrongly (thus the idiomatic keyword), but I find that less declarative then being able to define intersections in groups. 

[Michael DeHaan]

It's not an issue of declarative vs imperative here, but saying "talk to these groups".

The children are not technically unions either, they are child relationships.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/37de6fe4-748e-444e-be96-1508ded8675c%40googlegroups.com.

[Lorin Hochstein]

I would also find it useful to be able to define a group as an intersection of two groups. It's something I've wanted for a while but just haven't gotten around to submitting a feature request for.

If I use the "hosts: dbservers:&production:&northeast" syntax in N playbooks, then I have N opportunities to make a typo that will have very bad consequences (leaving out the & by mistake).

Also, if I was able to define a group called dbservers_production_northeast, and then I decided to change the definition of the group, I'd only have to change it in one place. If I use it in N places, I'd have to go and change dbservers:&production:&northeast in N places. Yuck. 

Relying on the "--limit" argument of ansible-playbook is even more error-prone: there's a risk of applying the playbook to the wrong hosts *every time* the playbook is invoked by forgetting the --limit argument.

It would be useful if ansible supported pattern syntax for defining children groups, so users could do something like this:

[dbservers_production_northeast:children]

dbservers:&production:&northeast

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CA%2BnsWgxy6d2Os1nX0ADQumK%2BRLcjKbqLv_qei7-eDy0%3DR0xqyQ%40mail.gmail.com.

[Michael DeHaan]

" would also find it useful to be able to define a group as an intersection of two groups. It's something I've wanted for a while but just haven't gotten around to submitting a feature request for."

This is not something we'd be interested in, most likely, as it's already doable from the host spec.

As a matter of saving time with the ticket system, always a good idea to discuss possible features on ansible-devel first to see if we have good solutions for them, and if we agree, maybe then it becomes a ticket.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CACkJihMfwk%3D9jBENe5WsZsnjNBprGkv82kOQ3g%2BUx660CcSMZg%40mail.gmail.com.

[Brian Jones]

Agree with this actually being useful.  The hosts syntax works fine, but makes the playbook less reusable.  Why would I want to create two copies of the same playbook, one for ec2 and one for local vms where the only difference is the hosts: specification?  Why would I start off making a vm and provisioning it using ec2 terminology in the hosts line?  It doesn't make sense.  Obviously you could just use the ec2 names as groups for your local vms and all is well, just more typing with lots of underscores.

I have little desire to put hosts: tag_Type_Web all over playbooks.

I could execute a group_by: ec2_tag_Type but that doesn't make the group available to --limit.

And apparently as discussed earlier in this thread, creating a static inventory file referencing a dynamic group as a 'child' doesn't work.

Assume you do the following, and set hostfile to ./inventory

./inventory/vagrant

./inventory/ec2.py

./inventory/production

# inventory/vagrant

[web]

vm1

[db]

vm2

[vagrant]

vm1

vm2

# inventory/production

[web:children]

tag_Type_Web

[production:children]

tag_Type_Web

tag_Type_Db

ERROR: child group is not defined: (tag_Type_Web)

The example given previously is just a more complicated version of this.

[dbservers_production_northeast:children]

dbservers:&production:&northeast

Goal is something like

ansible -l 'vagrant&web'

ansible -l 'production&web'

With playbooks referencing

- hosts: web

Brian

[Timothy Gerla]

And apparently as discussed earlier in this thread, creating a static inventory file referencing a dynamic group as a 'child' doesn't work.

<snip> 

ERROR: child group is not defined: (tag_Type_Web)

The example given previously is just a more complicated version of this.

 

It's actually possible to do this easily enough if you also define the tag_Type child groups in the static inventory as well:

# inventory/production

[tag_Type_Web]

[tag_Type_Db]

[web:children]

tag_Type_Web

[production:children]

tag_Type_Web

tag_Type_Db

-Tim 

--

Tim Gerla
t...@gerla.net

[Brian Jones]

I decided to just edit ec2.py to create the groups I wanted. 

Good to know about the empty group trick though!

Thanks,

Brian

--
You received this message because you are subscribed to a topic in the Google Groups "Ansible Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ansible-project/Ohx3KEYwZdw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/CAH4wdVXQhe8Lw0oEkp69nkV9EJeHxKgpmJet4gFvY8R2dcjgaQ%40mail.gmail.com.