Best scalable approach to bootstrapping an EC2 server from within AWS Cloudformation?

[KeithA]

I am relatively new to Ansible (and config. management in general) so please excuse me if I sound confused. I encourage you to correct me or clarify me if I am wrong.

 I have setup an AWS Cloudformation template through AWS and not Ansible, have defined my infrastructure, auto-scaling group and launch configurations for out-scaling servers in a VPC. I understand I have a few choices to bootstrap the servers as they are launched: (I'm not 100% sure if these are the correct ways to implement the choices)

1. Use metadata & cloud-init in Cloudformation to run bash scripts directly
   
2. Pre-bake AMI's with packages and run playbooks locally when the server is launched.
3. Install packages, Ansible, copy playbooks from a private repo, and run playbooks locally
4. Install packages, Ansible, and use 'ansible-pull'
   

What I am confused about is that choices #2,3,4 require Ansible to be installed just to run a local playbook. But would the servers even use Ansible after it's initial bootstrap? Is it really necessary to have Ansible on the remote server (if so, could you explain a use case)? I'm assuming after it's ready I would no longer use Ansible on the remote server and instead provision/update the servers from my control server.

I want to make it automated so that I don't have to:

1. install Ansible on the remote server if I don't have to or it can be done another, more scalable, way.
   
2. SSH into the remote server to install all the initial packages and code manually.

I was considering choice #1 because I can install packages, start nginx, and my application through bash directly in the cloud-init/metadata. However I am assuming it's not very dynamic.

What are your thoughts/opinions/advice? Thank you for your help in advance! 

[Stephen Granger]

I've been implementing a bit of 1, 3, 4 to bring up ASG instances.

I, like many people dislike having unnecessary software on my hosts (instances) so thought this could be another way to carry out the same process - Running Ansible from AWS Lambda

https://medium.com/@jacoelho/ansible-in-aws-lambda-980bb8b5791b#.u8t6gsbvp

Though I haven't yet figured out how to do this yet with the 2.0 api of Ansible. It would be really nice because then you could further restrict the access/rights that the instances have. It's really annoying when you aren't using an elb and have to then also update route53 values when the instances comes up.

--
You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/0511462d-9c02-4a95-822a-ee98e8d06992%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Steve

[KeithA]

How are you implementing these? I was thinking of taking advantage of AWS's Lifecycle Hooks to trigger a Lambda function to have Ansible run playbooks on the pending server but I think that may be a little too much. 

[Stephen Granger]

1. Use metadata & cloud-init in Cloudformation to run bash scripts directly

3. Install packages, Ansible, copy playbooks from a private repo, and run playbooks locally

4. Install packages, Ansible, and use 'ansible-pull'

I get the lc user data script to install ansible, pull from an s3 bucket and then just run Ansible locally (not ansible-pull). I have it run a bootstrap playbook which get's the instance tags from the metadata. In the ASG configuration I have a tag, ansible_host_group which includes the host groups this instance will belong to in the Ansible (dynamic ec2) inventory. The bootstrap playbook uses add_host to add the localhost i.e. the instance to the ansible_host_group group, site.yml is then included. site.yml includes the other application tier playbooks, webserver.yml, database.yml etc. The hosts: value restricts what instances run the plays/roles. 

# webserver.yml

---
- hosts: webserver:tag_ansible_host_group_webserver

  roles:

    - common

    - webserver

The tag* host value is just if I need to run the plays on running instances, which shouldn't really ever be the case.

It requires installing Ansible and aws-cli tools on the host and giving access to an s3 bucket and the instances metadata. A lot of people are fine with this, but I feel less is more. Autoscaling lifecycle hooks seems like a good idea to and can now call out to a lambda (python i.e. Ansilbe) functions. It's wait and success/failure features seem like a smart option too.

I was also thinking of triggering an SNS topic to execute a lambda function (i.e. Ansible). It's kinda the same thing, lifecycle hooks seem like a better choice though aren't as popular as SNS I've found.

I would like to find time to implement it a lambda, Ansible bootstrap solution.

--

You received this message because you are subscribed to the Google Groups "Ansible Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ansible-proje...@googlegroups.com.
To post to this group, send email to ansible...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ansible-project/65c4d109-9f94-454f-8c71-968181241adf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Steve

[KeithA]

Cool, so I feel like I'm on the right track! I was thinking of doing it similarly. Thank you for replying so quickly. 

On Thursday, February 25, 2016 at 8:05:31 PM UTC-8, Stephen Granger wrote:

1. Use metadata & cloud-init in Cloudformation to run bash scripts directly

3. Install packages, Ansible, copy playbooks from a private repo, and run playbooks locally

4. Install packages, Ansible, and use 'ansible-pull'

I get the lc user data script to install ansible, pull from an s3 bucket and then just run Ansible locally (not ansible-pull). I have it run a bootstrap playbook which get's the instance tags from the metadata. In the ASG configuration I have a tag, ansible_host_group which includes the host groups this instance will belong to in the Ansible (dynamic ec2) inventory. The bootstrap playbook uses add_host to add the localhost i.e. the instance to the ansible_host_group group, site.yml is then included. site.yml includes the other application tier playbooks, webserver.yml, database.yml etc. The hosts: value restricts what instances run the plays/roles. 

# webserver.yml

---
- hosts: webserver:tag_ansible_host_group_webserver

  roles:

    - common

    - webserver

The tag* host value is just if I need to run the plays on running instances, which shouldn't really ever be the case.

It requires installing Ansible and aws-cli tools on the host and giving access to an s3 bucket and the instances metadata. A lot of people are fine with this, but I feel less is more. Autoscaling lifecycle hooks seems like a good idea to and can now call out to a lambda (python i.e. Ansilbe) functions. It's wait and success/failure features seem like a smart option too.

I was also thinking of triggering an SNS topic to execute a lambda function (i.e. Ansible). It's kinda the same thing, lifecycle hooks seem like a better choice though aren't as popular as SNS I've found.

I would like to find time to implement it a lambda, Ansible bootstrap solution.