Hurdles for Using EC2 Autoscale Groups with Salt

[Jay Ambi]

Hi All,

Just tested out the ec2-autoscale reactor (https://github.com/saltstack-formulas/ec2-autoscale-reactor) and have things working nicely with SNS notifications hitting the salt-api for any launches or terminations on my EC2 Autoscale Group.

Reactor then goes off and does its thing, renaming the fresh autoscaled instance to the aws instance-id and bootstraps salt and accepts the key on the master.

Working nicely, but it presently poses to hurdles for my organisation..

1) Wishing to switch from our present bash script userdata method of bringing autoscale instances online (currently use the launchconfig userdata init to run a lil bash script which downloads the latest files and sets everything up for the particular server that has autoscaled.

Want to change the process so it is more salt-master-centric eg. the autoscale group launches a server, and the salt ec2-autoscale reactor bootstraps and accepts key and then the salt master runs either the servers highstate or runs a custom state that performs any init tasks to update the environment and bring it online.

I am not sure how to approach this. Searching around seems to suggest that the salt-minion should run a cron to highstate itself, which seems to be more of a 'masterless' solution to me which is not what I want. I always want the master to be running things always. Also as its autoscaling, it needs to respond instantly to configure the instance asap. Again assuming some kind of reactor would be necessary..

Any suggestions to achieve this?

2) The ec2-autoscale reactor above renames instances to the aws instance-id, I'm assuming to simplify things. Admittedly would be nice if I had more control over the naming conventions here eg. allowing something like "$environment-$hostname-$instance-id", and its always only checking for the instance-id portion.

However, its a bit beyond me atm to attempt this functionality myself and this kind of forces me to move from having salt target on minion-id's to targeting with grains.

Currently with all my autoscale groups I configure ec2 tags for the hostname, environment and the class of server eg. environment:web-server, class:production

Rather than setting custom grains for everything how can I make use of existing ec2 tags so that Salt can target them?

[Megan Wilhite]

I have never personally setup this ec2-autoscale-reactor but I have some possible suggestions you could implement. 

In order to ensure a salt-master is running the highstate instead of a cron job, couldn't you just use the reactor as well. If you want to ensure you are only running this on the first initialization of the minion you could try the following:

React off of the following event:  salt/minion/*/start 

During the highstate or state.sls make sure to set a grain like the following:

initialize:
 grains.present:
   - value: True

Then have a reactor sls file similar to the following:

clean_tmp:
 cmd.state.highstate:
   - tgt: {{ "{0} and not G@initialize:True".format(data['id']) }}
   - expr_form: compound

This will only run if the grain initialize has not already been set to True, which means it will only run once, since part of your highstate is to set the grain. 

Also to answer your question about tags and saltstack it looks like salt has the ability to interact with ec2 tags: https://docs.saltstack.com/en/latest/topics/cloud/aws.html#modify-ec2-tags

Thanks,

Megan