Setup SaltStack Reactor for AWS EC2 Events with SQS

[Theodore Cowan]

Created Friday 22 April 2016

This tutorial will show you how to setup SaltStack Reactor system to react to AWS EC2 events like instance start, stop and terminate. This method does not require salt-api or webhooks (Hooray)

Requires SaltStack 2016.3 or this commit https://github.com/saltstack/salt/pull/32557/commits/3031d1b69732559da93cce0fb30eade8cfab7177

With this we will be able to:

1. pre-seed the salt minion key on the master
2. delete a minion key on instance termination
3. Anything else you can think to do with salt executions, runners or salt-cloud.

Instructions

1. Configure AWS SQS queue.

aws sqs create-queue \ 
  --queue-name "saltstack"

2. Create an IAM user with keys. Alternatively, you can use a `machine role` on the Salt Master.

aws iam create-user \ 
  --username <pick a username>

3. Create a SQS policy for the user OR role.

#Assign to a role
aws sqs add-permission \
 --queue-url <from create-queue output> \
 --aws-account-ids "arn:aws:iam::<AWS Account number>:role/saltmaster" \
 --actions * 
#Assign permissions to a user
aws sqs add-permission \
 --queue-url <from create-queue output> \
 --aws-account-ids "arn:aws:iam::<AWS Account number>:user/<username picked in step 2>" \
 --actions *

4. Modify the salt master configuration /etc/salt/master. If you are using machine roles you do not need the key and keyid.

sqs:
  region: us-west-2
  message_format: json
  key: OTHISHISbFaKE014b3gQky3VGET6YOUR8OWNsKEY
  keyid: ALSOIFAKEUNICEITRYI0

engines:
  sqs_events:
    queue: saltstack

5. Configure CloudWatch EC2 Events to send to the SQS queue we created earlier

aws events put-rule \
 --name saltstack \
 --description "Sent instance change events to the SaltStack event bus on the Salt Master." \
 --event-pattern "{ \"source\": [\"aws.ec2\"], \"detail-type\": [\"EC2 Instance State-change Notification\"]}" aws events put-targets \
 --rule saltstack \
 --targets Id=<some random id>,Arn=<From output of sqs create-queue>

6. Configure Reactor. Create the file /srv/reactor/aws.sls on the Salt Master. Put your Reactor events here. Below is an example rule that will delete a minion key on instance termination. IMPORTANT- THIS EXAMPLE ASSUMES THE MINION NAME IS THE SAME AS THE INSTANCE ID

{%- if data['message']['detail']['state'] == "terminated" %}
delete_key:
  wheel.key.delete:
    - match: "{{ data['message']['detail']['instance-id'] }}"
{%- endif %} 

[David Boucha]

This is awesome, Theo!  Thanks for sharing. I wonder if there's a good spot in the official docs for this.

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Ballet]

Yes, this kind of documentation/tutorial is really cool and should really be stored somewhere.

It reminds of the (now missing) "Into the Salt Mine" blog (or I forgot what the name was, which was really useful when I started to learn Salt.

This kind of slightly more advanced topic is very cool, too.

[Ion Mudreac]

Nice solution to the problem

Only one issue I could see as of now May 2016 issue related to AWS CloudWatchRoule availability for all regions. This is not working in all regions Ex ap-southeast-1 will not work rule setup.

I am thinking to setup cron job to check every 5 min will post once make it work.

[seba...@lallemand.fr]

Hello,

This is very interesting and I have tried to implement it.
By the way, I can't get it to work with version 2016.3

The reactor get the message but everytime I had such an error in the logs :
2016-07-01 21:15:40,732 [salt.utils.templates][ERROR   ][14704] Rendering exception occurred: Jinja variable 'str object' has no attribute 'detail'
SaltRenderError: Jinja variable 'str object' has no attribute 'detail'

For me, the message seems good and have the "detail" attribute and to check, I have changed the reactor to :
log :
  local.cmd.run:
    - tgt: 'salt'
    - arg:
      - echo {{ data['message'] }} >> /tmp/aws.log

What I have in the /tmp/aws.log is :
{version:0,id:c9e7c407-49b8-4678-8d8a-2dea9fab8cbc,detail-type:EC2 Instance State-change Notification,source:aws.ec2,account:XXXXXXXXX,time:2016-07-02T08:56:38Z,region:eu-west-1,resources:[arn:aws:ec2:eu-west-1:376141692146:instance/i-2ca045a7],detail:{instance-id:i-2ca045a7,state:running}}

So basicaly, we can se the detail atrribute ... Any ideas ?

Thanks for your help.

[Florian Ermisch]

Hi Sebastien,

I would guess the message is not parsed
(from JSON?) and your reactor just gets a
raw string. Time for some `{% if
data['message'] is not mapping %} …` and
write the message's contents and type
somewhere for debugging.

Regards, Florian

> > 1. pre-seed the salt minion key on the master
> > 2. delete a minion key on instance termination
> > 3. Anything else you can think to do with salt executions,
> runners or
> > salt-cloud.
> >
> > Instructions
> >
> >
> > 1. Configure AWS SQS queue.

> >
> > aws sqs create-queue \
> > --queue-name "saltstack"
> >
> >

> > 1. Create an IAM user with keys. Alternatively, you can use a

> `machine
> > role` on the Salt Master.
> >
> > aws iam create-user \
> > --username <pick a username>
> >
> >

> > 1. Create a SQS policy for the user OR role.

> >
> > #Assign to a role
> > aws sqs add-permission \
> > --queue-url <from create-queue output> \
> > --aws-account-ids "arn:aws:iam::<AWS Account
> number>:role/saltmaster" \
> > --actions *
> >
> > #Assign permissions to a user
> > aws sqs add-permission \
> > --queue-url <from create-queue output> \
> > --aws-account-ids "arn:aws:iam::<AWS Account number>:user/<username
> picked in step 2>" \
> > --actions *
> >
> >

> > 1. Modify the salt master configuration /etc/salt/master. If you

> are
> > using machine roles you do not need the key and keyid.
> >
> > sqs:
> > region: us-west-2
> > message_format: json
> > key: OTHISHISbFaKE014b3gQky3VGET6YOUR8OWNsKEY
> > keyid: ALSOIFAKEUNICEITRYI0
> >
> > engines:
> > sqs_events:
> > queue: saltstack
> >
> >

> > 1. Configure CloudWatch EC2 Events to send to the SQS queue we

> created
> > earlier
> >
> > aws events put-rule \
> > --name saltstack \
> > --description "Sent instance change events to the SaltStack event
> bus on the Salt Master." \
> > --event-pattern "{ \"source\": [\"aws.ec2\"], \"detail-type\":
> [\"EC2 Instance State-change Notification\"]}" aws events put-targets
> \
> > --rule saltstack \
> > --targets Id=<some random id>,Arn=<From output of sqs create-queue>
> >
> >

> > 1. Configure Reactor. Create the file /srv/reactor/aws.sls on the

[seba...@lallemand.fr]

Thanks for your answer Florian.

Here is what I have done so.

The reactor is now :
{% if data['message']['detail'] is not mapping %}
debug:
  local.cmd.run:
    - tgt: 'salt.cormeilles.lallemand.fr'
    - arg:
      - echo DEBUG : {{ data['message'] }}
{% else %}
log :
  local.cmd.run:
    - tgt: 'salt.cormeilles.lallemand.fr'
    - arg:
      - echo {{ data['message']['detail'] }} >> /tmp/aws.log
{% endif %}

and what is in the master log file :
2016-07-02 15:03:09,353 [salt.template    ][DEBUG   ][28676] Rendered data from file: /srv/reactor/aws.sls:

debug:
  local.cmd.run:
    - tgt: 'salt.cormeilles.lallemand.fr'
    - arg:
      - echo DEBUG : {"version":"0","id":"69313f00-2a80-42ff-b3a5-fb262330598c","detail-type":"EC2 Instance State-change Notification","source":"aws.ec2","account":"xxxxxxxx","time":"2016-07-02T13:03:08Z","region":"eu-west-1","resources":["arn:aws:ec2:eu-west-1:xxxxxxxx:instance/i-2ca045a7"],"detail":{"instance-id":"i-2ca045a7","state":"stopped"}}


so, yeah, I guess you are right and the message is badly parsed.

I have also check the events received on the master :
salt-run state.event pretty=True
salt/engine/sqs    {
    "_stamp": "2016-07-02T13:17:13.342244",
    "message": "{\"version\":\"0\",\"id\":\"8b7905bd-4974-4358-a882-04fce3d58815\",\"detail-type\":\"EC2 Instance State-change Notification\",\"source\":\"aws.ec2\",\"account\":\"xxxxxxxx\",\"time\":\"2016-07-02T13:17:12Z\",\"region\":\"eu-west-1\",\"resources\":[\"arn:aws:ec2:eu-west-1:376141692146:instance/i-2ca045a7\"],\"detail\":{\"instance-id\":\"i-2ca045a7\",\"state\":\"pending\"}}"

> Notification,source:aws.ec2,account:XXXXXXXXX,time:2016-07-02T08:56:38Z,region:eu-west-1,resources:[arn:aws:ec2:eu-west-1:xxxxxx:instance/i-2ca045a7],detail:{instance-id:i-2ca045a7,state:running}}

[Florian Ermisch]

Hi Sebastien,

so you're getting raw JSON, that's not too bad.
Can parse the message with the `load_json` tag:

/srv/salt/base$ sudo salt-call pillar.items test
local:
----------
test:
----------
json:
{"foo": "bar", "message":
"salt FTW!"
, "id": "localhost"}
/srv/salt/base$
/srv/salt/base$ cat test.sls
{% set data = salt['pillar.get']('test:json') %}
{% if data is not mapping %}
{% load_json as parsed %}
{{ data }}
{% endload %}
{% else %}
{% set parsed = data %}
{% endif %}
/tmp/saltiest:
file.managed:
- contents: {{ parsed['message'] }}
/srv/salt/base$
/srv/salt/base$ cat /tmp/saltiest
salt FTW!

Of course, my data here is always JSON but
with such a construct you won't care if you
got the data parsed or have to parse it
yourself.

Regards, Florian

[Szasz Tamas]

I can confirm that with salt 2016.3 this isn't working as expected. I have the same issue, that data['message'] is not parsed as json, it's a string. 

[Daniel Wallace]

The commit that is mentioned in the first message is not in 2016.3, it will be in Carbon though.

If you want to use this now, you will have to do the {% load_json as thing %} mentioned by Florian.

Daniel

[Jens Rantil]

Just stumbled across this thread. Interesting! Does anyone have a solution also for accepting minions? Also, is there a race condition between SQS event is published and the minion has connected to the master (and submitted its key)?

Cheers,

Jens

[Michael McKinney]

I found that accepting the key when you get the SQS message does not work, since the minion hasn't started yet. My auto accept reactor looks like this and looks for any salt/auth events, rather than the SQS event.

{%- if data['act'] == "pend" %}

{%- set instanceid = data['id'].split('.')[-5] %}

invoke_check_id_orch:

  runner.state.orchestrate:

    - mods: orchestration.checkid

    - pillar:

      event_tag: {{ tag }}

      event_data: {{ instanceid }}

accept_key:

  wheel.key.accept:

    - match: {{ data['id'] }}

    - require:

      - salt: invoke_check_id_orch

{%- endif %}

I build an orchestration/runner to compare the instance id being provided (it's build in as part of the hostname in the autoscaling group) to a list of active instances and to accept if it is found. 

[Theodore Cowan]

7. Add the reactor to the salt master configuration.

reactor:
  - 'salt/engine/sqs':
    - /srv/reactor/aws.sls

Forgot one step.  Note that the changes required for this to work in salt/engines/sqs_events.py have not been released yet.

[Jens Rantil]

Hi,

I put together a working auto-scaling reactor in our systems over the weekend. So far it's been working like a charm. I got a request on Twitter to share. Maybe could of interest for someone else in this thread:

https://github.com/JensRantil/saltstack-autoscaling

Also, I'd love input if you have any comments.

Cheers,

Jens

[Love Nyberg]

Really nice work Jens! Looking forward to play around with it! 

[sfresher]

Is there any way to put an timed event?  e.g.  create an event that will start 30 days from today?

Thank you.

[Jens Rantil]

Hi sfresher,

You could always use the UNIX `at` command to schedule sending an event to the Salt event bus. Or you could use `cron`. Or you could use Salt's scheduler[1].

[1] https://docs.saltstack.com/en/latest/topics/jobs/#scheduling-jobs

Cheers,

Jens