How do you guys store secrets?

[Stephen Wood]

I'm interested in hearing about how you all store your secret files -- the things that should never end up in source control.

We've started keeping everything in a special directory which becomes part of the file/pillar_roots of the salt master. When changes are made the directory is encrypted, versioned, and stored some place off of the server.

How about you guys?

--

Stephen Wood

[Corey Quinn]

Separate git repo for pillar with tightly controlled access is the current standard. 

--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Bowen]

I've seen a few people ask about encrypted pillar data previously and thought to myself "surely it can't be too hard".

For the case of simply making sure that passwords don't get pushed to git in clear text (which is much easier than trying to secure them in the case that the saltmaster or a minion was compromised) you could use the following:

1. Create a GPG key pair on your salt master without a password (insecure yes, but the goal is to encrypt data before it's pushed to git, not secure it on the server).

2. Save any pillar data that needs to be encrypted under /srv/pilllar/secrets.sls.gpg

3. Add the following to /etc/salt/master

ext_pillar:

  - cmd_yaml: gpg -d -q /srv/pilllar/secrets.sls.gpg

Note: The -q is for "quiet" and ensures that only the yaml is returned.

I've done this in a test (Vagrant) environment and it works well. When editing the .gpg file in emacs (and I assume vi could do the same) the file is automatically decrypted and encrypted on opening and saving.

I have thought about ways to better secure pillar data to help reduce the implications of a minion or the master being compromised. I have some ideas, but haven't settled on an "optimal solution" yet. There's a lot more that could be done by writing an external pillar handler in python (including hooking in to a password management server as I've suggested in a previous thread).

-

Bowen

[Bowen]

I should note that I haven't used this method in production (nor do I have a need to at present) and don't consider it to improve the security of anything, other than encrypting the pillar data before it gets pushed to github.com / whereever. Also, the security that is added using this method is only going to be as good as your ability to keep the private gpgkey secure (so if you're taking backups of your salt master, be aware that it's going to be on there unprotected and with your encrypt pilllar file)

So yeah, many draw backs to this, but it might be enough to help point people in the right direction of a more secure solution.

-

Bowen

[Sas]

Anyone usinf Salt with Vault?  I am looking into any solution for using Vault opensource version with Saltstack .

Thanks

Sas

[Mike Freitas]

I played with it a little bit and overall it was promising. It has been a while since i tried and there were some big improvements that are probably released. I am not incredible familiar with ext_pillars, but it seemed like you couldn't target systems using the external pillar so if you setup vault as an external pillar, all minions would get that data. In addition you have to choose a "mount" point to "export" as a pillar. The SDB interface on the other hand looked promising for being able to target queries more easily. I plan to use Vault with Salt, but i have not had the time to dig into the Vault/Consul production worthy deployment.

Anyone who is more familiar with external pillars please correct me if I am mistaken on the behavior i described above. I know it's not the best answer, but I did see enough value in my POC to keep it on the list of things to add.

Mike

[C. R. Oldham]

There is a vault formula in the salt-formula's repository.

https://github.com/saltstack-formulas/vault-formula

--cro

C. R. Oldham

Senior Platform Engineer, SaltStack

c...@saltstack.com / 801-564-4673

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/2ba2667c-7270-4244-9044-a0e551244a3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Phipps]

okay, couple of things to correct here.

ext_pillars pass the minion that is requesting the pillar to the ext_pillar function. so targeting can happen inside of the pillar. in fact one of the best known ext_pillars has it's own full top. git_pillar. however if a ext_pillar is returning all of the pillar data to every node. that is an issue with that ext_pillar that needs to be addressed. at the very least minion id targeting should be respected.

not sure what you mean by a mount point. having used several ext_pillars over the years i never had to do anything like this. unless you are talking about the path setting for pillar vault. if so that is for pathing with in vault. not a mount point but a way to save the keys in a path within vault. see https://www.vaultproject.io/docs/commands/help.html for more about pathing. 

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.

[Tobias Macey]

Hi Sas,

I have been doing a fair bit of work to integrate Vault with Salt at my work. For a reasonably detailed writeup of how it hangs together you can refer to https://github.com/mitodl/salt-ops/blob/master/docs/infrastructure/vault.md

That has pointers to all of the associated repos that I wrote to get things hanging together. I have also opened an issue here (https://github.com/saltstack/salt/issues/40043) to discuss getting that worked merged into upstream which is currently stalled due to other projects at work. If any of that is useful to you I am happy to answer questions, and if you want to help me get that merged upstream then that would be great too :)

Regards,

Tobias Macey

On Friday, June 2, 2017 at 4:08:50 PM UTC-4, Sas wrote:

[Mike Freitas]

Hi Thomas,

Thanks for the ext_pillar clarification. When I get back to Vault that will be helpful. I think the issue I had was that I didn't know how to restrict secrets in Vault to minions. I wasn't able to figure out how to do what the pillar top file does. Again, it was very likely user error. 

I was referring to a metaphorical mount point because at the time, and possible still, you had to choose a base path to add as an external pillar, much like mounting a remote filesystem. I agree it was a bit abstract, but made sense at the time to my tired Friday brain . :)  

[Thomas Phipps]

what you are describing sounds like namespaces.

but not every ext_pillar supports those. nor has it ever been something that needs to be configured within ext_pillar.

To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/95754bdc-d9cc-4fd7-a2a8-261c1262920c%40googlegroups.com.

[Charles McLaughlin]

At my job we use Salt's built-in GPG renderer extensively.  Here are the docs:

https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html

That allows us to store encrypted secrets in Pillar and commit to Git  In addition to protecting against GitHub leaks, all developers at my company can work on our Salt repo... but only admins can encrypt/decrypt our secrets.

As previously noted, the downside is the GPG key does exist on the salt-master.  That's good enough for me at the moment, but I wonder if something like AWS Key Management Service and possibly hardware security modules could be used to harden this scenario. 

We also use Vault for in-house application tokens, but our source-of-truth for Vault data is Pillar - we push secrets from Pillar to Vault.

[Christian McHugh]

We have started using the newly added vault module/runner/util/state https://github.com/saltstack/salt/pull/39212

Vault itself is a bit of a framework more than a complete solution. If you take the problem of keeping username/passwords out of config or script files in git, vault can replace that with an https get to the vault server. You then have a vault token which has a policy which allows it to lookup the secret, and you place that call in your script or config file. This is better than keeping your password secret and all versions in git, but it still allows anyone with the token to perform the vault lookup themselves, so it isn't great.

The referenced salt modules use vault, but tie it in very nicely with salt.

The communication begins with the salt-minion which uses a vault.read_secret module to ask the salt-master for a vault token (over the encrypted salt bus). The salt-master then uses information about the salt-minion to ask the vault server for a one time token and in that process attempts to apply policies to that token based on the information about the minion. The vault server then replies with the one use token to the salt-master, and the salt-master sends it on to the minion. The salt-minion then performs a normal vault https lookup for the password, using it's new token and vault replies with a response.

This allows for vault use where secrets can be delegated to minions based on grain or pillar info. This allows for secrets access use cases such as:

• All mysql servers in web app tier in production
• Single Jenkins Master
• All salt-minions

By using salt as an interface on top of vault, it allows you to leverage salt's knowledge of the environment to more or less replace vault token handling.

We are now using vault as the source of truth for our passwords, and use this system to seed Jenkins credentials.

Additionally, since you can both read and write vault secrets, this allows for other use cases such as using a bastion host to write a let's encrypt ssl cert key to vault, then allowing the web host to read it. Thus creating an encrypted file transfer which only allows the necessary minions to view contents. 

This is a much more robust solution over the current vault pillar which simply makes vault secrets available to all salt minions. If you have any interest in using vault, I would recommend looking at the newer modules. 

[Anthony Martinet]

Seems promising !

Any way on how to automate policy creation ?

Salt is getting token with this policies secret/{minion}/*

But I need to create all policies on vault before that.

It would be great if that could be dynamic

-- Anthony Martinet

alkivi.fr

--

You received this message because you are subscribed to the Google Groups "Salt-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/c30155c8-781e-4a01-81e7-ba981976c321%40googlegroups.com.

[Tobias Macey]

Hi Anthony,

I've implemented an alternative interface to Vault which you can
find in the formula at https://github.com/mitodl/vault-formula

The details of our interaction with Vault are detailed at
https://github.com/mitodl/salt-ops/blob/master/docs/infrastructure/vault.md

> *

>
> The communication begins with the salt-minion which uses a
> vault.read_secret module to ask the salt-master for a vault
> token (over the encrypted salt bus). The salt-master then uses
> information about the salt-minion to ask the vault server for a
> one time token and in that process attempts to apply policies
> to that token based on the information about the minion. The
> vault server then replies with the one use token to the
> salt-master, and the salt-master sends it on to the minion. The
> salt-minion then performs a normal vault https lookup for the
> password, using it's new token and vault replies with a
> response.
>
> This allows for vault use where secrets can be delegated to
> minions based on grain or pillar info. This allows for secrets
> access use cases such as:
>

> * All mysql servers in web app tier in production
> * Single Jenkins Master
> * All salt-minions

>
> By using salt as an interface on top of vault, it allows you to
> leverage salt's knowledge of the environment to more or less
> replace vault token handling.
>
> We are now using vault as the source of truth for our
> passwords, and use this system to seed Jenkins credentials.
>
> Additionally, since you can both read and write vault secrets,
> this allows for other use cases such as using a bastion host to
> write a let's encrypt ssl cert key to vault, then allowing the
> web
> host to read it. Thus creating an encrypted file transfer which
> only allows the necessary minions to view contents.
>
> This is a much more robust solution over the current vault
> pillar which simply makes vault secrets available to all salt
> minions. If you have any interest in using vault, I would
> recommend
> looking at the newer modules.
>
> --
> You received this message because you are subscribed to the
> Google Groups "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from

> it, send an email to salt-users+...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/salt-users/c30155c8-781e-4a01-81e7-ba981976c321%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
Regards,
Tobias Macey
https://linkedin.com/in/tmacey

[Charles McLaughlin]

Here's how I'm doing policy creation:

https://gist.github.com/cmclaughlin/500fba42eb26e8091c1c20cdbe5e094d

--
You received this message because you are subscribed to a topic in the Google Groups "Salt-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/salt-users/Qi_cDyG4GJc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to salt-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/salt-users/CADFNXNHcaWnSt0HsrCN%3Dti9Kb7dbCwusJtyfdGA9TmM0UUOGxQ%40mail.gmail.com.

[Christian McHugh]

Rather than just limiting yourself to policies named secret/{minion}/* you are also also to add your own policy names. Our vault config in the salt master looks like:

    vault:

      url: https://vault.url

      auth:

        method: token

        token: sdb://osenv/VAULT_TOKEN

      policies:

        - saltstack/minions

        - saltstack/minion/{minion}

        - saltstack/minion/{grains[ec2_tags][Environment]}/{grains[ec2_roles]}

        - saltstack/minion/{grains[ec2_tags][Environment]}

        - saltstack/minion/roles/{grains[ec2_roles]}

Thus we are able to set a policy for a host based on dynamic information. (there are some issues with a grains based approach as mentioned in https://github.com/saltstack/salt/issues/43287. But for our environment, the same team operations team controls machines and vault, so there is limited additional risk)

This allows us to then use the vault policy management states to set needed policies:

salt-master-policy:

  vault.policy_present:

    - name: saltstack/minion/roles/saltmaster

    - rules: |

        path "secret/AD/saltmaster-service" {

          policy = "read"

        }

        path "secret/salt-master/jenkins" {

          policy = "read"

        }

-- Anthony Martinet

alkivi.fr

To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+...@googlegroups.com.