Salt Pillar Encryption Question

1,292 views
Skip to first unread message

Bryan Murphy

unread,
Dec 1, 2013, 4:52:58 PM12/1/13
to salt-...@googlegroups.com
I'd like to pre-encrypt the salt-pillar data before deploying to the salt-master.  When a minion requests configuration, the minion (not the master) would be pre-seeded with the key to decrypt the data.  The salt-master would never have the decryption key.

Is there a way I can do this?  

Thanks,
Bryan

Joseph Hall

unread,
Dec 1, 2013, 5:47:15 PM12/1/13
to salt-...@googlegroups.com
This sounds similar to encrypted data bags in Chef, and has been
requested in the past. I don't think it makes sense in the Salt
architecture, since the salt-master has full access to the minion, and
therefore would have access to any keys on the minion.
> --
> You received this message because you are subscribed to the Google Groups
> "Salt-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to salt-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.



--
"In order to create, you have to have the willingness, the desire to
be challenged, to be learning." -- Ferran Adria (speaking at Harvard,
2011)

Bryan Murphy

unread,
Dec 1, 2013, 7:58:37 PM12/1/13
to salt-...@googlegroups.com
That's unfortunate, that could be a deal breaker for us (at least for storing configuration inside of salt pillars).  

I guess the problem is that because the salt-master can push out arbitrary python code to the minion, the salt-master once compromised could push code out to the minion which exposes the decryption key as grain data.  This effectively gives the salt-master access to the encrypted data.

With our existing configuration management system (which we would like to replace with salt) we store the configuration as encrypted tar files.  We fetch them from a web server, decrypt, apply, and discard.  We could do something similar with salt, but we'd lose all the benefits of the salt master/minion architecture which I was hoping to avoid.

Joseph Hall

unread,
Dec 1, 2013, 8:16:28 PM12/1/13
to salt-...@googlegroups.com
I think any architecture (Salt or otherwise) that involves a master
will encounter the same kind of problem. As soon as the master is
compromised, it can be configured to push arbitrary code out to its
minions/clients/etc. I've seen various workarounds to this specific
problem proposed, and every single one has quickly been proven
insecure. It's a great idea, but as one user pointed out several
months ago, it's little more than security theater.

Seth House

unread,
Dec 1, 2013, 8:27:54 PM12/1/13
to salt-...@googlegroups.com

What is your use case for storing configuration in pillar? Pillar is private to individual minions so each minion would not have access to other minions pillar.

However if you're hoping to keep configuration data private from the master it becomes a lot trickier. Since, as Joseph said, the minion does everything the master asks it and the master could just ask the minion for its key. On top of that since the minion is usually running as the root user it has free rein of the system.

Not impossible. The minion does not have to run as the root user. However that becomes a lot more tricky to set up which brings me back to my question about your use case. Perhaps there's another way.

Seth House

unread,
Dec 1, 2013, 8:33:56 PM12/1/13
to salt-...@googlegroups.com

One other thing I meant to point out is you can limit what the master is able to do if you configure salt's external authentication system or client ACL system and run the Salt master as a non-root user.

Bryan Murphy

unread,
Dec 2, 2013, 10:20:01 AM12/2/13
to salt-...@googlegroups.com
On Sun, Dec 1, 2013 at 7:27 PM, Seth House <se...@eseth.com> wrote:
What is your use case for storing configuration in pillar? Pillar is private to individual minions so each minion would not have access to other
minions pillar.

We have a few custom built processes that generate our configuration via templates in a manner very similar to how salt uses pillar data + jinja2.  The problem is it's messy (bash/ruby scripts), not well documented, not well tested and we have a lot of configuration.  I see no reason to continue supporting this infrastructure if salt provides the same functionality.

What I have to do is keep our customers isolated inside their own environments (multi-tenancy).  At the moment, we partition our customers onto completely separate infrastructure with severely restricted employee access (us, not them).  The allocation of this physical infrastructure is costly, as is the burden of maintaining it.  I'd like to decrease our footprint where possible (more shared infrastructure) but we have to continue to maintain the metaphorical firewall between our customers.

I've been through 3 security audits in the last year, so balancing all these variables has been... challenging... :)  

I can envision a scenario where salt-minions would only execute code that has been cryptographically signed for their environment, and thus the master becomes more of a distribution mechanism.  I realize that may not be realistic given the current state of the project.  That's ok, I'm simply exploring my options. 

Thanks,
Bryan

Seth House

unread,
Dec 3, 2013, 2:07:31 AM12/3/13
to salt-...@googlegroups.com
Very interesting. Thanks for sharing.

Does the data need to be encrypted to pass a security audit or would
it be enough to simply restrict access to unauthorized
users/employees?

Bryan Murphy

unread,
Dec 3, 2013, 12:33:38 PM12/3/13
to salt-...@googlegroups.com
Our customers require that their data is encrypted at "rest".  That means if it's sitting on a server somewhere it should be encrypted and the decryption key is stored somewhere else.  If a machine (and the data on it) is compromised, the data is useless unless the decryption key is also compromised.

If the data is decrypted, it should only be decrypted for as short of a period as possible and the decryption key should be discarded as soon as it is no longer required.  Needless to say, this leads to a bit of a key management nightmare.  

We have separate systems for key management.  The problem is, you also need to authenticate with these systems and that's where salt comes in.  We need to distribute authentication information for these systems somehow and this leads us back to the original problem, if that authentication information is stored unencrypted on a server somewhere and that server is compromised our entire system is blown wide open.

We can pre-seed systems with all of the necessary configuration, but that process is painful and time consuming.  We also spin up and shut down 1000's of servers/day which complicates the picture.

I see it a bit like inception.  How deep inside the security onion can you go?  As far as we need in order for our customers to be happy. ;) 

More practically, this comes from one of our security audits.  During the audit a server was compromised and the security team was able to access content from a competitor who also uses our system.

As I mentioned earlier, we don't have to store the data on shared infrastructure.  We can always physically isolate the hardware, but we're also trying to keep costs under control. 

Shawn Butts

unread,
Dec 3, 2013, 12:42:04 PM12/3/13
to salt-...@googlegroups.com
Though it's not a perfect solution and I haven't found any information about it's use with Salt, hiera-gpg seems to be used in the Puppet community to address some of these concerns. 

Bryan Murphy

unread,
Dec 3, 2013, 1:03:46 PM12/3/13
to salt-...@googlegroups.com
Thanks.  I'll take a look. If nothing else it could be informative.

I'm leaning towards cherry-picking the configuration, encrypting it, sticking it on a web server, and going master less.  We'll lose some of the flexibility of the master/minion architecture, but the change control will be easier to implement and the obvious security implications will be more easily addressed. 

This way we'll still get the benefits of the state/pillar/jinja2 model which I really like more than anything else.

Csaba Okrona

unread,
Dec 27, 2013, 2:25:17 PM12/27/13
to salt-...@googlegroups.com
Any updates on this?
I see this as a valid request.
By storing the pillars on gitfs we get really nice features, like versioning, jenkins integration for tests, code reviews, etc, but we expose sensitive data (e.g. Django secret hashes, internal api urls, etc) to another service (e.g. Github).
It'd be really nice to be able to pre-encrypt externally stored pillar/state data...

Luminous Salt

unread,
Dec 27, 2013, 5:22:45 PM12/27/13
to salt-...@googlegroups.com
You could probably write your own ext_pillar for this..

or use hiera-gpg with ext_pillar:
http://www.craigdunn.org/2011/10/secret-variables-in-puppet-with-hiera-and-gpg/

Matthew Williams

unread,
Dec 28, 2013, 12:07:24 PM12/28/13
to salt-...@googlegroups.com
I did some experimental work on two different ideas regarding encrypted data. At this point I do not plan to continue with the experiment, but feel free to use the following gists for ideas:

gpg encrypted grains:
https://gist.github.com/mgwilliams/c24865fb381d482c8855
https://gist.github.com/mgwilliams/22d88f51e4faf42452ac

rsa encrypted pillar:
https://gist.github.com/mgwilliams/1230869f7b114b5fcf37
https://gist.github.com/mgwilliams/5ea039eb05f2ef3bc32c


--
You received this message because you are subscribed to the Google Groups "Salt-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to salt-users+unsubscribe@googlegroups.com.

Corey Quinn

unread,
Dec 28, 2013, 12:22:15 PM12/28/13
to salt-...@googlegroups.com, salt-...@googlegroups.com


> On Dec 27, 2013, at 11:25 AM, Csaba Okrona <ochr...@ochronus.com> wrote:
>
> Any updates on this?
> I see this as a valid request.
> By storing the pillars on gitfs we get really nice features, like versioning, jenkins integration for tests, code reviews, etc, but we expose sensitive data (e.g. Django secret hashes, internal api urls, etc) to another service (e.g. Github).

Nonsense.

Speaking as a de facto "Best Practices" guy, you should already be storing pillar in git, in a separate repository from your state tree. If this is in GitHub and contains sensitive data, you've already crossed that rubicon.

Encrypting the data in pillar is potentially a good idea, but "it lives in GitHub unencrypted" isn't a compelling argument for doing so.

--Corey

Bryan Murphy

unread,
Jan 1, 2014, 11:16:53 PM1/1/14
to salt-...@googlegroups.com
Since I started this thread, I'll follow up with where we are now.

1. We ditched the salt-master.  It's too big of a liability.  Once it's compromised, everything is compromised.

2. We're moving towards pre-encrypting key configuration before we check it into source control.  This way, only a select few who have access to the keys can view/modify the raw configuration, but everybody who has access to the repository/ci server can make a build and deploy.

3. We tar the salt state+pillar data, aes encrypt it, rsa encrypt the aes key, and then upload everything to a web server.

4. Each environment is pre-configured with the private rsa key, has a script that downloads the latest configuration from the web server, decrypts, applies it locally, and then discards it.

5. We have a cron-job that runs every 5 minutes and checks for an updated configuration and applies it.

I'd like to have something more robust than the cron job, but this meets our needs for the moment.

Bryan

Bowen

unread,
Jan 3, 2014, 9:19:31 PM1/3/14
to salt-...@googlegroups.com
I'm thinking out loud here (always dangerous) but one solution could be to store your passwords in a password management system that has an API for retrieving passwords[1]. You could write a small script that gets called by ext_pillar to retrieve the passwords and populate them as pillar data so that they can be used in your state files.

In your state files you could then reference the passwords using:
{{ salt['pillar.get']('passwords:secretsquirrel', 'defaultpassword123') }}

This way, in your development environment (which doesn't have access to your password management server) the passwords will fall back to a default value which you can check-in to git and not care about so much (assuming your test environment isn't externally accessible or something equally as silly). When the code is used in production it will instead retrieve the passwords from your password management server, which should keep them all nice and safe. In this way, as least the passwords aren't stored in your git repo, and aren't saved directly on the saltmaster (in case someone manages to compromised the server and rsync the harddisk, though in this scenario you probably have a lot more to worry about).

When it comes to making passwords only accessible fromm the minions, and not available on the master, I'm stuck. Because the rendering of the state files is done on the master, the password is always going to need to be available to the master. Someone also pointed out previously that SaltStack always has root access to each of the minions anyway, so even if you used symmetric encryption with GPG the salt master could still just retrieve the private key from the minion. The only solution I can think of to this is what Bryan has done, by doing away with the master and executing the states locally (salt-call --local), but it really does take away the elegance and convenience of SaltStack, which is a great shame.

-
Bowen

[1] PMP - Password Manager Pro (by ManageEngine) is one production I know of that supports retrieving passwords via webservice calls.

Patrick

unread,
Oct 14, 2016, 11:49:08 AM10/14/16
to Salt-users, bmurp...@gmail.com
Not only that, but sequestering this functionality on the master makes 'salt-call --local' useless, unless you happen not to hit any encrypted secrets. We do use a master, but it's pretty important to me that --local isn't broken.
Reply all
Reply to author
Forward
0 new messages