Dealing with secrets in configuration management

[Nick Silkey]

Many of us want our infrastructure as code to be portable and usable at will. However management tends to not like secrets (hot data like passwords) walking around.

Thoughts on how to best mitigate? Dueling vcs sources? Encrypted Chef data bags? Locked down CFEngine cfservd/cf-serverd?

Curious. Thanks.

NB: "No secrets in configuration management" is not what I'm after.

[John Vincent]

Perspective: You have tools that are running with root privileges and
can change any aspect of your system at will. Obsessing over the
method for storing your encryption keys is sort of a minor concern ;)

Having said that, the encryption issue is a "painful" one in general.
Not being self-serving here but I actually went through the exercise:

http://lusislog.blogspot.com/2010/12/chef-and-encrypted-data-bags.html

You can go as deep or shallow as you want but at some point, you've
got to have a key to decrypt the data. There's no easy answer. I'm
pretty happy with PSKs stored in a queryable endpoint. It's more
obfuscation than anything but it's better than storing a flatfile
around on disk.

--
John E. Vincent
http://about.me/lusis

[Peco Karayanev]

I have used keepass in the past, and got to a point where I explored programmatic access (they have APIs to the DB) but have not pulled the trigger. I am certain people would have other key stores/chains to recommend.

 

Cheers 

[James White]

I use LDAP, the deployment system creates a "short-time-pad" which it
inserts into the host's simpleSecurityObject's userPassword attribute
(and ships to the host) at deployment time. This "short-time pad"
allows the host to create a private key, and a CSR log into our
LDAP-backed PKI infrastructure and get it's CSR (and only it's CSR)
signed by it. It then rotates out it's LDAP password with a random one
only the host knows, and publishes the cert (and thus the public key)
back to LDAP.

What you're left with is a private key and userPassword that have
never left the host, and are completely unseen by human eyes, and a
public key that is query-able by anyone or any thing that can grok
LDAP.

So if you need to ship/store something on the host, just wrap it with
the public key and only that host's private key can decrypt it.

In the event of a compromise, revoke, redeploy which re-keys (and
re-issues) certs without human intervention.

Of course if your host get's rooted then any password (encrypted or
otherwise) should be treated as compromised; But there are some things
you just need to keep on the host because that's how the applications
pull them in.

We use these same certs for anything from SSL to IPSec (phase 1). and
they can be re-issued on a whim (90-days, 30 days, daily? no problem)
because you can just have the host log back into the LDAP
authenticated PKI infrastructure, and revoke, re-issue.

The same setup works for users for two-factor authentication. LDAP
doesn't care if you're meat or metal, just that your hashes match.

[Pat O'Brien]

We ended up writing a Chef library which can be called during attribute loading, which decrypts one of two JSON files (depending on whether it's prod or not-prod) and throws the k,v pair into a hash which the cookbooks can reference. There are two separate GPG keys which are deployed to the hosts, one for prod and one for not-prod, which the library uses to do the decryption. We also have a LWRP which does the same, although for whole files rather than k, v pairs.

All of our developers can encrypt and decrypt the not-prod files via a webui (where we also give them the ability to do dry and diff runs against the cookbooks at various revisions) to test out the cookbooks, but when it's time to go to prod only admins and a few "sous-chefs" have access to the prod keys, decryption and encryption.

It works well for us and beats the alternative that was used here, which was to have "secure" repositories, which had passwords in plain text, although was locked down via apache so only admins had access. 

-pat

[Luke Kanies]

With Puppet, this is done largely by giving servers trusted access to this secret data, but giving clients little to no access.  This is one of the major benefits to doing your compiling on the server (with the downside being that your servers need to be beefier).  You have to trust your Puppet servers, but you never trust your clients.

E.g., if you have a password database that provides access to all of your backend databases and you want to restrict access so each web service can access only its database but no others, you write a function that retrieves the appropriate password for each web server.  This function runs on the Puppet compile server, which has access to the entire database (this host is trusted anyway, because it effectively has root on all of your machines), and the resulting compiled catalog only has a single password in it, not the entire database.

Note that this is the default Puppet mode - the server has complete access, and the clients have the least privilege possible[1] - so in general it doesn't take any extra work on your part to end up with this security.  This works well in places where you want clients in the DMZ but you don't want to trust anything from the DMZ.  You can actually sneakernet your configurations into untrusted locales, if you need to be so extreme.

There's another security benefit of this model, in that you rarely if ever trust anything the clients say about themselves. You use things like the client's operating system to build its configuration, but one client can't affect another client's configuration (unless you build a lot of complicated stuff to make this specifically possible), and the default mode 

1 - http://en.wikipedia.org/wiki/Principle_of_least_privilege

-- 

Luke Kanies | http://about.me/lak | http://puppetlabs.com/ | +1-615-594-8199

[Noah Campbell]

James…this is very cool. I think if you put together a more in depth write up of this technique, many folks would be interested.

Regarding the OP, if folks are concerned about passwords laying around in a flat file, then I'd wager it's a legacy gut reaction. These folks accustom a service password being created and then baked into the infrastructure, never to be changed in fear of causing some remote system to stop working.

The right mindset, regardless of tools, is to consider rotating passwords every 3 months. Just this will cause a whole cascade of changes to your infrastructure so you can push passwords around or remove passwords. Centralized services removes the need to "move files" like the ldap system James talks about.

I've seen puppet push passwords, for better and worse. The key element is that any breach is only one cron cycle away from being closed. I don't think keeping passwords in SCM is a horrible idea as long as it's adequately guarded and you can quickly respond and change passwords, restart services and continue on.

What does adequately guarded mean, you ask? That depends on your environment. Highly secure environments will have many layers of defense to protect critical systems and domain isolation. For example, one business unit uses their own set of credentials and it isn't shared with other business units. One business unit may have fine grained access control at different layers while anther may just have a single root user/password combo for everything. It just depends on what they're securing and the value to the business.

I would also suggest that you rely on existing security controls (shadow passwords, ssh public keys, 2-way ssl) to transfer trust throughout your system.

-Noah

Noah Campbell
415-513-3545
noahca...@gmail.com

[Johan Andersson]

Interesting thoughts.
Its funny; I dealt with a similar issue at work just a couple of hours ago.

Our environment does not involve any tools like chef or puppet. However, we have a git repo containing data you would find in chef.
Also worth mentioning is that we are windows 2008 server (workgroup mode) only. All admin work is done using powershell/winrm and traditional SMB file sharing.
Some automation is performed using a Jenkins CI instance.
This leaves us with passwords in plain text in scripts run by Jenkins.

We are managing some passwords using tools like KeePass but those are no use in scripts.

So today I chosed to go for the master key path using PKCS. I put together a simple PowerShell module which lets you create file based "key rings" which therefore easily can be managed in our git repo. All that is needed to add or retrieve a set of credentials is that you have the certificate in cert:\CurrentUser\My.
The goal is to separate the encrypted passwords and the master key. That means the content in the config repo is just jibberish unless the certificate is available for the current user. Distribution of the master key to config servers and ops is no harder than sending the PFX file or storing it on a USB key..

Retrieving a key from a store (username/password pair) is simple as the builtin get-credential cmdlet, however credentials are identified by hostname+service.

GetCredential "fqdn" "service"

The directory (key store) does not reveal the fqdn nor service name.

This tool is about 100 rows of PS and have no dependencies. If someone has further interest in this I will make it public at github.

- Johan

Senior dev at RemoteX

[Peco Karayanev]

The catch to all this is that the "secrets" need to be admin(human) accessible. If people start keeping their own pass repo-s there is something broken. Also I would prefer the password/key repo to be super simple, I don't want to be stuck waiting on server to come up to get my accounts and passwords to troubleshoot the server that just broke. 

[Johan Andersson]

I agree.
I really like James set up using LDAP but in our case we're on Windows and use independent servers in workgroup mode which means we only have local user accounts involved.
As an alternative to the current solution we've looked into different SSH servers to be able to use the same auth mechanism on both *nix and windows. However the available solutions added to much license cost and did not integrate with Powershell very well.
Stuck with SMB file sharing we cannot use key auth..

- Johan

[Jeremy Brinkley]

We use Puppet and built on this idea for our secrets management.

The develop-modify cycle/workflow for configuration management is different than secret attributes, so we need to accommodate this (for example, Puppet configurations are under source code control and are worked on by people who should not have knowledge of secrets like client credentials). We have an out-of-band method of distributing secret attributes to Puppetmasters (and only the Puppetmasters), and a custom function in Puppet which fetches the secret from this store. All the Puppet manifests and configuration file templates use this function.

In the future, if we make different security decisions about how the secrets are stored or managed, the templates' and manifests' interface to the system will still just be the same function with a different implementation. Similarly, access controls (servers for product A are not permitted to retrieve secrets for product B) would be enforced behind this interface

--

   Jeremy Brinkley

   Technical Infrastructure Designer

   Proofpoint, Inc.

[Joe Miller]

I don't think I saw `hiera-gpg` mentioned for puppet. I have not used it, but it looks useful: https://github.com/crayfishx/hiera-gpg

[John Vincent]

Luke brings up some awesome points about the benefit of how Puppet's
server-side model does it. The only thing I would bring up is that you
STILL have "sensitive" information on the client. You'll never get
away from that sadly.

At a previous company, they actually implemented a really complex
hierarchical signing process of delegates. The topmost key was limited
to access by 3 people with, I think two more keys in the hierarchy.

With java apps you can do stuff like JNDI to keep sensitive
information outside of application code bases. In the end, though, the
sensitive information HAS to be accessible to the system that needs it
and the trick is really to minimize how much damage one compromised
system can cause. This is almost a no brainer with a CM tool.

[Noah Campbell]

I think Kerberos was invent to limit the amount of information on the disk at any given time. Unfortunately the complexity of these configurations and protocols ate typically overkill for the standard business.

I found that a simple split of inside VPN outside VPN made for easy division of secret management policy. Passwords were basically everywhere and folks were basically trusted to do the right thing. If they were not trusted their login was revoked. There was also limits that a particular business unit could could do to other business units.

This setup definitely favored rapid development and release. Not for everyone, but something worth considering.

Sent from my iPhone

[James White]

I feel it's important to note that we *do* use puppet to fire off the
scripts that do the key-generation and signing request submission.
Most of the work is just setting up the pkild server
(https://github.com/fapestniegd/pkild) that does the signing, then
it's just a matter of having puppet generate the key, the csr, and do
a HTTPS POST.

There is a complete rewrite of pkild started to make pkild
decentralized (the one on github is the proof-of-concept) and should
be treated as PoC code. It actually started as a bash script that was
fired off by cfengine2 to generate the CA-tree, then converted to perl
(catalyst) to get some basic web functionality going, and then the
Websages crew converted our core network from cfengine2 to puppet, and
now we're trying to standardize our applications in ruby.

So it's messy, and the documentation is crap when it's not completely
absent. But the code works.