Pros and cons for automated vault unseal

[Ho-Sheng Hsiao]

Hi folks,

My team is looking at deploying Vault. One of the features I understand about vault is the ability to seal the vault that requires a quorum of keys to unseal. My understanding is that, when implemented this way, no one single individual have full control over the encrypted secrets. When backed by a team process that backs this, it can help mitigate certain security scenarios.

However, in practice, it seems as if people look for ways to work around it:

- I remember reading a Hacker News comment about the motivation behind writing an automated unseal script for an HA deployment of Vault. That having to unseal the vault (through reboots, maintenance, etc.) happens often enough that the operator wrote a script to unseal them all at once.

- Google Cloud's intro tutorial shows how unseal keys could be stored in the GCP KMS, and how to manually unseal things. For the purpose of the tutorial, all keys are stored in the KMS, all of them available to the same operator.

- Vault's own webpage shows an enterprise feature for automatically unsealing Vault through AWS KMS and GCP AWS.

I'm trying to figure out:

(1) Why does it seem that in production deployment of Vault, operators seem to get around the key quoram. At that point, wouldn't it be better just to have a single unseal key?

(2) What is the advantage of storing the key in one of the cloud KMS? I know AWS's documentation mentions something about hardware support. I suppose if a human operator cannot get direct access to the key, and the Vault gets automatically and regularly rekeyed, it would be harder to steal the unseal keys.

Am I misunderstanding something about Vault?

Ho-Sheng Hsiao
Legal.io

[Jeff Mitchell]

Hi Ho-Sheng,

On Mon, Jun 11, 2018 at 6:13 PM Ho-Sheng Hsiao <hoshen...@legal.io> wrote:

However, in practice, it seems as if people look for ways to work around it:

Some do, but don't make the mistake of assuming that because a some peope have posted publicly about working around it that it's general practice. Most customers we talk to have very strict procedures around unseal keys because they understand that it's a core part of using Vault as a root of trust.

Usually people that work around it are startups/small shops with a single operator or small operator team where they are concerned about availability, e.g. if there are 10 people in the company and the threshold is 3, that's a pretty large percentage of the people at the company that need to be available for any unseal operation.

That's just to say that many startups and small shops treat security as an impediment to getting things going, because security is hard and time consuming, and are more willing to deal with it later. But not all -- even when HashiCorp was 20 people, the co-founders could not unseal our Vault instance alone.

 

- I remember reading a Hacker News comment about the motivation behind writing an automated unseal script for an HA deployment of Vault. That having to unseal the vault (through reboots, maintenance, etc.) happens often enough that the operator wrote a script to unseal them all at once.

Depends on a lot of things. I recall one conversation with someone who wanted to run Vault in an auto-scaling group because company policy was that all applications -- even stateful ones (I assume their goal was that all applications should be stateless) -- had to run in auto-scaling groups, so they were concerned about Vault instances starting up and having to be unsealed and nobody noticing, while already-unsealed ones were taken down. Their solution, IIRC, was to script an unseal with hardcoded keys sitting in the script, which basically defeated the security model entirely. They would have been far better off simply understanding that Vault isn't stateless, is an important part of their infra as a root of trust, and should be an exception to the "everything must be on an ASG" mentality.

In normal operation, it doesn't seem to be a problem for most people. They upgrade Vault when they want or at normal upgrade cycles, and just make sure a quorum of unseal key holders are around at that time. Often in this scenario you'll have a high number of shares with lower threshold, like 15:3, so that each operator has a share, five are always available at any given time of the day, so you're always assured to have quorum, with some buffer left over.

 

- Google Cloud's intro tutorial shows how unseal keys could be stored in the GCP KMS, and how to manually unseal things. For the purpose of the tutorial, all keys are stored in the KMS, all of them available to the same operator.

Ideally you would use GCP policies to never make those same keys available to human operators, with the caveat that a high enough level operator could overcome those restrictions (and hopefully this action would be monitored/audit logged). I don't know enough about GCP's policies but I know that in AWS for instance, you can control which users have access to which KMS keys and which S3 buckets.

- Vault's own webpage shows an enterprise feature for automatically unsealing Vault through AWS KMS and GCP AWS.

Yes, and policies should ensure that the storage/keys used for this are not also available to human operators, but only available to the instances running Vault.

 

I'm trying to figure out:

(1) Why does it seem that in production deployment of Vault, operators seem to get around the key quoram. At that point, wouldn't it be better just to have a single unseal key?

I don't think this is as common as it seems, it's just that the large majority running things the way we suggest they run don't feel the need to blog/comment about the fact that they're running things normally :-)  . So on the net and in comments you're mostly going to see the people that worked around it talk about how they worked around it. And yes -- if you are giving all operators all the keys, you might as well just set your shares/threshold to 1.

 

(2) What is the advantage of storing the key in one of the cloud KMS? I know AWS's documentation mentions something about hardware support. I suppose if a human operator cannot get direct access to the key, and the Vault gets automatically and regularly rekeyed, it would be harder to steal the unseal keys.

 

Right -- use your cloud's access policies to ensure that only the Vault instances get access to the keys, and ideally set up alerting to watch for any change (what changes depends on the capabilities of the cloud beyond my ken, so I can't speak to that at any real level of granularity).

Another part of this is Seal Wrap functionality -- HSMs or cloud KMS systems that are FIPS-certified can not only allow for auto-unseal but also be used to wrap critical security parameters within Vault by FIPS-certified encryption, allowing Vault to satsify some FIPS KeyStorage/KeyTransport requirements.

Hope that helps!

Best,

Jeff

[Ho-Sheng Hsiao]

Thanks! That has been very helpful and answers a lot of questions I have. I also have some leads for further reading. 

-Hosh