How to automate unsealing of vault?

[Sean Bollin]

It seems that Vault seals each time it restarts. If several applications rely on Vault being unsealed to use, what is the best practice way to unseal vault incase of, for example, a system restart?

[Adam Greene]

Hi Sean,

'best practice' is to run vault under HA mode and have at least one, if not two spares up and running and already unsealed.

If you look back through this group you'll see a lot of discussion on this topic, and in most cases keeping the unsealing a manual process will be your best bet.  

I run vault on top of coreOS, which can auto-update and restart an instance, and thus seal vault.  What I ended up doing was to keep coreOS auto-updating on for all instances except the ones vault runs on, and for those I do rolling restarts with manual unsealing when needed.  I have a helper script that I keep stored on an encrypted thumb drive (in a vault), so when i get everything unsealed I just need to run one command rather than 3 with potential copy/paste issues.  But that is as far as I want to automate... otherwise I'll have to put keys somewhere, sure that, and it is turtles all the way down.

Cheers,

adam

[Jeff Mitchell]

"But that is as far as I want to automate... otherwise I'll have to
put keys somewhere, sure that, and it is turtles all the way down."

Hah, I love it.

I fully agree, but in a bit more detail: Vault is designed to be the
thing that keeps *everything else* from having to have hardcoded
secrets. But the flip side of that is that Vault needs manual
unsealing. If you hardcode Vault's unseal key somewhere, you've just
shifted the problem...then you have to figure out how to protect that,
and potentially automate *that* protection. As Adam said, turtles.

Security is always butting heads with convenience. You can automate
unsealing for convenience, but you'll be giving up a core part of
Vault's security, and I can't really recommend a particular way to go
about doing so. My suggestion is: let Vault be your one pain point,
and if need be, treat it as a special flower -- for instance, make it
the one thing you don't put in autoscaling groups, so that you can
safely put everything else in them.

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/5db0919b-ee40-41bf-95bb-33078582e934%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Sean Bollin]

Thanks Jeff, that really makes sense - saying "Security is always butting heads with convenience" gives me some relief, I don't feel like I'm doing it all wrong so much anymore! ;)

I'm curious - perhaps this is a naive question.. but how does sealing the vault on restart make it more secure?  There are long periods of time when one can leave the vault unsealed.. what makes it more secure to seal it just when the system reboots (vault restarts)?

[Elliot Murphy]

I think this makes the system a bit safer in the scenario when someone makes a copy of the hard drive and boots it somewhere else to try and extract secrets.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/2ccde2f7-4914-40e2-a2a1-9063b000c208%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Elliot Murphy, CEO KindlyOps.com  

Get advice on securing your cloud servers: Instantly book a consulting session

Watch my talk on safeguarding sensitive data: Span conf London 

[Jeff Mitchell]

On Fri, Dec 18, 2015 at 8:55 AM, Elliot Murphy <ell...@kindlyops.com> wrote:
> I think this makes the system a bit safer in the scenario when someone makes
> a copy of the hard drive and boots it somewhere else to try and extract
> secrets.

This is one such scenario. There are other possible scenarios, for
instance: an attacker is able to install (say, through incorrect
filesystem permissions) a modified kernel that gives them elevated
privileges on the system, and reboots the system to get the kernel up
and running. If they have root access to the system with an unsealed
Vault, they can dump memory to try to find secrets.

Generally speaking, if your machine reboots and you didn't just tell
it to, something is seriously wrong. Most of the time it's likely a
hardware failure or maybe a kernel bug, but it could be something
else. Autoscaling groups throw a bit of a curveball, but as Vault
doesn't do active-active there's really no reason to put Vault in an
autoscaling group in the first place.

Hope that helps!
--Jeff

[Sean Bollin]

Awesome, great.  We are planning to move forward with Vault for the encryption keys and we'll also be tying it in to our ansible secrets.. thanks for the quick responses this is definitely a plus in helping us choose a tech.  I'll keep ya posted with how it turns out

[Christopher Campanale]

Not sure if it would be any help, but this is how I ended up doing our implementation: https://gist.github.com/ccampanale/e230ed3e96a978776404

You would need to change a few things around of course but essentially I've placed our unseal keys in Vault as secrets and set this script up in Cron. So long as there is at least one unseal instance in our environment any sealed instances *should* unseal themselves (so long as they are properly configured and the correct token is supplied to the host). Since writing this script I've also written a way for our consul cluster nodes to check themselves to see if they are the leader and register/deregister a service with consul for `vault-leader`. This is especially helpful if you are using SSL for your vault instances and the cluster can't pass requests around. I'll probably throw that on GH soon.

Hope this helps!

[Janos Matyas]

We (auto) unseal it using KMS - https://banzaicloud.com/blog/vault-unsealing/

[Jason Martin]

Automating unsealing of Vault tends to subvert one of the major points
of its use, which is to enforce that no one person is able to
abscond with all the secrets.

In terms of HA, you can run multiple Vault instances in a
cluster (on multiple nodes / multiple cloud availability zones)
to increase the likelihood of at least one instance being
available and unsealed. This assumes you are using a HA-capeable
backend.

-Jason Martin

> --
> This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/4104d3c6-8db1-4496-91c0-0e4dde21c8dc%40googlegroups.com.