S3 Bucket backend and S3 Bucket Versioning

[Timothy Kimball]

Hello -

Thanks again for your great tool. 

Digging into policy, we are also looking at disaster recovery.

Can we use the build in S3 bucket versioning to do a rollback on the vault?

Is the following example possible?

store value1

store value2

roll back the s3 bucket to a previous version

read value1

We want to do this in case a user accidentally deletes or over-writes a critical asset. 

We are planning on only storing these key credentials within Vault. This of course makes us nervous. Hence our question :D

Thanks,

Tim

[Jeff Mitchell]

Hi Tim,

Broadly speaking this should be fine -- the way to backup Vault is to
back up your underlying storage. There are some potential gotchas that
you should be aware of:

1) You can't do this online. You'll need to shut down Vault, do the
rollback, and start it up again.

2) If your backup has old versions of unseal keys (you've rekeyed
since) you'll need those old unseal keys or you won't be able to
unseal

3) Revocations that have occurred since will not appear to have been
processed. This can lead to unexpected consequences, because if
attempting to revoke such a lease leads to failure (e.g. the
third-party database you're trying to remove a user from), Vault will
keep attempting to perform this revocation. In most cases, this is
probably not an issue, but if S3 versioning supports it, my
recommendation would generally be to do the following:

store value 1
store value 2
stop Vault
note the current version (B)
roll back S3 to version A
start Vault
read value 1
stop Vault
roll back to version B
start Vault
store value 1

From a quick glance at S3 versioning it looks like it's per-bucket.
Depending on which backend you're using, you might be able to do
something a bit more surgical. For instance, if you don't care about
the current version of the value, you could stop Vault, look through
S3 to find the path (the paths are in plain text, although the mount
has a UUID, so you'll have to look to see which mount it is), then
restore only that one object, then start Vault.

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/c99e6da7-a1d7-4bd2-9182-77dc994ef5e7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Timothy Kimball]

Jeff -

Thanks for the quick reply. I like that we can surgically rollback a particular value change. That certainly helps.

In terms of doing regular backups - it seems that your proposal of needing to seal the vault means that we can not do this automatically - correct?

Thanks,

Tim

[Jeff Mitchell]

Hi Tim,

For backups, it usually depends on the underlying physical store -- as
long as it's consistent it shouldn't matter (and even if it isn't, it
*probably* doesn't matter). I don't know S3's behavior here, but for a
store that was on a filesystem, something like a ZFS snapshot would be
pretty good. At HC internally we do live snapshots of our Consul
backend.

The reason I said you'd want to stop and start Vault when restoring
(note, not seal -- stop) is because Vault has caches internally for
the physical store. This provides a huge speedup but does mean
underlying changes won't be noticed, and could be overwritten.

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/99bf44c1-4bfc-4180-a8d3-a1de0fc2c069%40googlegroups.com.