Vault Backups on GCS and Consistency Concerns

[jo...@cockroachlabs.com]

Hello,

We have been exploring options for taking backups with Vault. The backup story for Vault seems to be highly dependent on the backend being used. We are currently using GCS, but there doesn't seem to be any way to take a point-in-time snapshot of the bucket that is easy to restore to. Is there any guidance on managing backups using the GCP backend? If not, we would be willing to switch backends if one provided a better backup and restore experience.

One of our big concerns before making such a switch is that, according to https://groups.google.com/d/msg/vault-tool/GDhj-KVqtHk/ckYWt6UdDAAJ, even if we used a backend that provided an atomic, point-in-time snapshot (such as Consul), it would not be enough to guarantee that the snapshot was consistent. Is there any way, using any of the backends that support high-availability, to take a snapshot that will guarantee consistency without taking Vault offline for the duration?

Thanks,

-Joel

[Pete Bohman]

I am also confused by possible consistency issues after restoring from snapshots. From the high-level arch doc, it sounds like the consistency issues would be resolved by reconciling the WAL log after restoring from a snapshot,  "Additionally, Vault handles certain partial failure cases by using write ahead logging with a rollback manager. This is managed transparently within the core and is not user visible." - https://www.vaultproject.io/docs/internals/architecture.html