Vault Enterprise Across-DC Replication/Sync Questions

[David Li]

Hi,

I am investigating a solution with a cluster of Vault servers (w/ Consul backends) as part of key management system. We have 2 DCs and each will have a cluster of Vault servers backed by 2 HSMs for unsealing purpose. 

After reading through https://www.vaultproject.io/docs/internals/replication.html, I still have questions about what's required to make it happen. I assume all my user secrets are encrypted by the Vault's encryption key and stored in Consul backends. For the 2 DCs to have the same encrypted secrets, does it require the 2 Vault clusters have to have the same encryption key and same master key has to be provisioned in the 2 HSMs?  This isn't mentioned in the documentation so I thought I might have misunderstood something. 

David

[Calvin Leung Huang]

David,

It is possible for each replication cluster to have an HSM, and the HSMs do no need the same encryption key. When you replicate from primary to secondary, the master key will be bootstrapped, and each cluster will encrypt it with its own encryption key from the HSM.

We have 2 DCs and each will have a cluster of Vault servers backed by 2 HSMs for unsealing purpose.

Do you mean that each DC will have a single cluster, or that each DC will have Shamir and HSM-specific clusters? A single Vault cluster has to be either fully using Shamir or HSM for all its server nodes (a.k.a HA setup). 

- Calvin

[David Li]

Hi Calvin,

We have 2 DCs and each has a single cluster of Vault servers backed by
a single HSM. The HSM would be used to unseal the Vault servers in its
own cluster.

Maybe I am still not quite clear about how inter-DC replication
happens. Say for example in the DC1 we have 2 Vault servers - V11 and
V12, user secrets are encrypted by the in-memory KEK11 of the Vault
servers. Here I assume KEK11 is the same on both V11 and V12 as they
have the same master key MK1 from the HSM1.

Now in DC2, we have Vault servers V21 and V22 backed by HSM2 using a
different master key MK2 and no user secrets. Can you explain how user
secrets in DC1 can be replicated to DC2 if MK1 is different from MK2?


Thanks.
David

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "Vault" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/vault-tool/BKzk2VYTEPA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/fb2e1c73-48a1-41b0-ba93-d411f2a15b7e%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Calvin Leung Huang]

David,

The master key MK1 is encrypted by KEK1 on DC1. When you enable replication, that same MK1 gets sent over to DC2, and it's encrypted by KEK2 from the HSM in DC2. User secrets are encrypted by the master key, not KEK1 or KEK2. That's why each cluster can have its own encryption key from the HSM while still keeping secrets accessible. One thing to note is that when you enable replication on the secondary cluster (e.g. DC2), all existing data will be wiped, and will be populated with data replicated from DC1.

- Calvin

[David Li]

Hi Calvin,

The replication makes sense now based on your feedback. But I have a
few more questions:

1. I am surprised to know that KEK is used to encrypt the MK not the
user secrets. This doesn't seem to be consistent with the docs:
https://www.vaultproject.io/docs/internals/architecture.html.

Is there a difference between open source version and enterprise version?

2. From what you described, it seems that I can achieve the same
results by simply provisioning the same MK on both HSMs and disable
the replication feature. Instead I can let Consul backends to sync
cross DCs for user secrets. Will this work?

Thanks.
David



On Tue, Oct 17, 2017 at 12:35 PM, Calvin Leung Huang

> https://groups.google.com/d/msgid/vault-tool/d2a56b39-827f-4f67-846e-290666f69e31%40googlegroups.com.

[Jeff Mitchell]

Hi David,

1. Vault's keyring encrypts user secrets. That keyring is encrypted
with the MK. The MK is encrypted with the HSM. So the MK and HSM's key
function as a two-layer KEK setup. The only difference here between
the open-source and enterprise versions is the ability to add the HSM
as that third key. (In either version, you can use Shamir to split the
MK instead, which is Vault's default.)

2. We do not support external replication methods.

Best,
Jeff

> You received this message because you are subscribed to the Google Groups "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAEuTsAy6qiktT61aVPgQhGH%2BEpzgKCV-E-%2Bf_agt18RBhQMozg%40mail.gmail.com.

[David Li]

Hi Jeff,

When replication starts from DC1 (with HSM1, MK1 and KEK1 as kering)
to DC2, would both MK1 and KEK1 be sent over to DC2?

Another related question is how DC2 would be unsealed by HSM2 if it
uses MK1 from the replication process? Would DC2 be unsealed during
the replication process? How would HSM2 be used?

Thanks.
David

> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GH7_DaGkjwjztUTowtojhFL0-%3DRUTGJiYS6TMjbdhPQxg%40mail.gmail.com.

[Jeff Mitchell]

Hi David,

> When replication starts from DC1 (with HSM1, MK1 and KEK1 as kering)
> to DC2, would both MK1 and KEK1 be sent over to DC2?

MK1 yes, KEK1, no. KEK1 stays inside HSM1.

> Another related question is how DC2 would be unsealed by HSM2 if it
> uses MK1 from the replication process? Would DC2 be unsealed during
> the replication process? How would HSM2 be used?

The master key is always the same. It's the KEK that is different,
based on the HSM in that datacenter.

Best,
Jeff

[David Li]

Hi Jeff,

I got it. So this means (at least theoretically) only one HSM is needed to unseal the Vault in that DC to provide the MK, the Vault server in the second DC would get it's MK from the first one via replication. Is that correct?

David