Vault Multi-Datacenter Explained?

[John Anderson]

So, I think I'm fundamentally misunderstanding Vault/Consul's HA & Multi-Datacenter models.  I've read the documentation on these subject several times, and I'm just not seeing the proper path forward for having multiple datacenters, with vault in an HA configuration within each datacenter, and having a single source of truth.

What I have are two different datacenters, "NLD" & "PHX".    Within these datacenters, I have two vault servers each "vault-0{1,2}-{nld,phx}".    I also have 3 consul servers in each datacenter, "consul-0{0-3}-{nld,phx}".    The vault servers run consul in agent mode, and vault on each server connects to localhost (the consul agent).    Each consul in "agent" mode, connects back to the consul servers in the local datacenter.   So, for instance, a vault service in NLD would connect to consul on localhost, which would then query one of the NLD consul servers.


So with things setup this way, I initialized and unsealed vault on one of the vault PHX nodes.   The other vault PHX node can see that vault is initialized, and it is in standby state.   Neither of the vault servers on the NLD node can see that vault is initialized.    Is this the intended functionality?     From looking at the documentation, the "federation" pages say that data which is not present on the consul instance will be RPC'd back to the datacenter in the WAN group that has the data.   This was how I thought multi-datacenter worked in Vault.   Is this not the case?

Is there any scenario in which the consul agents on the vault nodes connect only to the local consul servers, and do not go across the wan, but they share the same vault storage information?  (i.e. information under the /vault kv is available in both NLD & PHX, and I only have to register secrets in one location or other, and not both)?

In other words, how is multi-datacenter configuration supposed to look/work, and under said configuration, would I have two completely separate vault data sets?    

(If there were a way to pass ?dc=phx to all queries invoked by vault in NLD, it might work, provided consul cached the cross-datacenter results).

John A.

[Armon Dadgar]

Hey John,

Let me help clarify, since the two systems are a bit different.

Consul silos data by datacenter. Within each datacenter, the data is replicated between all the servers. There is a single active leader, the rest are replicating data and potentially servicing read requests. Multiple datacenters can federate together, which means they know about each other and can route requests between datacenters. Consul does not actively replicate data between datacenters. This is a design decision so that a failure of an entire DC does not affect the other Consul datacenters. There is no coordination or conflict resolution required between data centers since writes are local to each DC.

Vault depends on storage backends to provide shared storage and coordination. The coordination features allow Vault to support an active/standby model and to failover automatically. When Consul is used, Vault acquires a lock and associates a health check to release the lock on process or machine failure. The shared storage allows Vault to failover to a standby without dealing with data replication, since they are sharing the same backend. Vault uses a write-through cache for performance, so within a local cluster all reads/writes go to the active instance to avoid consistency issues.

Bringing this all together, in your setup “NLD” and “PHX” are one Consul cluster that is federated together. However, each DC has a separate Key/Value store which is local to that datacenter. From the perspective of Vault, when PHX initializes and acquires a lock that is scoped to the K/V dat of PHX DC. The NLD Vault is totally independent and separately initializes and acquires a lock in the NLD DC. This is expected, and you will have two separate Vault data sets.

With Vault 0.7 Enterprise, we added support for multi-datacenter replication. This is a feature of Vault, and doesn’t depend on Consul doing anything differently. This allows a single Vault cluster to be the source of truth, and to replicate to any number of secondary clusters. For more information on replication, see the docs here: https://www.vaultproject.io/docs/vault-enterprise/replication/index.html

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/51a65cb4-978d-41ee-8085-fe6e7b91965a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Anderson]

This helps immensely, and really cleared some things up for me.

Thanks for the info!