Can Vault clusters be orchestrated to use the same multi-region storage backend?

[Laurentius P]

Hello all,

I was wondering if Vault can be orchestrated to use the same multi-region storage backend and only one region of vault is active at a time.

Thank you,

Laurent

[Calvin Leung Huang]

Hi Laurent,

A single Vault cluster  (which can be composed of 1 or many Vault nodes) shares the same underlying storage, and it's common that the storage lives within the same region. However, you could have Vault clusters set up for each of those regions, each with their own distinct storage, and use performance replication to replicate configuration, secret methods, and auth engines across those clusters. There would be a primary cluster that can process reads and writes, and secondary clusters that can process reads and forwards (non-local) writes to the primary.

Regards,

Calvin

[Laurentius P]

Hi Calvin,

Thanks for your response. The performance replication is only available for Vault Enterprise; Is there anything similar for OSS? What about consul-replicate?

Thank you,

Laurent

[Calvin Leung Huang]

That's correct, performance replication is an Enterprise feature. Using consul-replicate for Vault data store replication is not a supported mode of operation, and will most likely fail to safely replicate data in many scenarios.

Regards,

Calvin

[Eric Horst]

Laurent, this is possible using Google Cloud. The Google Cloud Storage
Storage Backend supports HA and the underlying GCS storage can be
configured to be multi-regional. For production, use Terraform to
bring up a HA Vault cluster in one region using this backend and
multi-regional storage. KMS keyrings are also multi-regional. Fail
over to another region by turning off the cluster in the primary
region and deploying a new cluster in a different region using the
same config, storage bucket and KMS keyring for unseal. With this you
can be running a HA cluster in one region, say us-west1 and easily
redeploy a cluster of the same configuration in us-east1 with the same
data. There is downtime in this failover but the Vault data is
preserved and it comes up working exactly the same way. We're doing
this with Vault in a GKE cluster but it could just as easily be done
with standalone GCE VMs.

-Eric

> --
> This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/67376699-aeda-4df0-9d5e-7232fbea63fb%40googlegroups.com.

[Laurentius P]

This is great. Thank you Erick.

So, basically the backend should be GCS instead of Consul, since GCS can be configured to be multi-regional. Is that correct statement?

Regarding fail over, is this manual process or can be automated?

I'm going to draw a diagram from your response, hopefully, I can share it.

Thank you,

Laurent

On Wednesday, August 28, 2019 at 12:16:13 PM UTC-4, Eric Horst wrote:

Laurent, this is possible using Google Cloud. The Google Cloud Storage
Storage Backend supports HA and the underlying GCS storage can be
configured to be multi-regional. For production, use Terraform to
bring up a HA Vault cluster in one region using this backend and
multi-regional storage. KMS keyrings are also multi-regional.  Fail
over to another region by turning off the cluster in the primary
region and deploying a new cluster in a different region using the
same config, storage bucket and KMS keyring for unseal. With this you
can be running a HA cluster in one region, say us-west1 and easily
redeploy a cluster of the same configuration in us-east1 with the same
data. There is downtime in this failover but the Vault data is
preserved and it comes up working exactly the same way. We're doing
this with Vault in a GKE cluster but it could just as easily be done
with standalone GCE VMs.

-Eric

On Tue, Aug 27, 2019 at 6:20 PM Laurentius P <laurent...@gmail.com> wrote:
>
> Hello all,
>
> I was wondering if Vault can be orchestrated to use the same multi-region storage backend and only one region of vault is active at a time.
>
> Thank you,
> Laurent
>
> --
> This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups "Vault" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Eric Horst]

On Wed, Aug 28, 2019 at 11:32 AM Laurentius P
<laurenti...@gmail.com> wrote:
> So, basically the backend should be GCS instead of Consul, since GCS can be configured to be multi-regional. Is that correct statement?

Yes, storage backend GCS with a multi-regional storage bucket.

>
> Regarding fail over, is this manual process or can be automated?

I have only done manual since it is intended for disaster recovery
only. However, we do use Terraform so the Vault deployment is fully
defined in Terraform. The manual failover process is to shut down the
existing region, point the Terraform to a new region and rerun
Terraform to deploy the new infrastructure. If you needed more
automation I'm sure it could be done. The important part is that the
original cluster needs to be shut down and not touching the backend
storage before bringing up the new.

-Eric

> To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/7ea6a5aa-be86-4c3f-a43d-f82cdefc8af6%40googlegroups.com.