Multiple vault instances with replicated postgres backend

[jbd...@gmail.com]

Currently I'm running a single vault cluster with postgresql backend and life is good. I'd like to add another vault instance, because you know, more is better. I understand that Postgres is not a supported HA mode, but I was just curious if on the good/bad scale of ideas what y'all thought about running two vault instances running pointing to the same postgres server (or different servers that are replicated).

At this point is it just easier to switch to consul as the backend?

Thanks,

Josh

[Vishal Nayak]

Hi jbdatko,

Pointing two Vault servers to the same postgres storage will not work.
There are chances that you might corrupt your working setup. If you
replicate your storage and bring up independent Vault servers on each
of them, it works to start with, but from that point on, each of these
servers will act independently and there will be no synchronization
between these servers in terms of secrets and leases.

We don't officially support migration between the storage backends.
You could bring up a new Vault cluster on Consul, read whatever
secrets you could using the API from the old server on postgres and
populate the new one. Note that, this way, only secrets that are
accessible via the API can be migrated, not the leases.

Or, if you need to retain the *entire* state of Vault and migrate
over, you'll need to work up a script to copy over the postgres
storage into Consul's with necessary translations as required,
retaining the file system hierarchy.

Hope this helps!

Regards,
Vishal

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/0c3525cf-8522-43e2-b28b-d83769f71db7%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
vn

[James]

Vishal,

Thanks for shedding light on this.

Would it work if the two didn't write any secrets to the single backend? 

If the answer is backend specific, how would "readonly vault servers" work with the s3, dynamodb and this (postgres) for backends? 

[Jeff Mitchell]

Hi James,

There are two separate questions here: HA and replication.

Someone recently proposed writing an HA capability on Postgres so that may appear at some point.

Replication is a whole nother ball of wax, and everything Vishal said is true. It may seem to work until things really don't and you can end up making the data store inconsistent even if you think you're only reading. There's a reason horizontal scaling is a part of the paid version; distributed data synch is a difficult thing to do properly!

Best,

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/f319b8a7-6b87-4803-a024-eec37405277d%40googlegroups.com.

[James]

Jeff,

Thanks for the reply. I understand that replication can get tricky but I seek clarification on pitfalls regarding a different setup:- multiple vault services reading from a single secrets backend - no replication. ie, s3 or dynamo or postgre with more than one vault instance fetching from the same.

P.S. I apologize in advance if I'm hijacking the thread. It's totally inadvertent.

[Jeff Mitchell]

Hi James,

Vault doesn't know about invalidations to the physical layer unless it
makes them, so without disabling Vault's cache (which significantly
slows down its operation) it's just totally unsafe to do this. Even if
you disable Vault's cache, there is a lot of state that it loads,
keeps in memory, and writes out at certain events, so multiple Vaults
pointed at the same backend are going to stomp on each other. This can
be especially problematic with some of Vault's more advanced
functionality like leased secrets -- which includes tokens, so doing
this is a great way to allow e.g. limited-use tokens to be used for
more than the use-count they ought to support.

To a fairly large extent, this is a price that's paid for being
flexible in the underlying datastore support. We have to look at the
storage layer as a very basic abstracted object store, and can't take
rely on features that individual storage backends might provide. This
significantly increases complexity in many ways, but it also means
that customers can run Vault even if they're only comfortable with (or
in some cases, policy only allows) data storage in Dynamo, or etcd, or
ZK, or Postgres.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/dcd1bae6-8722-4da3-92e1-3fc142093a2e%40googlegroups.com.

[James]

Thanks Jeff. That's crystal-level clear now!

[Joel Thompson]

James,

I haven't personally tried this before, but Vault allows you to specify a different storage backend (which is what you're using Postgres for) and the HA backend for the coordination that Jeff is talking about: https://www.vaultproject.io/docs/configuration/index.html#ha_storage

I imagine it should be possible to add an HA backend of Consul (or any of the other HA-supporting backends) while maintaining all of your existing data in Postgres. It leaves you with both Postgres and Consul infrastructure to support, but it does allow you to accomplish your original goal of adding more Vault servers to the cluster.

--Joel

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/2e918d1c-f58a-4ba9-9166-267b1a12dbda%40googlegroups.com.

[Jeff Mitchell]

Hi James,

Joel is correct, but to be clear -- this allows adding failover Vault
instances, but it doesn't allow simultaneous data store access -- at
any given time only the active node serves requests (but the others
forward).

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/CAOXnK5Tu4d-8vWJG7GmF5qD7T-vzpq-N4a6Kq84w9X%2B4hYHyAA%40mail.gmail.com.