Vault Cluster behind Loadbalancer

[gran...@distilnetworks.com]

Hello,

I run a Vault cluster that communicates using local advertise-addresses. In order to access the cluster, I currently use a load balancer that points to the master server, and this works fine, but I need to make the failover work properly in case the master node dies.

When I open up the load balancer to access the standby Vault nodes, I get redirected to the master node, which fails because it attempts to use the local IP.

What is the best way to solve this problem? I was think HAProxy with Consul Template, but I wanted to ask here first.

Thanks!

[David Adams]

If you're using Consul and have Consul DNS up and running on your servers, you can just register Vault as a Consul service and point your clients to `vault.service.consul` and the rest will take care of itself. No need for a load balancer (which is useless for Vault anyway since only one node will be active at a time). But if you are using a load balancer with a static IP, you can just set the Vault advertise address to that static IP, and then set the load balancer's health check to send clients to the currently-active host only. If your load balancer has dynamic IPs (like Amazon ELB) you probably should use something else.

-dave

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/f955ac5e-9036-4c9e-9b56-73d2a82b75b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gran...@distilnetworks.com]

So, it's ok to have all the vault advertise addresses to the actual DNS entry and all 3 servers will have the same advertise address set? I thought that was used by the vault servers to communicate.

If that works, that makes it easy.

[Jeff Mitchell]

Hi Grant,

Vault communication between servers happens only via the physical data store.

The advertise_addr, on any given active node A, is what standby nodes
X, Y, and Z should use when redirecting clients to A. So if you're
using a load balancer (which we generally do not recommend, but in
some cases is not avoidable) that is automatically directing clients
to the active node, you want the advertise_addr for each node to be
the address of the load balancer. That way if a client gets a standby
node by accident (say, if the active node is failing over), it'll just
redirect back to the load balancer to try again.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/19d899fc-1822-4d90-ba25-b171ff51dbf2%40googlegroups.com.

[gran...@distilnetworks.com]

That makes sense. I will follow your advice and change the advertise address.

It does seem like this could be improved using Consul Template to writes the HAProxy config for the load balancer to always point to the master Vault server.

Jeff, thanks again for the help and I hope we get to meet at HashiConf!

- Grant

[Francisco Javier Romero Mendiola]

Finally, did you use ELB? I thought use an ELB but it looks anybody recommend it..

Francisco.

[Matt Hurne]

Jeff,

You mentioned that fronting Vault with a load balancer is not generally recommended. Can you direct me to a discussion or documentation that describes why putting Vault behind a load balancer is not recommended but is sometimes unavoidable? And/or a discussion or documentation that presents alternatives and considerations for choosing between them?

Thanks,

Matt

[gran...@distilnetworks.com]

Matt,

One of the problems with using a load balancer for Vault is that if you connect to a non-active node, you will be forwarded to the active node, so if you use a load balancer, you will end up being redirected anyways so it doesn't make sense.

However, I was able to get around this by using Consul Template with the newest version of HAProxy (which allows SSL passthrough) to always forward the load balancer to the active Vault node. If the active node changes, the HAProxy config changes and HAProxy restarts.

Hope that helps, I'm sure Jeff can give you a better answer.

Grant

[Jeff Mitchell]

Commercial load balancers:

1) Can provide some protection against things like DDoS attacks
2) Can provide an entrypoint if you do not have an architecture that
includes service discovery

But:

3) They are usually slower to pick up a change in the active node than
the Vault instances, which can lead to redirect loops and service
delays
4) Many implementations and/or configurations decrypt and re-encrypt
traffic, which is not compliant with some industry/government
standards

Using Vault behind a load balancer is not recommended in a very
literal sense -- we do not recommend it. We also don't recommend
against it. :-) Generally, we just don't see a reason to use it
unless you actually have a reason to use it, depending on your
infrastructure and security needs.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/516a2d5a-ac01-4fb8-9b09-e4a21c812359%40googlegroups.com.

[Matt Hurne]

Thanks, Grant and Jeff! It is helpful to know what considerations are
involved in the decision.

Unfortunately it seems the recommendations in this area have evolved
over time. For example, we were thinking it would be nice if
consul-template (our preferred Vault client) supported direct
integration with Consul to discover the active Vault node, especially
after the latest Vault release tightened up the integration between
Vault and Consul on the Vault side. But it seems that idea was already
proposed back in October of 2015, and was not pursued because use of a
load balancer was recommended instead:
https://github.com/hashicorp/consul-template/issues/425

Would you recommend reopening that issue under the assumption that the
thinking behind that decision has likely changed?

Regards,
Matt

> You received this message because you are subscribed to a topic in the Google Groups "Vault" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/vault-tool/Ep6hBDqoBAY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GGjog3afMExLnDxMMta%3Dgh60a7Sfr357Q7boREb-jF6%2Bw%40mail.gmail.com.

[Jeff Mitchell]

On Wed, Jun 22, 2016 at 3:54 PM, Matt Hurne <ma...@thehurnes.com> wrote:
> Thanks, Grant and Jeff! It is helpful to know what considerations are
> involved in the decision.
>
> Unfortunately it seems the recommendations in this area have evolved
> over time. For example, we were thinking it would be nice if
> consul-template (our preferred Vault client) supported direct
> integration with Consul to discover the active Vault node, especially
> after the latest Vault release tightened up the integration between
> Vault and Consul on the Vault side. But it seems that idea was already
> proposed back in October of 2015, and was not pursued because use of a
> load balancer was recommended instead:
> https://github.com/hashicorp/consul-template/issues/425

That ticket is about someone trying to figure out an entrypoint to the
Vault cluster with a static address without having service discovery,
which I noted as a reason you might want to use a load balancer:

>> 2) Can provide an entrypoint if you do not have an architecture that
>> includes service discovery

It doesn't sound like this applies to your use-case!

Best,
Jeff

[Francisco Javier Romero Mendiola]

Hi,

At the moment,I am using a loadbalancer as a entrypoint, It checks which node is primary and forward requests.

In this case loadbalancer will never send a request to secondaries servers.

Regards.

Francisco

[Matt Richter]

I'm not happy with the Master-Slave scheme for Vault.. You'd think that in this day and age of stateless services that Vault would be able to work like that out of the box. Would be great for those of us trying to deploy it in Marathon, Kubernetes etc.

[Michael Fischer]

There is no master-slave scheme.  It's active-standby.  What is your issue, exactly?

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/b0351198-3ed1-4e91-a60f-f94e32d9319e%40googlegroups.com.

[Jeff Mitchell]

A real benefit to the current approach is that you don't need to worry
about consistency. That's a _very_ good property in a security
product.

It's also not reasonable to think that all services should be
stateless. I won't discuss the many other services for which stateful
operation makes the most sense, but think about lease/token expiration
management within Vault. Do you really want Vault to poll every single
lease in its data store (all 100, or 1000, or 10000, or 100000, or
1000000) in a tight loop to check expiration status? Considering that
almost all data store backends in Vault are networked, the answer
should be a resounding "no". If you're running 100,000 transit
operations per minute through Vault, do you want Vault to perform a
new read(, parse, prepare) against its data store 100,000 times to
load the key fresh every time in case it's been rotated rather than
store its state in a cache and persist as needed? Again, the answer
should be a resounding "no".

Taking a stateful service and having proper master-slave or
master-master semantics is also hard, especially when you want to be
able to reason about the current state of the world, as you probably
do in a security product. Vault will support this if and when we have
an approach that we know is workable and correct, but not before.
Brief availability blips during a failover that can be worked around
by using client retries is better than outages due to split-brain
problems. Obviously HashiCorp has experience with Raft, but at the
same time, we have enough experience with it to know that simply
shoving it into Vault isn't an obviously right thing to do.

Stateless services have their place, but anyone that tells you that
all services should be stateless all the time does not have a good
grasp of real-world problems. Or is trying to sell you something.

--Jeff

> https://groups.google.com/d/msgid/vault-tool/CABHxtY5s01FuVkYm-e-75GfLmvKcUFE7uLU4FuJ%3DgtDDuWL_mw%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.

> https://groups.google.com/d/msgid/vault-tool/CABHxtY5s01FuVkYm-e-75GfLmvKcUFE7uLU4FuJ%3DgtDDuWL_mw%40mail.gmail.com.

[Michael Fischer]

Not only that, but proper configuration of a load balancer is not identical for every backend service.  Stateless services may be amenable to round-robin load balancer configurations, but active-standby services such as Vault are not.  Good load balancers are flexible enough to capably front both kinds of services.

Best regards,

--Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GEG_sK5trajcLDXBBM4Afd2EisYDDhMDRSPUboaxy_BJg%40mail.gmail.com.

[Matt Richter]

Sounds like I hit a nerve! I was having some frustrating issues running an HA clustered Vault in Marathon. I've been able to smooth most of those over, I am now battling with TLS which is a whole different story.