Vault ha Consul docker setup, takes at least 30 seconds to failover

314 views
Skip to first unread message

Werner Dijkerman

unread,
Jan 12, 2017, 12:17:30 PM1/12/17
to Vault
Hi,

After spending several hours on the webs and in this group, I'm creating this post. I have a Consul backend and want to achieve a Vault HA setup.

I have 2 vault nodes, both running in docker (Same as Consul).

Host 1:
---8<---
backend "consul" {
    address = "10.10.10.11:8500"
    check_timeout = "5s"
    path = "vault/"
    scheme = "http"
}

listener "tcp" {
    address = "0.0.0.0:8200"
    cluster_address = "10.10.10.11:8200"
    tls_disable = 0
    tls_key_file = "/vault/ssl/vault.service.dj-wasabi.local.key"
    tls_cert_file = "/vault/ssl/vault.service.dj-wasabi.local.crt"
}

disable_mlock = false
---8<---
Host has ip: 10.10.10.11 and Consul (Agent) is listening on 10.10.10.11. The following environment vars are set:
-e VAULT_CLUSTER_ADDR=https://10.10.10.11:8200 \
-e VAULT_REDIRECT_ADDR=https://10.10.10.11:8200 \
-e VAULT_ADVERTISE_ADDR=https://10.10.10.11:8200 \

Host 2:
---8<---
backend "consul" {
    address = "10.10.10.12:8500"
    check_timeout = "5s"
    path = "vault/"
    scheme = "http"
}

listener "tcp" {
    address = "0.0.0.0:8200"
    cluster_address = "10.10.10.12:8200"
    tls_disable = 0
    tls_key_file = "/vault/ssl/vault.service.dj-wasabi.local.key"
    tls_cert_file = "/vault/ssl/vault.service.dj-wasabi.local.crt"
}

disable_mlock = false
---8<---
Host has ip: 10.10.10.12 and Consul (Agent) is listening on 10.10.10.12. The following environment vars are set:
-e VAULT_CLUSTER_ADDR=https://10.10.10.12:8200 \
-e VAULT_REDIRECT_ADDR=https://10.10.10.12:8200 \
-e VAULT_ADVERTISE_ADDR=https://10.10.10.12:8200 \

Is this a correct vault ha configuration?

When I do a seal or stop the "active" container (Checked with my own and the official vault docker container) it takes at least 30 seconds before 'active.vault.service.dj-wasabi.local' is returning the other ip address. When checking the Consul UI it keeps holding the 'active' tag for a while. In the mean time, the 'active.vault.service.dj-wasabi.local' isn't returning a correct ip and around 20 - 30 seconds later it returns the ip of the 2nd vault instance (And in the Consul UI the 'active' tag is on the 2nd vault too')

After reading a lot of information in this group and the official documentation, I don't see - a hopefully small - configuration error in my setup...
Please tell me you see it? :-)

Consul: 0.7.2
Vault: 0.6.4

Thanks in advance.

Kind regards,
Werner

Jeff Mitchell

unread,
Jan 12, 2017, 12:21:12 PM1/12/17
to vault...@googlegroups.com
Hi Werner,

Nothing seems obviously wrong. One immediate thing to try would be to
see whether you see the same behavior when Consul and Vault are
running on the host instead of in containers...this works surprisingly
often and if so points to some Docker iptables/firewall/nat issues
being the root cause.

Best,
Jeff
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/eb53d012-036c-440c-8c6b-8405cda246f4%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Randy Fay

unread,
Jan 12, 2017, 12:33:25 PM1/12/17
to vault...@googlegroups.com
Our prod HA vault/consul config is at https://github.com/drud/vault-consul-on-kube - it's kubernetes-focused, but it's containers and the whole process is explained in an extensive README. It's still incubating, but like I say it's working in production.

And of course any comments on it are welcome.

-Randy

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GGDcN_Ng3bRVR6ebWcg1bH3OLKvnYdnTVb6m5wxLRU0Mg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.



--

Werner Dijkerman

unread,
Jan 12, 2017, 12:55:14 PM1/12/17
to Vault
@ Jeff

Thank you for your reply. I manually installed Consul and Vault on my 2 hosts, but they have the same behaviour as running in containers.

@ Randy

Also thank you for replying. Will take a look at it! Thanks!

Op donderdag 12 januari 2017 18:21:12 UTC+1 schreef Jeff Mitchell:
Reply all
Reply to author
Forward
0 new messages