Moved Consul servers backend and now I get an error

[Khusro Jaleel]

Hi, I have a Consul cluster setup at "consul1, consul2, consul3" hosts, and my Vault nodes are running Consul agents locally. These agents were talking to the Consul backend just fine and everything was working.

I wanted to shut down the Consul servers on "consul1, 2, 3" because I wanted Vault to use a completely different Consul cluster. So I went to each of my Vault nodes and did a "consul leave" so that the local agent left the existing cluster. I also shut down Vault itself on each node. In addition, I also moved the "data" directory of the Consul agents to a different name, so that they could start fresh when join the other cluster. The other cluster also has a different datacenter.

I then re-started the local Consul agent nodes and did a "join" to the different cluster that I wanted to use. The local Consul agents were able to successfully join the other cluster.

Now, when I start up Vault, it complains with the following error:

"2018-05-31T10:43:55.656Z [WARN ] check unable to talk with Consul backend: error="Unexpected response code: 500 (CheckID "vault:vault-1.foo.com:443:vault-sealed-check" does not have associated TTL)""

I can't "unseal" Vault either at the moment. Have I missed a step here somewhere? 

This is the HA config in my Vault config file:

===

ha_storage "consul" {

  address = "localhost:8500"

  path = "vault"

}

===

[Jeff Mitchell]

Hi Khusro,

Can you list full logs with trace mode enabled? I don't believe that warning should cause Vault to not start up.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a425d9b7-b614-42f1-87c0-ba8dc81bf88f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Khusro Jaleel]

Hi Jeff, apologies for my late reply. 

The problem was simply that communication between the Consul agents (running on the Vault nodes) and the Consul "servers" on port 8300 was not allowed. I noticed the following in the Consul agent logs:

===

    2018/06/04 10:04:50 [ERR] consul: "Coordinate.Update" RPC failed to server consul-server-1:8300: rpc error getting client: failed to get conn: dial tcp <nil>->consul-server-1:8300: i/o timeout

    2018/06/04 10:04:50 [ERR] agent: Coordinate update error: rpc error getting client: failed to get conn: dial tcp <nil>->consul-server-1:8300: i/o timeout

    2018/06/04 10:04:50 [ERR] consul: "Catalog.NodeServices" RPC failed to server consul-server-1:8300: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection

    2018/06/04 10:04:50 [ERR] agent: failed to sync remote state: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection

===

I've fixed that on the firewall and things seem to be working fine now, thanks for your help! 

Khusro