Entire Vault Cluster is Mysteriously Sealing

[Wesley Staples]

I have a process that uses app_id (or was it app_role?) to login to vault and get some secrets.
The process is based on the official ruby vault client. 3 days ago the process started returning an error that the vault is sealed.
Upon inspection I found that all 3 of my vault servers were sealed.

$ vault status
Seal Type: shamir
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
Unseal Nonce:
Version: 0.9.1

High-Availability Enabled: true
        Mode: sealed

I unsealed all 3 vault servers and everything seemed to work fine for 2 days. Today I have found that all 3
vault servers are again in a sealed state. In the interest of getting the servers working again I have unsealed the vaults.

My question is how can I get more information on whats happening? is someone actually running the seal command?
What information do I need to collect to troubleshoot this problem?

Here is my setup:
I have 3 vault servers at version Vault v0.9.1 ('87b6919dea55da61d7cd444b2442cabb8ede8ab1')
These are running on the same servers as my Consul cluster. Vault uses consul as its backend.
Consul is at v0.7.0

[PePe Amengual]

Hi.

Per the documentation you can only have one Active Vault server.

Note: if the secondary is in an HA cluster, you will need to ensure that each standby is sealed/unsealed with the new (primary’s) unseal keys. If one of the standbys takes over on active duty before this happens it will seal itself to remove it from rotation (e.g. if using Consul for service discovery), but if a standby does not attempt taking over it will throw errors. We hope to make this workflow better in a future update.

https://www.vaultproject.io/guides/operations/replication.html.

are you using HA ?

[Jeff Mitchell]

Hi Wesley,

Logs should provide some insight -- you might want to get them at debug or trace level. I'd also, separately, recommend upgrading Consul in case some Consul stability issue caused Vault's storage to go wonky and Vault to seal itself. Consul has gotten a lot more stable since 0.7!

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/5ad2f80a-d57a-410d-a12f-d5a3a0d01bb2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Wesley Staples]

We are quite a bit behind on Consul. I think I will just update the servers and go from there. Thanks for your help.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.