No active vault nodes. All are in standby

2,607 views
Skip to first unread message

je...@srcclr.com

unread,
Dec 28, 2016, 2:28:14 AM12/28/16
to Vault
I have 3 vault instances connected to 3 etcd-servers. All vault instances are in standby node. Is there a way to forcefully set a node as active.

backend "etcd" {
  path = "vault/"
  advertise_addr = "https://vault.###:8200"
  ha_enabled = "true"
}


listener "tcp" {
  address = "0.0.0.0:8200"
  tls_cert_file = "/ssl/server.pem"
  tls_key_file  = "/ssl/server-key.pem"
}

Is there a way i can clear the active node details from etcd that would help.  

Jeff Mitchell

unread,
Dec 28, 2016, 12:57:04 PM12/28/16
to vault...@googlegroups.com
Hi Jeeva,

You can try removing the vault/core/lock value. Please note that the
current etcd backend has known HA problems (such as you're seeing
here) that may cause data issues. We recommend using a different
backend for now. In 0.6.5 we should have a backend that works against
the etcd3 API and is maintained by CoreOS and will hopefully not have
these issues.

Best,
Jeff
> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/d4ccefc2-cba6-4687-8918-3490737c5795%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

je...@srcclr.com

unread,
Dec 28, 2016, 7:57:12 PM12/28/16
to Vault
Thank you Jeff.
There is no path /vault/core/lock in the etcd. Removing the /vault/core/leader will help?

Regards,
Jeeva

Jeff Mitchell

unread,
Dec 28, 2016, 8:19:17 PM12/28/16
to vault...@googlegroups.com
Hi Jeevan,

Independently I learned today that etcd has virtual locks that don't correspond to physical entries, so you may not be able to do it this way. However, I also found out that it uses a different path ( :-( -- this will be fixed with the etcd3 support). Try removing vault/core/_lock/

Best, 
Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/bf32e215-72ca-4689-8c21-6d8d2ea0af9f%40googlegroups.com.

je...@srcclr.com

unread,
Dec 28, 2016, 8:34:03 PM12/28/16
to Vault
There is not lock key in the core. All i can see is,

 vault/core
            leader
             .seal-config
             .audit
             .keyring
             .mounts
             cluster
             .auth
             .master

  I tried deleting the leader expecting vault will create a leader when restarted. But vault status show no leader.

  "
High-Availability Enabled: true
Mode: standby
Leader: <none>"


Thanks,
Jeff
                 
Jeff

Jeff Mitchell

unread,
Dec 28, 2016, 8:43:57 PM12/28/16
to vault...@googlegroups.com
My understanding is that the _ in front makes it hidden from listing. So you won't see it in list output. 

Best,
Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/4c1e30bc-301f-442e-b16a-a041412289f3%40googlegroups.com.

je...@srcclr.com

unread,
Dec 28, 2016, 9:04:15 PM12/28/16
to Vault
Thank you Jeff. It works fine now.

Ahmed Nasir

unread,
Mar 23, 2018, 3:02:46 AM3/23/18
to Vault
We also faced a similar situation today but deleting the `vault/core/_lock` key didn't help. We are using dynamodb table, i deleted from the key from the table.
Reply all
Reply to author
Forward
0 new messages