Vault with consul and 2 server nodes, suddenly both shows as standby.

[Redsmile]

Hi, 

I have initialized vault with a basic consul backend. This is my vault config:

backend "consul" {

  address = "127.0.0.1:8500"

  path = "vault/"

  scheme = "http"

  token = "consul-write-acl-token"

  cluster_addr="http://127.0.0.1:8201"

}

listener "tcp" {

  address = "0.0.0.0:8200"

  tls_disable = 1

}

disable_mlock = true

I have a consul server and two agents running, and the 'vault server config ' command starts the server, 

i set the VAULT_ADDR to 'http://127.0.0.1:8200' to avoid the https conflict, but when i try to 'vault init', this is the log output:

core: security barrier not initialized

core: failed to write seal configuration error=Unexpected response code: 403 (Permission denied)

core: failed to save barrier configuration error=failed to write seal configuration: Unexpected response code: 403 (Permission denied)

I can then use 'vault status' to get this:

Node 1:

$ vault status

2016/12/19 11:51:57.023423 [INFO ] core: seal configuration missing, not initialized

Error checking seal status: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/seal-status

Code: 400. Errors:

* server is not yet initialized

If i try vault init, i get this:

$ vault init

2016/12/19 11:54:50.581163 [INFO ] core: security barrier not initialized

2016/12/19 11:54:50.581849 [ERROR] core: failed to write seal configuration: err      or=Unexpected response code: 403 (rpc error: rpc error: Permission denied)

2016/12/19 11:54:50.581882 [ERROR] core: failed to save barrier configuration: e      rror=failed to write seal configuration: Unexpected response code: 403 (rpc erro      r: rpc error: Permission denied)

Error initializing Vault: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init

Code: 400. Errors:

* barrier configuration saving failed: failed to write seal configuration: Unexpected response code: 403 (rpc error: rpc error: Permission denied)

Node 2:

$ vault status

Sealed: false

Key Shares: 5

Key Threshold: 3

Unseal Progress: 0

Version: 0.6.2

Cluster Name: vault-cluster-3e7ad5c3

Cluster ID: 33ad5d4d-7277-3edf-5d14-7bc4b2716ef1

High-Availability Enabled: true

        Mode: standby

        Leader: <none>

Question:

The Node 1 was earlier acting as active leader and it was all fine. Suddenly, i see both are in standby. I tried restarting vault server on lx1394, now it shows as uninitialized.

How do i resolve this and make one of these nodes a leader.

Really confused, appreciate all your help, thanks!

[Jeff Mitchell]

Hi,

That error is coming from Consul -- the ACL token you're using doesn't
actually have correct permissions, which is why you're seeing the
permission denied messages.

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/db648902-ec35-44fd-b6a6-59a14da5c034%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Redsmile]

Hi Jeff,

Do i need to use a ACL token with write access to vault/ key  like this for it to work?

key "vault/" {
    policy = "write"
}

[Jeff Mitchell]

Hi,

At first glance that looks reasonable to me -- I do see that you have
a token specified in your config, so maybe the ACLs on it are not
correct?

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/ba307ebe-9119-40f4-a1bf-35b9b05ca8d7%40googlegroups.com.

[Redsmile]

Yes, My bad, the docs clearly mentioned the consul token we use should have write access to Consul from Vault, the problem was the token I specified had write access to consul but only to the kv store.

This issue got resolved when i created a separate write token in consul ACL for vault/ with write policy.

After doing that, I'm now able to see the node 1 is active and unsealed and vault read/write works fine.

Question on Consul though :

After all this is done, i was expecting Consul Web UI, to show node 1 with vault service being active, instead it shows Vault lock -sessions, what does that mean? Picture below:

[Jeff Mitchell]

Hi there,

Active nodes are active because they hold a session lock inside
Consul. Having a lock is normal!

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/d0da643b-2d13-42d2-9d6d-eb14e9749cb1%40googlegroups.com.

[Redsmile]

Thank you for the clarification.

Earlier, I remember that it used to show vault status: active for the active node.

Now it only shows vault status: standby for the standby node. Is it some change with any recent Consul Web UI versions?  

[Jeff Mitchell]

Not sure -- you'd have to ask the Consul team!

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/bbba00c0-4ba3-4038-b944-b609c0c203fa%40googlegroups.com.

[Redsmile]

Thank you once again for all your help!

I have posted that question in the Consul community.