Vault Seal / Unseal - Backend Consul - Production Ready Enviroment

[Alex Kamalov]

Greetings folks,

I've deployed containerized  Vault without "-dev" flag on and integrated it with Consul. As it stands Consul has registered Vault. Problem is, when I ran the container with "-dev" flag on, I had VAULT_TOKEN, "Unseal Key" and "Root Token" generated for me. However, when I ran the container without "-dev" flag, none of them were generated. Here is my vault.json file:

backend "consul" {

  address = "10.9.9.65:8500"

  path = "vault"

}

listener "tcp" {

  address = "0.0.0.0:8200"

  tls_disable = 1

}

Container log does not show any tokens been generated:

wwserver194 vault # docker logs vault

==> Vault server configuration:

         Log Level: info

             Mlock: supported: true, enabled: true

           Backend: consul (HA available)

 Advertise Address: http://10.9.9.65:8200

        Listener 1: tcp (addr: "0.0.0.0:8200", tls: "disabled")

==> Vault server started! Log data will stream in below:

wwserver194 vault # 

Container was run with the following command:

wwserver194 vault # docker run  -d -p 8200:8200 --name vault --memory-swap=-1 -v /var/lib/vault:/etc/vault --cap-add IPC_LOCK akamalov/vault

Now, as it stands, Vault is sealed. I can't create a token unless Vault is unsealed. The only thing I found in Consul interface was the following key:

vault/sys/token/id/fd73f0c9c83e60b5366e3dcf1ddae4e2d61402ea

Is this the token I can use to unseal the vault? I tried it with 'curl' with 'vault', however, none of them worked. I did look into https://vaultproject.io/intro/getting-started/authentication.html , https://vaultproject.io/docs/auth/app-id.html, and https://vaultproject.io/docs/secrets/generic/index.html but couldn't find answers. 

Thanks so much,

Alex

[Armon Dadgar]

Hey Alex,

When you start Vault in “-dev” mode, it automatically invokes “vault init” to initialize the backend

and provides the unseal keys and the root token.

In normal operation, you must initialize the Vault the first time and store the unseal keys and

root token. Those are only provided once when “vault init” is invoked. Given that you have

data in Consul, it seems that has already been done, but you maybe did not store the output.

Your best bet is to clear all the data in Consul, and then run “vault init” again, this time preserving

the output.

Hope that helps!

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To post to this group, send email to vault...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/aea15cb2-64d6-4f3d-ac97-81868aa6c4de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alex Kamalov]

Thank You very much Armon. As per your recommendation, I went ahead and re-initialized Vault and it is all good now :) Thanks so much again for your help!!!

Alex

On Tuesday, June 23, 2015 at 5:39:33 AM UTC-4, Armon Dadgar wrote:

Hey Alex,

When you start Vault in “-dev” mode, it automatically invokes “vault init” to initialize the backend

and provides the unseal keys and the root token.

In normal operation, you must initialize the Vault the first time and store the unseal keys and

root token. Those are only provided once when “vault init” is invoked. Given that you have

data in Consul, it seems that has already been done, but you maybe did not store the output.

Your best bet is to clear all the data in Consul, and then run “vault init” again, this time preserving

the output.

Hope that helps!

Best Regards,

Armon Dadgar

[Jack Hsu]

Hi Alex,

I'm having similar problem.  I was able to test vault in '-dev inmem' mode up to this point and everything seems to work fine.  Now, I want to enable consul for backend persistence.  But whenever I add consul to vault config json, I can't issue any vault CLI command anymore as vault is sealed when consul is added to the startup config.  I try to unseal it with the unseal key obtained from '-dev inmem' run earlier, but it doesn't accept it. 

How did you unseal when you introduced the consul to config file in the first place?  (Note: I'm not using docker.)

startup json when introduce consul on the config:

listener "tcp" {

  address = "10.0.0.24:8443"

  tls_cert_file = "/apps/vault/conf/my_mac.crt"

  tls_key_file = "/apps/vault/conf/my_mac.key"

}

listener "tcp" {

  address = "127.0.0.1:8200"

  tls_disable = 1

}

backend "consul" {

  address = "10.0.0.24:8500"

  datacenter = "dc1"

  path = "vault"

}

After server is started and in seal mode, I tried this unseal cmd but failed:

# vault unseal 14837a18f9392a4518616a8232a1f294a658e49a8b9823321720142e32ed8ded

Error checking seal status: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/seal-status

Code: 500. Errors:

* failed to check seal configuration: Unexpected response code: 500

vault log:

2015/06/26 19:50:56 [ERR] core: failed to read seal configuration: Unexpected response code: 500

Thanks

[Armon Dadgar]

Hey Jack,

The initialization process is done once per backend. Having switched to a new backend, 

you need to run “vault init” again. This will return a new set of unseal keys and root key.

When in dev mode, you are running against a purely in-memory Vault. It does not persist

anything, and so everything is lost when it stops. It is only suitable for testing and development

as such.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---

You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/13cbdc99-e822-4f7b-bdae-f3df17591ac7%40googlegroups.com.

[Jack Hsu]

Hi Armon,

Same error when "vault init" the backend.

# vault init

Error initializing Vault: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/init

Code: 400. Errors:

* failed to check for initialization: Unexpected response code: 500

# vault status

Error checking seal status: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/seal-status

Code: 500. Errors:

* failed to check seal configuration: Unexpected response code: 500

vault log:

2015/06/27 09:06:31 [ERR] core: barrier init check failed: failed to check for initialization: Unexpected response code: 500

2015/06/27 09:07:38 [ERR] core: failed to read seal configuration: Unexpected response code: 500

config file:

listener "tcp" {

  address = "10.0.0.24:8443"

  tls_cert_file = "/apps/vault/conf/my_mac.crt"

  tls_key_file = "/apps/vault/conf/my_mac.key"

}

listener "tcp" {

  address = "127.0.0.1:8200"

  tls_disable = 1

}

backend "consul" {

  address = "10.0.0.24:8500"

  datacenter = "dc1"

  path = "vault"

}

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jack Hsu]

Resolved the "vault init" problem by specifying "bind_addr" as "0.0.0.0" on all consul nodes.  I'm able to unseal the vault with consul backend.  Thanks, Armon

# vault init

Key 1: 8b0fdf7abdaa2e522400438e4cad00849361ac5379a783b77de3bb715a523d5f01

Key 2: 69b53e73d09e71831b969769667d9dee5dcfe7814e891d456c88a6989f71643a02

Key 3: 96fef18bc47fc93b68aa1dfaafa64a86c9f709580e20c41888854b8dca2f7df903

Key 4: 833ad61304ca453051162acaebce928d0cdb8022260e41fbf0c969f2dcdf772704

Key 5: 7c7119eb102bfd88222aa059221545e598e36efb66a798a614c484e789816ee405

Initial Root Token: 60bb9e9b-d2be-129c-b2bf-f51a716c1850

Vault initialized with 5 keys and a key threshold of 3. Please

securely distribute the above keys. When the Vault is re-sealed,

restarted, or stopped, you must provide at least 3 of these keys

to unseal it again.

Vault does not store the master key. Without at least 3 keys,

your Vault will remain permanently sealed.

# vault unseal 8b0fdf7abdaa2e522400438e4cad00849361ac5379a783b77de3bb715a523d5f01

Sealed: true

Key Shares: 5

Key Threshold: 3

Unseal Progress: 1

# vault unseal 69b53e73d09e71831b969769667d9dee5dcfe7814e891d456c88a6989f71643a02

Sealed: true

Key Shares: 5

Key Threshold: 3

Unseal Progress: 2

# vault unseal 96fef18bc47fc93b68aa1dfaafa64a86c9f709580e20c41888854b8dca2f7df903

Sealed: false

Key Shares: 5

Key Threshold: 3

Unseal Progress: 0

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jack Hsu]

Hi Armon,

After restart the vault (using consul as backend), the vault is sealed again.  Is it true that we need to unseal with 3 keys every time when we restart the vault using consul?

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jack Hsu]

Also, when I mount the consul right after I unseal it at startup, it fails because the vault is still in "standby" mode at that time.  I had to sleep for 10 seconds in my startup script and wait for vault to become 'active' before mounting the consul.  Is this the normal behavior?

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Armon Dadgar]

Hey Jack,

That is correct. The unseal keys are required any time Vault starts or is sealed.

This is because the unseal keys are used to compute the actual encryption key,

so Vault cannot access anything in the storage backend without that.

Best Regards,

Armon Dadgar

From: Jack Hsu <jhs...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: June 29, 2015 at 10:27:03 AM
To: vault...@googlegroups.com <vault...@googlegroups.com>>
Subject:  [vault] Re: Vault Seal / Unseal - Backend Consul - Production Ready Enviroment

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/7030f900-9b0f-469b-ac0f-7b171f4aab9b%40googlegroups.com.

[Armon Dadgar]

Hey Jack,

If you are using a highly available backend, only the currently active instance can

service requests, so there may be a delay while lock acquisition and leader election

takes place.

Best Regards,

Armon Dadgar

From: Jack Hsu <jhs...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: June 29, 2015 at 10:56:30 AM
To: vault...@googlegroups.com <vault...@googlegroups.com>>
Subject:  [vault] Re: Vault Seal / Unseal - Backend Consul - Production Ready Enviroment

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/00b2d925-879b-4d25-9a20-3f75b2796d38%40googlegroups.com.

[Jack Hsu]

Armon,

Here's my thought of running vault with consul in HA:

- Total 3 servers in one DC.  (we're discussing only one DC here)

- Each server has both consul and vault running.

- Each vault instance only connects to local consul on port 8500.

- One of vault standby instance will become active if it sees the active vault died (I just test this ok).  I guess vault standby instance detects active vault died via its connected consul as there is no inter-communication between vault instances.  Correct?

- We'll create a VIP and bind it to the 3 vault instances so client can access vault via this VIP rather developing smart client to discover which vault instance is active.

Question:

- How would load balancer know which vault instance is 'active' and which is 'standby' if the port (8443 in this case) is being listened by all the vault instance regardless it's active or standby which is only visible to vault internal?  In another word, they are all 'active' from load balancer standpoint because port 8443 is up on all vault instances.

- Do you provide some kind of ECV health monitor REST API for load balancer to check whether that instance is accepting client request (e.g. active instance) or not so load balancer can mark that route as "down"?

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jack Hsu]

Correction:

- Do you provide some kind of ECV health monitor REST API for load balancer to check whether that instance is accepting client request (e.g. active instance) so load balancer can mark that route as "UP" and other routes as "DOWN"?

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Armon Dadgar]

Hey Jack,

Your explanation for how it works is dead on. We do something very similar for our internal Vault cluster as well.

You want to use the “/v1/sys/health/“ endpoint (https://vaultproject.io/docs/http/sys-health.html). This matches

the semantics of the Consul native HTTP health checks, but can also be used as a LB health check endpoint.

Best Regards,

Armon Dadgar

From: Jack Hsu <jhs...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: June 29, 2015 at 1:52:00 PM
To: vault...@googlegroups.com <vault...@googlegroups.com>>
Subject:  [vault] Re: Vault Seal / Unseal - Backend Consul - Production Ready Enviroment

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/2646a1ba-16e4-41bc-82ab-7ed7c1869e59%40googlegroups.com.

[Jeff Mitchell]

Jack,

If you connect to a non-active instance via HTTP it will issue a 307
temporary redirect guiding your client to retry the call on the active
instance.

In addition, if you look at
https://vaultproject.io/docs/http/sys-health.html, you will see that a
non-active instance returns a 429 for that call. You can use this to
tell your load balancer, in its checks, which instance is active. If
you are using Consul for service discovery, this will "just work".

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/2ed5758d-906f-45aa-99d6-138fdf4cba44%40googlegroups.com.

[Jack Hsu]

Works beautifully !!   Thanks, Jeff

When vault instance is active:

$ curl -sw '%{http_code}' --tlsv1.2 -X GET --cert /Users/jack.hsu/Desktop/vault/conf/my_mac.crt --key /Users/jack.hsu/Desktop/vault/conf/my_mac.key https://mac-guest-linux01:8443/v1/sys/health

{"initialized":true,"sealed":false,"standby":false}

200

When vault instance is standby:

curl -sw '%{http_code}' --tlsv1.2 -X GET --cert /Users/jack.hsu/Desktop/vault/conf/my_mac.crt --key /Users/jack.hsu/Desktop/vault/conf/my_mac.key https://mac-guest-linux01:8443/v1/sys/health

{"initialized":true,"sealed":false,"standby":true}

429

When vault instance is sealed:

curl -sw '%{http_code}' --tlsv1.2 -X GET --cert /Users/jack.hsu/Desktop/vault/conf/my_mac.crt --key /Users/jack.hsu/Desktop/vault/conf/my_mac.key https://mac-guest-linux01:8443/v1/sys/health

{"initialized":true,"sealed":true,"standby":true}

500

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jack Hsu]

Hi Jeff,

In consul multi-datacenter scenario, do we have only one active vault instance across all data centers, or one active instance per data center?  Document on https://www.consul.io/docs/agent/options.html doesn't have details on how multi-datacenter works.  Do you have sample consul cluster config file for multi-datacenter?

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Jeff Mitchell]

On Mon, Jun 29, 2015 at 6:40 PM, Jack Hsu <jhs...@gmail.com> wrote:
> Hi Jeff,
>
> In consul multi-datacenter scenario, do we have only one active vault
> instance across all data centers, or one active instance per data center?
> Document on https://www.consul.io/docs/agent/options.html doesn't have
> details on how multi-datacenter works. Do you have sample consul cluster
> config file for multi-datacenter?

One Vault server per Vault cluster is active at a given time. It's
actually pretty flexible, though: any Vault servers that you connect
to the same Consul server at the same path (the "path" and "address"
options for the Consul backend, as seen here:
https://vaultproject.io/docs/config/index.html) will form a Vault
cluster. You can hang multiple Vault clusters off of a single Consul
instance, or you can have multiple Vault clusters across multiple
Vault instances, or a single Vault cluster across all of your
datacenters.

--Jeff

[Clay Bowen]

Maybe I'm being obtuse, but I thought with the way Vault works you could have several vault servers behind a load balancer going to a conul backend.  It shouldn't matter which one a user connects to, as the information is stored on the backend (tokens, leases, etc.).  The only issue I could see is if there is a write operation to the same location on consul (I don't know consul well enough to know how it handles multiple writes to the same location).

Am I missing something?

Thanks,

Clay

[Jack Hsu]

My original question to Jeff is to verify whether we could end up two active vault instances in following setting:

DC1

server 11: one vault instance connecting to one local consul instance (this vault will be active in local consul LAN)

server 12: one vault instance connecting to one local consul instance (this vault will be standby)

server 13: one vault instance connecting to one local consul instance (this vault will be standby)

DC2

server 21: one vault instance connecting to one local consul instance (QUESTION BELOW)

server 22: one vault instance connecting to one local consul instance (this vault will be standby)

server 23: one vault instance connecting to one local consul instance (this vault will be standby)

Consul instance on server 11, 12, 13 are joined each other via join "retry_join".

Consul instance on server 21, 22, 23 are joined each other via join "retry_join".

Consul instance on server 11, 21, 22, 23 are joined each other via join "retry_join_wan".

Consul instance on server 12, 21, 22, 23 are joined each other via join "retry_join_wan".

Consul instance on server 13, 21, 22, 23 are joined each other via join "retry_join_wan".

Consul instance on server 21, 11, 12, 13 are joined each other via join "retry_join_wan".

Consul instance on server 22, 11, 12, 13 are joined each other via join "retry_join_wan".

Consul instance on server 23, 11, 12, 13 are joined each other via join "retry_join_wan".

To simplify what I said above, all Consul instances are join each other either via local LAN (for nearby neighbors) or remote WAN (for relatives live in remote).  This will form a single large Consul cluster for HA and disaster recovery.

Question:

Will vault instance on server 21 also be active on its local consul LAN?  Or, we can only have one active vault in such cross-DC consul cluster?

Hope I explain my question clearly.

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Armon Dadgar]

Hey Jack,

In this case you will have 2 distinct Vault clusters. You can think about a Vault cluster as

a group of instances that share the same backend data. (Same data, same cluster).

Because Consul has a distinct K/V store per datacenter, the DC1 and DC2 data are completely

distinct. Thus the Vault servers [11-13] are reading/writing to different data than [21-23].

I hope that helps!

Best Regards,

Armon Dadgar

From: Jack Hsu <jhs...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: July 1, 2015 at 1:37:52 PM
To: vault...@googlegroups.com <vault...@googlegroups.com>>
Subject:  [vault] Re: Vault Seal / Unseal - Backend Consul - Production Ready Enviroment

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/8d6db193-5724-45bd-a7a4-b0eafc2e8a08%40googlegroups.com.

[Jack Hsu]

Hi Armon,

But when I point "retry_join_wan" on consul server in DC1 to consul servers in DC2, will it replicate the data between the two consul clusters?  If not, then how do we normally set up consul on DR site if remote consul cluster is always "different" K/V than local consul cluster?

Thanks

On Monday, June 22, 2015 at 11:13:23 AM UTC-7, Alex Kamalov wrote:

[Rusty Ross]

Jack,

WAN-joined datacenters will still not automatically replicate their local KV stores to one another.

You might have a look at:  https://github.com/hashicorp/consul-replicate

Though, I haven't given too much thought to how well replicated KV stores would work. My guess is: not too well, as at the very least, it seems you would run into a number of conflict resolution issues.

Rusty