Backup and Restore of Vault with Consul Backend

[Joshua Keys]

I am testing the scenario of a complete failure and recreation of both consul and vault. I understand consul supports multiple datacenters, but in the case where there is a complete loss of the consul and vault systems I am unable to get vault back online. The scenario is below.

Starting with a healthy consul and vault. I use python-consul to backup the vault key value store while the vault is sealed. This backup is a list of python dictionaries. The initial seal/unseal keys and root_token were also backed up at the time of vault initialization

I delete the "vault" top level folder in consul.

I destroy the server running vault.

I then run a small python script to replay the list of keys from the above backup into consul. That seems to work perfectly and consul now has a top level "vault" folder with "core" and "sys" subfolders. All key/values seem to be in place.

I then create a new vault server and start vault with config.hcl pointing to the correct consul backend endpoint.

I proceed to successfully unseal the vault with the keys backed up during initialization of the old and now destroyed vault system.

The problem is that the vault is unsealed and in standby mode. It cannot be utilized and cannot be sealed again as it is in standby mode. I understand that in production we would have multiple servers but again this is testing a complete loss and restore.

Thanks for any help provided.

[Joshua Keys]

Additional information.

Consul version 0.6.3

Vault version 0.5.1

Output of /sys/leader after unsealing from restored data in consul.

{"ha_enabled":true,"is_self":false,"leader_address":""}

Notice there is not a leader address. Health is below

{"initialized":true,"sealed":false,"standby":true,"server_time_utc":1457181071}

[Armon Dadgar]

Joshua,

Do you have any logs from the Vault server indicating a failure to elect a leader?

It seems that Vault is starting and unsealing correctly, but is failing to acquire the

lock and enter active operation. What are the contents of “vault/core/lock”?

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/088f17bf-16f6-4a8f-8d2b-29ad56aba66a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Mitchell]

Hi Joshua,

To add on to what Armon said, the key 'vault/core/lock' is a lock
within Consul. My guess is that something along the way of backing up
and restoring that value (not sure if it'd lie with python-consul,
Consul, Vault, or what) is causing that value to be permanent, and as
a result no other Vault node can grab that lock. If you exclude
'vault/core/lock' from your backup (or restore) I think things will
work properly.

Best,
Jeff

> https://groups.google.com/d/msgid/vault-tool/etPan.56dce777.2a85edeb.151dd%40Armons-MacBook-Air.local.

[Joshua Keys]

Thanks both Jeff and Armon. That was the fix. That lock was throwing the new server into standby mode. Once I excluded that the backup worked well.

-Josh

[Will Pinney]

@Josh and Jeff, 

How did you exclude the lock from the backup? Did you just backup everything then manually remove the /vault/core/lock entry? 

Thanks.

[Jeff Mitchell]

Hi Will,

Actually at this point I highly recommend using newer Consul with 'consul kv export'. This does a single-transaction read of the path or key that you give it. It dumps to JSON but properly base64s values on the way out, and you can then send this dump back in via 'consul kv import'. Once that's done you can simply 'consul kv delete core/lock' and you're done!

Best,

Jeff

[Will Pinney]

Jeff,

Thank you for the info. We are using vault 0.6.1.  Are the consul 0.7.x, which this feature (consul kv import) is in,  support vault 0.6.1? I think so. But I would like to double check before try out. 

Thanks again. 

-Will

[Jeff Mitchell]

Hi Will,

Yes, I believe it should work fine.

Best,

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/2bfb0679-4464-4daf-94fe-4aa8d5730562%40googlegroups.com.