Backing up and Restoring HA Vault via Consul migration to and from File Backend

[John Ruiz]

I have a Vault 0.7.3 HA cluster backed by a Consul cluster.  I use https://github.com/nebtex/vault-migrator to migrate the Vault mount in Consul to a file backend:

{

    "to": {

        "name": "file",

        "config": {

            "path": "/Users/jar349/vault-backups/data"

        }

    },

    "from": {

        "name": "consul",

        "config": {

            "address": "localhost:8500",

            "path": "vault/"

        }

    }

}

I use GPG to create an encrypted tarball of the migrated files and store it as a backup.  So no problems with backups.

The problem is figuring out how to restore to a NEW Vault (and Consul) cluster for disaster recovery.  I've tried allowing the migration to restore the <vault-mount>/core keys and restoring everything but <vault-mount>/core and neither option leads to a working vault with its data restored.  I've tried initializing the new cluster and then restoring as well as restoring before initialization and nothing seems to work.

I'm looking for advice on exactly which keys I *should* and *should not* migrate back into consul when moving to an entirely new Vault cluster backed by a new Consul cluster.

[Nathan Basanese]

  // , I do not know much about the Vault Migrator tool. 

But I do know a little about Consul Snapshot. If you're using a Consul backend, that may be a better alternative. 

https://www.consul.io/docs/commands/snapshot/agent.html

I'm also curious, what's your procedure for restoring from these backups? 

Do you start the new Vault cluster before you do any of these restoration efforts? 

Have you tried writing a custom, API driven approach to re-writing the secrets back in from your backup files, in a Data as Code approach? https://www.hashicorp.com/blog/codifying-vault-policies-and-configuration

But honestly, because it supports replication/restoration/continuous backup/Consul Snapshot Agent natively, the Enterprise version of Vault 0.7.3 might end up being cheaper in the long run. 

That's especially true if this is taking up time, or the potential data loss is a risk to the business.

https://github.com/hashicorp/vault/blob/master/CHANGELOG.md#073-june-7th-2017

[John Ruiz]

Hello!  Thanks for responding!

I'm not quite ready to abandon vault-migrator, but I will certainly look into Consul Snapshot and I thank you for the pointer.

You ask about my procedure for restoring from backups and my answer is that I'm trying to find a procedure that works.  The first thing that I tried was to restore everything, including the core/ folder of keys.  The problem there is that my new vault cluster thinks it's the old vault cluster and no leader ever gets elected because none of the other participants exist (because it's a new cluster of Vault servers on new hostnames)

Then I tried restoring everything - except the core/ folder - to a new uninitialized vault server.  The error when I try to initialize Vault after that is:

$ vault init

  Error initializing Vault: Error making API request.

  URL: PUT http://127.0.0.1:8200/v1/sys/init

  Code: 400. Errors:

  * error fetching default policy from store: failed to read policy: decryption failed: cipher: message authentication failed

I'm not sure what that means other than some pieces of the core/ folder are mandatory if this is going to work.


We haven't yet considered an API-driven approach because we're not willing to essentially re-write what vault-migrator already does: take the files from the Consul backup and restore them back one by one.

Thank you for the information about Enterprise features.  I think we'd put in the effort to upgrade our Vault before going down the path of an enterprise deployment.  But we'll keep it in mind!

Thanks again for your response!

[Jim Kalafut]

Hi John,

Can you explain why you're running "vault init" after the restore? Your Vault has already been initialized. The migrator tool you're using should be copying all of the structures under the specified source path to the backup and back again. You should be able to simply point Vault's configured storage to the restore location and start/unseal Vault.

FYI, a migration command was recently introduced: https://www.vaultproject.io/docs/commands/operator/migrate.html

That command is very similar to the migrator tool you're using, moving offline/encrypted data between storage backends. 

Regards

Jim

[John Ruiz]

Hello Jim!

I've tried that and it doesn't work.  We have an HA cluster, which means that the cluster knows about its members so that leaders can be elected in the case of failure.  When I do nothing more than migrate from Prod Consul Cluster --> File Backup --> Restored Consul Cluster and then point my Vault cluster to the Restored Consul Cluster... I can't unseal the Vault cluster because there's never a leader because (I believe but might be wrong) the electors are attempting to reach out to nodes on the Prod Consul Cluster, which aren't there.

Does this make sense, or have I made the water even muddier?  :-)

Thanks for the pointer to the migration command.  Unfortunately, we are stuck on 0.7.3 for the foreseeable future.

[Jim Kalafut]

Hi John,

After a full restore like you're doing, you will need to delete core/lock from the destination Consul K/V for Vault. This will allow one of the nodes to become leader.

Please give that a try if you've not already done so.

Thanks,

Jim

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/33dec5ae-5dff-4e3a-b48f-b8aed8b1d696%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Ruiz]

consul kv delete -recurse vault/core/lock

This is the magic that was missing from my life that made everything work!  Thank you so much!