Vault/consul data backup and restore to another instance
1,281 views
Skip to first unread message
Martin Devlin
Oct 20, 2016, 5:51:01 PM10/20/16
to Vault
Hi,
I want to backup vault data from one environment (vault with consul storage backend) and restore it to another. Let's call them A and B.
Consul data is restored fine and the consul cluster starts up as expected. I have tried using both consulate and backup-consul for this.
I delete the vault/core/lock and vault/core/leader keys from B consul kv after the restore.
I generate a fresh (management) consul access token for vault on B, start vault on B and use the keys from A to unseal it.
So far, so good: vault unseals.
However it then enters into a loop of doom with the following error:
==> Vault server configuration:
Backend: consul (HA available)
Listener 1: tcp (addr: "192.168.253.84:8243", tls: "enabled")
Listener 2: tcp (addr: "192.168.253.84:8200", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Redirect Address: https://192.168.253.84:8243
Version: Vault v0.6.1
==> Vault server started! Log data will stream in below:
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
2016/10/20 18:17:34.808484 [INF] core: vault is unsealed
2016/10/20 18:17:34.808663 [WRN] physical/consul: Concurrent sealed state change notify dropped
2016/10/20 18:17:34.808762 [INF] core: entering standby mode
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
2016/10/20 18:17:34.881561 [INF] core: acquired lock, enabling active operation
2016/10/20 18:17:34.922255 [WRN] physical/consul: Concurrent state change notify dropped
2016/10/20 18:17:34.922280 [INF] core: post-unseal setup starting
2016/10/20 18:17:34.930221 [INF] core: successfully mounted backend type=generic path=secret/
2016/10/20 18:17:34.930282 [INF] core: successfully mounted backend type=cubbyhole path=cubbyhole/
2016/10/20 18:17:34.930493 [INF] core: successfully mounted backend type=system path=sys/
2016/10/20 18:17:34.930669 [INF] rollback: starting rollback manager
2016/10/20 18:17:34.942618 [INF] core: pre-seal teardown starting
2016/10/20 18:17:34.942647 [INF] rollback: stopping rollback manager
2016/10/20 18:17:34.942669 [INF] core: pre-seal teardown complete
2016/10/20 18:17:34.942713 [ERR] core: post-unseal setup failed error=expiration state restore failed: failed to read lease entry: decryption failed: cipher: message authentication failed
2016/10/20 18:17:34.983435 [INF] core: acquired lock, enabling active operation
2016/10/20 18:17:35.026885 [WRN] physical/consul: Concurrent state change notify dropped
2016/10/20 18:17:35.026914 [INF] core: post-unseal setup starting
2016/10/20 18:17:35.036136 [INF] core: successfully mounted backend type=generic path=secret/
2016/10/20 18:17:35.036167 [INF] core: successfully mounted backend type=cubbyhole path=cubbyhole/
2016/10/20 18:17:35.036241 [INF] core: successfully mounted backend type=system path=sys/
2016/10/20 18:17:35.036360 [INF] rollback: starting rollback manager
2016/10/20 18:17:35.060106 [INF] core: pre-seal teardown starting
2016/10/20 18:17:35.060141 [INF] rollback: stopping rollback manager
2016/10/20 18:17:35.060164 [INF] core: pre-seal teardown complete
2016/10/20 18:17:35.060187 [ERR] core: post-unseal setup failed error=expiration state restore failed: failed to read lease entry: decryption failed: cipher: message authentication failed
This continues and the vault server never enters active state.
Stepping through the responses from A (which works fine when I restore the same consul backup) I see it is failing when it gets to expires. Sure enough, if I also delete vault/sys/expire along with vault/core/lock and vault/core/leader keys before starting vault then it works, but all my existing vault tokens are now invalid in B.
What am I doing wrong? What is the recommended pattern to recover vault data to another datacentre? The consul cluster is 0.7.0 by the way.
Thanks,
Martin
I want to backup vault data from one environment (vault with consul storage backend) and restore it to another. Let's call them A and B.
Consul data is restored fine and the consul cluster starts up as expected. I have tried using both consulate and backup-consul for this.
I delete the vault/core/lock and vault/core/leader keys from B consul kv after the restore.
I generate a fresh (management) consul access token for vault on B, start vault on B and use the keys from A to unseal it.
So far, so good: vault unseals.
However it then enters into a loop of doom with the following error:
==> Vault server configuration:
Backend: consul (HA available)
Listener 1: tcp (addr: "192.168.253.84:8243", tls: "enabled")
Listener 2: tcp (addr: "192.168.253.84:8200", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: false
Redirect Address: https://192.168.253.84:8243
Version: Vault v0.6.1
==> Vault server started! Log data will stream in below:
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 1
Sealed: true
Key Shares: 5
Key Threshold: 3
Unseal Progress: 2
2016/10/20 18:17:34.808484 [INF] core: vault is unsealed
2016/10/20 18:17:34.808663 [WRN] physical/consul: Concurrent sealed state change notify dropped
2016/10/20 18:17:34.808762 [INF] core: entering standby mode
Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0
2016/10/20 18:17:34.881561 [INF] core: acquired lock, enabling active operation
2016/10/20 18:17:34.922255 [WRN] physical/consul: Concurrent state change notify dropped
2016/10/20 18:17:34.922280 [INF] core: post-unseal setup starting
2016/10/20 18:17:34.930221 [INF] core: successfully mounted backend type=generic path=secret/
2016/10/20 18:17:34.930282 [INF] core: successfully mounted backend type=cubbyhole path=cubbyhole/
2016/10/20 18:17:34.930493 [INF] core: successfully mounted backend type=system path=sys/
2016/10/20 18:17:34.930669 [INF] rollback: starting rollback manager
2016/10/20 18:17:34.942618 [INF] core: pre-seal teardown starting
2016/10/20 18:17:34.942647 [INF] rollback: stopping rollback manager
2016/10/20 18:17:34.942669 [INF] core: pre-seal teardown complete
2016/10/20 18:17:34.942713 [ERR] core: post-unseal setup failed error=expiration state restore failed: failed to read lease entry: decryption failed: cipher: message authentication failed
2016/10/20 18:17:34.983435 [INF] core: acquired lock, enabling active operation
2016/10/20 18:17:35.026885 [WRN] physical/consul: Concurrent state change notify dropped
2016/10/20 18:17:35.026914 [INF] core: post-unseal setup starting
2016/10/20 18:17:35.036136 [INF] core: successfully mounted backend type=generic path=secret/
2016/10/20 18:17:35.036167 [INF] core: successfully mounted backend type=cubbyhole path=cubbyhole/
2016/10/20 18:17:35.036241 [INF] core: successfully mounted backend type=system path=sys/
2016/10/20 18:17:35.036360 [INF] rollback: starting rollback manager
2016/10/20 18:17:35.060106 [INF] core: pre-seal teardown starting
2016/10/20 18:17:35.060141 [INF] rollback: stopping rollback manager
2016/10/20 18:17:35.060164 [INF] core: pre-seal teardown complete
2016/10/20 18:17:35.060187 [ERR] core: post-unseal setup failed error=expiration state restore failed: failed to read lease entry: decryption failed: cipher: message authentication failed
This continues and the vault server never enters active state.
Stepping through the responses from A (which works fine when I restore the same consul backup) I see it is failing when it gets to expires. Sure enough, if I also delete vault/sys/expire along with vault/core/lock and vault/core/leader keys before starting vault then it works, but all my existing vault tokens are now invalid in B.
What am I doing wrong? What is the recommended pattern to recover vault data to another datacentre? The consul cluster is 0.7.0 by the way.
Thanks,
Martin
Martin Devlin
Oct 20, 2016, 6:16:36 PM10/20/16
to Vault
It seems that vault cannot decrypt the data in the vault/sys/expire keys.
If I understand it correctly, unsealing B gives it the master key to decrypt the keyring and this then contains the keys to decrypt these values. It seems to be this bit that isn't happening.
Note that, with vault/sys/expire deleted before I start vault in B, all the other (logical/) keys from A are in place and can be read fine from vault.
I can also dump the kv from B straight from consul and it looks the same as the A one, apart from the ModifyIndex values.
If I understand it correctly, unsealing B gives it the master key to decrypt the keyring and this then contains the keys to decrypt these values. It seems to be this bit that isn't happening.
Note that, with vault/sys/expire deleted before I start vault in B, all the other (logical/) keys from A are in place and can be read fine from vault.
I can also dump the kv from B straight from consul and it looks the same as the A one, apart from the ModifyIndex values.
Jeff Mitchell
Oct 21, 2016, 7:10:14 AM10/21/16
to vault...@googlegroups.com
Hi Martin,
Sadly many Consul "backup" tools make a very incorrect assumption that any value stored in Consul is JSON safe and simply dump to JSON.
You need to use the -b flag with Consulate to force it to base64 values; see https://github.com/gmr/consulate/issues/71
Best,
Jeff
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/fe85031c-9b3e-4a6e-8d61-584a9f812475%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Martin Devlin
Oct 21, 2016, 11:48:59 AM10/21/16
to Vault
Jeff,
Thanks for the reply.
Yes I tried -b on consulate. Also, that project looks to be abandoned anyway.
It seems however that the tokens from A are in fact usable on B, but throw an error if you try to renew them:
{
"errors": [
"lease not found or lease is not renewable"
]
}
Martin
Thanks for the reply.
Yes I tried -b on consulate. Also, that project looks to be abandoned anyway.
It seems however that the tokens from A are in fact usable on B, but throw an error if you try to renew them:
{
"errors": [
"lease not found or lease is not renewable"
]
}
Martin
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
Jeff Mitchell
Oct 21, 2016, 12:27:01 PM10/21/16
to vault...@googlegroups.com
Just to check, you did -b *both* ways (during restore too)?
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/90efa5b1-37df-4d83-88ee-3a0b8fdd7953%40googlegroups.com.
Martin Devlin
Oct 21, 2016, 4:35:42 PM10/21/16
to Vault
Jeff,
Yeah I think so. The big issue with consulate though is it's not maintained hence me switching to consul-backup.
What I don't get is why the other keys can be decrypted but the leases can't. Surely they're all encrypted with the same key?
Jeff Mitchell
Oct 21, 2016, 5:20:39 PM10/21/16
to vault...@googlegroups.com
They are... I'm not sure either at the moment. I've never heard of this behavior.
--Jeff
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d42e1d70-bdcd-4fd8-a64c-1faddc025f78%40googlegroups.com.
Martin Devlin
Oct 22, 2016, 4:42:43 PM10/22/16
to Vault
Jeff,
Hey thanks- looks like it was switching to backup-consul that caused this: as you guessed my double-check using consulate had omitted the -b arg on the restore. Why that would affect the sys/expires only is another question but one I'm happy to ignore for now!
Unfortunately there's not much ongoing support for consulate so it looks like I'll be looking to forward-fix backup-consul instead.
Thanks again,
Martin
Hey thanks- looks like it was switching to backup-consul that caused this: as you guessed my double-check using consulate had omitted the -b arg on the restore. Why that would affect the sys/expires only is another question but one I'm happy to ignore for now!
Unfortunately there's not much ongoing support for consulate so it looks like I'll be looking to forward-fix backup-consul instead.
Thanks again,
Martin
On Friday, October 21, 2016 at 10:20:39 PM UTC+1, Jeff Mitchell wrote:
They are... I'm not sure either at the moment. I've never heard of this behavior.
--Jeff
On Oct 21, 2016 16:35, "Martin Devlin" <dev...@gmail.com> wrote:
Jeff,
Yeah I think so. The big issue with consulate though is it's not maintained hence me switching to consul-backup.
What I don't get is why the other keys can be decrypted but the leases can't. Surely they're all encrypted with the same key?
--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
Martin Devlin
Oct 23, 2016, 4:08:46 AM10/23/16
to Vault
Jeff, and anyone else still following this:
Apologies, my above statement that it works with consulate is false- it only appeared to as I was still removing vault/sys/expire before starting vault.
The base64 encoded values in backups from both products are the same and neither works for vault/sys/expire
Regards,
Martin
Apologies, my above statement that it works with consulate is false- it only appeared to as I was still removing vault/sys/expire before starting vault.
The base64 encoded values in backups from both products are the same and neither works for vault/sys/expire
Regards,
Martin
Reply all
Reply to author
Forward
0 new messages