Unseal using recovery keys after lost GCP KMS unseal key

[Geir Emblemsvåg]

Hi,

We are practicing a scenario where the Google KMS key we use for unsealing is lost, and need to unseal the database using just the recovery keys.  But we are unable to do this using the migration or recovery options in the Vault documentation.

After deleting the KMS keys (simulating a total data loss), we update the unseal stanza with the "disabled" option and restart Vault for the change to take effect:

  seal "gcpckms" {

    disabled    = "true"
    project     = "vault-project"
    region      = "europe-west4"
    key_ring    = "vault-keyring"
    crypto_key  = "vault-key"
    # Credentials provided by $GOOGLE_APPLICATION_CREDENTIALS.
  }

We then use the "operator unseal -migrate" command to unseal the Vault. This allows us to input a quorum of recovery keys.  But after the final key is entered, unsealing fails because the KMS key is missing.  We thought that the recovery keys were an alternative to the KMS key, but is this understanding wrong? If not, could you help us with the recovery steps?  Or is there another way to recover the Vault if the KMS unseal key is lost?

Thank you,

Geir

[Chris Hoffman]

The recovery keys can not be used for unsealing and only the KMS key can be used.  The recovery keys are used for operations that require a quorum of users for highly privileged operations, such as generating a root token.

Chris

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/683f4ab1-13c9-4d12-ac53-073cdbb5b4d0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Francesco Ciocchetti]

I was looking for a solution to the same scenario.

So is it fair to say that if the KMS key is lost there is nothing that can be done to restore the encryption key ?

My understanding is that when auto-unseal with KMS is enabled the encryption key is encrypted with KMS and stored in the storage backend rather than being generated from the Shamir set of Unseal Keys.

Should it not be possible to have a backup procedure where a Shamir split set of keys could recover the encryption key ? 

What is the suggested workflow here other than never loose the KMS ? 

While I agree on principle that is not a good enough DR Scenario for us ... the cloud provider could mess up and loose that data. 

On Monday, June 3, 2019 at 2:02:19 PM UTC+2, Chris Hoffman wrote:

The recovery keys can not be used for unsealing and only the KMS key can be used.  The recovery keys are used for operations that require a quorum of users for highly privileged operations, such as generating a root token.

Chris

On Sat, Jun 1, 2019 at 4:15 AM Geir Emblemsvåg <gei...@gmail.com> wrote:

Hi,

We are practicing a scenario where the Google KMS key we use for unsealing is lost, and need to unseal the database using just the recovery keys.  But we are unable to do this using the migration or recovery options in the Vault documentation.

After deleting the KMS keys (simulating a total data loss), we update the unseal stanza with the "disabled" option and restart Vault for the change to take effect:

  seal "gcpckms" {

    disabled    = "true"
    project     = "vault-project"
    region      = "europe-west4"
    key_ring    = "vault-keyring"
    crypto_key  = "vault-key"
    # Credentials provided by $GOOGLE_APPLICATION_CREDENTIALS.
  }

We then use the "operator unseal -migrate" command to unseal the Vault. This allows us to input a quorum of recovery keys.  But after the final key is entered, unsealing fails because the KMS key is missing.  We thought that the recovery keys were an alternative to the KMS key, but is this understanding wrong? If not, could you help us with the recovery steps?  Or is there another way to recover the Vault if the KMS unseal key is lost?

Thank you,

Geir

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Chris Hoffman]

When using a seal, you are defining the root of trust for Vault.  Most, if not all, cloud providers allow you to bring your own key and some may provide key escrow too which should allow for a backup of your seal key.  With all DR and recovery scenarios, you must have backups and test them, which should include any hard dependencies, like your KMS keys.

Thanks,

Chris

Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
Registered in England: Number 1475918. | VAT Number: GB 232 9342 72

Contact details for our other offices can be found at http://www.mintel.com/office-locations.

This email and any attachments may include content that is confidential, privileged
or otherwise protected under applicable law. Unauthorised disclosure, copying, distribution
or use of the contents is prohibited and may be unlawful. If you have received this email in error,
including without appropriate authorisation, then please reply to the sender about the error
and delete this email and any attachments.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/3c001f5b-0d80-4728-a5ae-f83bf8c2b36f%40googlegroups.com.

[Francesco Ciocchetti]

Thanks, 

yeah i was looking into Importing the key , unforuntately the Gcloud feature is still alpha and a bit limited. (https://cloud.google.com/kms/docs/faq) so for now i have reverted to letting my vault operator ( https://github.com/banzaicloud/bank-vaults/ ) unseal the vault so i can handle the unseal-keys and distribute them to multiple parties for DR purposes.

With that and a backup ( I am using vault operator migrate ) i can successfully startup a vault with data from the bucket and unseal it 

[Jeff Mitchell]

Just to add something for those reading that are Enterprise customers:

If you are using DR replication in Vault Enterprise, each DR cluster uses its own independent seal. It can be the same KMS key but doesn't have to be -- it can even be a different cloud's KMS service, or an HSM. This gives you some protection against a single KMS key being lost.

Best,

Jeff

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/ee20f475-6b91-4aee-b4cd-2d4a4f33863b%40googlegroups.com.

[That guy]

Hi Jeff,

Could you please provide us with some links where it states that  "each DR cluster uses its own independent seal"? For now, I have checked the following ones:

https://learn.hashicorp.com/vault/day-one/ops-disaster-recovery

https://www.hashicorp.com/products/vault/enterprise

However, I didn't find such info. The question about losing access to a sole GCP Cloud KMS key is worrying us.

Thank you,

Alex