vault unseal failing with error code 400

[Saurabh Singh]

Hi,

I have 2 vaults, let's  say vault1 and vault2 both are running as docker container.

I am using vault2 for unsealing vault1.(transit mechanism for unseal) I followed the doc for unsealing  vault by transit mechanism and I was able to unseal vault1 by transit.

After that I killed vault2 and restarted it again and I followed all the steps I did initially for creating token that will be used by vault1 for unsealing. I got the token from vault2.

I copied this token to seal section of config on vault1.

"seal": {

                "transit": {

                        "address": "http://127.0.0.1:8100",

                        "token": "s.lwLrO6EafHmrN6wEEw3yDIq6", #### new token by vault2

                        "disable_renewal":"false",

                        "key_name": "autounseal",

                        "mount_path": "transit/",

                        "tls_skip_verify": "true"

                }

        },

I restarted vault1 after this, but now vault1 is failing with error:-

zvault_1         | 2019-07-27T09:11:13.180Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set

zvault_1         | 2019-07-27T09:11:13.219Z [INFO]  core: stored unseal keys supported, attempting fetch

zvault_1         | 2019-07-27T09:11:13.221Z [WARN]  failed to unseal core: error="fetching stored unseal keys failed: failed to decrypt encrypted stored keys: Error making API request.

zvault_1         |

zvault_1         | URL: PUT http://127.0.0.1:8100/v1/transit/decrypt/autounseal

zvault_1         | Code: 400. Errors:

zvault_1         |

zvault_1         | * invalid ciphertext: unable to decrypt"

Thanks and regards

Saurabh

[Saurabh Singh]

Hi,

After debugging I have understood few points and have some doubts.

Explanation :

I am using vault1 and vault2, vault2 is used for unsealing of vault1. Vault2 generated a token and gave it to vault1, vault1 used this token for unsealing and vault1 was able to unseal.

Now I killed vault2 and restarted it again. I did all the steps for creating the token, I got the new token and I gave it to vault1 in it's config. After this I restarted vault1 now vault1 is not able to unseal and it is failing with this error:

 invalid ciphertext: unable to decrypt"

Reason what I am thinking is: vault1 created some token entry in it's backend with old token and now config for vault2 has changed but vault1 is still trying to unseal using the old token.

My question is : is there a way to delete the initial token and tell vault1 to use the latest token  from the config that vault2 has generated?

[Michel Vocks]

Hi Saurabh,

when your Vault instance (vault1) unseals itself with the second Vault instance (vault2) it sends your unseal keys to the transit endpoint from your second Vault instance (vault2).

These keys will be stored encrypted at your initital Vault instance (vault1). The next time Vault (vault1) tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your second Vault instance (vault2).

This will not work since you transit key has been changed and the new key cannot be used to decrypt the unseal keys.

My question is : is there a way to delete the initial token and tell vault1 to use the latest token  from the config that vault2 has generated?

This is not possible. However, if you want to simply rotate your keys you can use the key rotation functionality from the transit engine: https://www.vaultproject.io/docs/configuration/seal/transit.html#key-rotation

Cheers,

Michel

[Saurabh Singh]

Thank you Michel.

[rohan nikhal]

Hi Saurabh, 

I am in a similar situation as your.

Our token to unseal vault 1 was expired hence we went ahead and created a new key and a token on Vault 2 and entered the new token on Vault 1 after which we are facing the same issue.

Would you share the steps upon what you did to get it working?

Thanks,

Rohan

On Saturday, July 27, 2019 at 2:45:30 PM UTC+5:30, Saurabh Singh wrote:

[Nick Cabatoff]

Hi Rohan,

Let me see if I understand your problem:

• vault2 is a Vault cluster running transit
• vault1 is configured to auto-unseal using vault2 with a transit key (let's call it "key1")
• the token vault2 uses to talk to vault1 expired

So to address this you created a new transit key (say "key2") in vault1 and a new token and modified your vault2 setup to use those.

Why did you create a new transit key ("key2")?  That's most likely your issue - not its creation, but I'm assuming you also modified vault1's seal config to reference key2, which is why you're getting "unable to decrypt", because vault1's seal is encrypted with key1.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0f3bbc25-89cf-4e16-b460-8ed76f32f3ee%40googlegroups.com.

[rohan nikhal]

That's right, I am new to the tool and did not find much on how to renew the expired token hence went ahead with creating the new Key and new token. 

Is there a way we can get this fixed? I do have the old token that I provided vault 1 in the config.hcl. 

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Nick Cabatoff]

Just revert your change to vault2's config so that it continues to use the old key but with the new token you created.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/852bb315-b0af-4b71-88d7-3ea449db092f%40googlegroups.com.

[Nick Cabatoff]

Sorry, I meant vault1.

[Saurabh Singh]

Hi Rohan,

First time when vault1 unseals itself with vault2 it sends your unseal keys to the transit endpoint.
Unseal key are stored encrypted at your vault1. Second time when vault1 tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your vault2. (described by @mic...@hashicorp.com  in the above mail)

Your vault1's unseal is failing because you are using token created with the vault2's new key, on vault2 Create a token with old key and use that token in vault1.

Next add a timer which will refresh your token before it expires.

For my case:
I was just playing around vault, so I deleted my vault1's backend and restarted it.(This was just for test and dev mode)

Thanks and regards

Saurabh

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0f3bbc25-89cf-4e16-b460-8ed76f32f3ee%40googlegroups.com.

[rohan nikhal]

This was my original config file on vault 1

I had the transit secret "autounseal" created with policy as mentioned by Hashicorp.

listener "tcp" {

  address          = "0.0.0.0:8200"

  cluster_address  = "xx.xx.xx.xx:8201"

  tls_disable      = "true"

}

ui = true

seal "transit" {

        address= "http://vault02.abc.com:8200"

        token= "s.LMZCExk4MxMe5W6bPlWvetPJ"

        disable_renewal = "false"

        key_name = "autounseal"

        mount_path = "transit/"

        tls_skip_verify = "true"

}

storage "consul" {

  address = "127.0.0.1:8500"

  path    = "vault/"

}

api_addr = "http://xx.xx.xx.xx:8200"

cluster_addr = "https://xx.xx.xx.xx:8201"

Now the token "s.LMZCExk4MxMe5W6bPlWvetPJ" got expired hence we deleted the transit key "autounseal" and created a new one which returned us a new token.

After which I changed the token value in Vault config as below:

listener "tcp" {

  address          = "0.0.0.0:8200"

  cluster_address  = "xx.xx.xx.xx:8201"

  tls_disable      = "true"

}

ui = true

seal "transit" {

        address= "http://vault02.abc.com:8200"

        token= "s.AhsgjlmvenjwWtrwHT"

        disable_renewal = "false"

        key_name = "autounseal"

        mount_path = "transit/"

        tls_skip_verify = "true"

}

storage "consul" {

  address = "127.0.0.1:8500"

  path    = "vault/"

}

api_addr = "http://xx.xx.xx.xx:8200"

cluster_addr = "https://xx.xx.xx.xx:8201"

After this is when I receive the error

[WARN]  failed to unseal core: error="fetching stored unseal keys failed: failed to decrypt encrypted stored keys: Error making API request.

URL: PUT http://127.0.0.1:8100/v1/transit/decrypt/autounseal

Code: 400. Errors:

* invalid ciphertext: unable to decrypt"

As mentioned by you i tried to change the token in the config file of vault 1 as it was earlier however that gives me a different error:

Started Vault.

Error parsing Seal configuration: Error making API request.

URL: PUT http://vault01p-04.principal.com:8200/v1/transit/encrypt/autounseal

Code: 403. Errors:

* permission denied

vault.service: main process exited, code=exited, status=1/FAILURE

Unit vault.service entered failed state.

vault.service failed.

Is there a way I can get going from here?

Sorry, I meant vault1.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/852bb315-b0af-4b71-88d7-3ea449db092f%40googlegroups.com.

[rohan nikhal]

[WARN]  failed to unseal core: error="fetching stored unseal keys failed: failed to decrypt encrypted stored keys: Error making API request.

URL: PUT http://127.0.0.1:8100/v1/transit/decrypt/autounseal

Code: 400. Errors:

* invalid ciphertext: unable to decrypt"

As mentioned by you i tried to change the token in the config file of vault 1 as it was earlier however that gives me a different error:

Started Vault.

Error parsing Seal configuration: Error making API request.

URL: PUT http://vault01p-04.principal.com:8200/v1/transit/encrypt/autounseal

Code: 403. Errors:

* permission denied

vault.service: main process exited, code=exited, status=1/FAILURE

Unit vault.service entered failed state.

vault.service failed.

Is there a way I can get going from here?

On Thursday, October 3, 2019 at 7:42:01 PM UTC+5:30, Saurabh Singh wrote:

Hi Rohan,

First time when vault1 unseals itself with vault2 it sends your unseal keys to the transit endpoint.

Unseal key are stored encrypted at your vault1. Second time when vault1 tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your vault2. (described by ...@michel@hashicorp.com  in the above mail)

Saurabh

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Nick Cabatoff]

It sounds like the new token you created doesn't have permission to read the transit key.  Make sure that it's created with a policy that does give it that permission.

Hi Rohan,

Unseal key are stored encrypted at your vault1. Second time when vault1 tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your vault2. (described by ...@mic...@hashicorp.com  in the above mail)

Saurabh

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/16097be0-8227-4c48-a3f1-b71d5debc0f8%40googlegroups.com.

[rohan nikhal]

Hi Saurabh,

Can you help me with the command for creating a new token on vault 2 using the old key on Vault 2. I have currently requested to restore my system to a backdate which will take the system where we had the old Key on vault 2.

Also is there any documentation on how can we set a timer to refresh the token say after every 25 days.

Thank you for all your help.

On Thursday, October 3, 2019 at 7:42:01 PM UTC+5:30, Saurabh Singh wrote:

Hi Rohan,

First time when vault1 unseals itself with vault2 it sends your unseal keys to the transit endpoint.

Unseal key are stored encrypted at your vault1. Second time when vault1 tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your vault2. (described by ...@michel@hashicorp.com  in the above mail)

Saurabh

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Saurabh Singh]

Hi,

You will need two things:
1) old key
2) old policy (created using old key). Let's call it autounseal policy.

1. Create a client token with autounseal policy attached, this token will be used by vault1 for unsealing.

# vault token create -address=http://127.0.0.1:8200 -policy="autounseal" -wrap-ttl=2120 2. Unwrap the token generated in step1. # VAULT_TOKEN="s.eH0lMqWb8NIydWj2FK1ZjhBE" vault unwrap -address=http://127.0.0.1:8200

3. Use the unwrapped token from step 2 and add it in the seal config of vault1.

For refreshing your token:
It depends upon how you want to do:
1) if you want execute script or command then use cron job(linux).
    token refresh  command: 

      curl --header "X-Vault-Token:<your token?" --data '{"increment":"168h"}' http://127.0.0.1:8200/v1/auth/token/renew-self

2. every programming language has some functionality for timers.
I work on golang, so I will do something like this:
 step 1: start a go-routine when your service/process starts.
 step 2: start a timer in your go-routine.
func main() {
    go renewVaultToken(<some time in minutes>)
}
func renewVaultToken(tokenRenewalInterval uint64) {
        ticker := time.NewTicker(time.Duration(tokenRenewalInterval) * time.Minute)
        for range ticker.C {
                //call api for token renewal
                //curl --header "X-Vault-Token:<your token?" --data '{"increment":"168h"}' http://127.0.0.1:8200/v1/auth/token/renew-self
                
        }
}

On Fri, Oct 4, 2019 at 11:46 AM rohan nikhal <ronni...@gmail.com> wrote:

Hi Saurabh,

Can you help me with the command for creating a new token on vault 2 using the old key on Vault 2. I have currently requested to restore my system to a backdate which will take the system where we had the old Key on vault 2.

Also is there any documentation on how can we set a timer to refresh the token say after every 25 days.

Thank you for all your help.

On Thursday, October 3, 2019 at 7:42:01 PM UTC+5:30, Saurabh Singh wrote:

Hi Rohan,

First time when vault1 unseals itself with vault2 it sends your unseal keys to the transit endpoint.

Unseal key are stored encrypted at your vault1. Second time when vault1 tries to unseal itself, it detects the encrypted unseal keys locally and tries to decrypt them via the transit endpoint from your vault2. (described by ...@mic...@hashicorp.com  in the above mail)

Saurabh

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/979ef9c3-11dd-486e-9dda-f06ffa81b092%40googlegroups.com.