Secret leases and renewals

[Warren Gray]

I'm a little confused about how leases work with regard to secrets. I understand that while clients *may* have a lease associated with them, secrets *must* have a least. My question is who is responsible for renewing the leases on a secret and when this should happen. I'm attempting to renew a lease on my test server using the following commands:

wgray@vault-test:~$ vault write secret/foo value=bar

Success! Data written to: secret/foo

wgray@vault-test:~$ vault read secret/foo

Key           Value

lease_id       secret/foo/9b0b20e4-e193-d01c-62e2-3265cfa63133

lease_duration 2592000

value         bar

wgray@vault-test:~$ vault renew secret/foo/9b0b20e4-e193-d01c-62e2-3265cfa63133

Renew error: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/sys/renew/secret/foo/9b0b20e4-e193-d01c-62e2-3265cfa63133

Code: 400. Errors:

* lease is not renewable

Am I doing something wrong?

[Armon Dadgar]

Hey Warren,

I think this is a bug in our CLI output. Each lease has another property which is “renewable”, that the CLI is not outputting.

Depending on the secret backend, a lease may or may not be renewable. A non-renewable lease always expires at the

end of the lease, and the lease ID can be used to revoke it early (if access is no longer needed). A renewable lease can

typically be extended either indefinitely or up to some much larger limit (the final renew returns non-renewable).

The reason for this is to allow reasoning about key-rolling periods. In this case, suppose you set a value for “secret/foo” and a client

reads it (with 30 day lease, or customized used lease=…). If you change the value of “secret/foo”, you know that all clients will

update their copy within the upper bound of the lease period. This lets you confidently roll the key in 30 days, or whatever the lease= was

set to.

Some backends are more sophisticated, and can update policies under the hood. Most often the dynamic secrets are the

ones able to do this, since they can just talk to the backend system and update the policies without forcing the client to

re-read the secret.

I’ve opened a ticket to track the enhancement here:

https://github.com/hashicorp/vault/issues/116

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To post to this group, send email to vault...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d3bc0a0c-804a-4f12-8c8f-149e4a4fe84e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Warren Gray]

I'm using the 'file' backend for my test server. How do I set the renewable flag on my secret?

To clarify, if I have a secret that I don't expect to change, when and who should renew it? I haven't dug too deeply into the code (I'm new to Go), but it sounds like you're implying that the renew operation is client-based. Can you explain a little bit more about the lifecycle of a secret when the client requests it? I'm also interested in what effect a "renew" has on the server side.

I'm really liking the product so far! Just trying to wrap my head around everything.

Thanks for the prompt response, by the way.

- Warren

[Armon Dadgar]

Hey Warren,

There are a lot of moving pieces to Vault, so it does take a while to get the full context.

The leasing contract is between Vault and the client, the storage backend (file/consul) are not involved at all.

When Vault returns a secret to the client, it returns a lease, which has {ID, Renewable, Duration}.

Vault will automatically revoke the secret at the end of the duration, unless it is renewable AND the client performs

an explicit renew (using the Lease ID). When a renew is done, Vault again returns {ID, Renewable, Duration} with

the same semantics.

If a lease is not renewable, then the client must re-read the secret. Instead of renewing an existing secret, performing

the read again allows Vault to return a new secret value with a different lease.

The renewable flag is not controllable, as it depends on the implementation of the secret backend.

In the case of “secret/“ it is a generic backend: https://vaultproject.io/docs/secrets/generic/index.html

An example of a backend that does support renewable leases is the PostgreSQL backend:

https://vaultproject.io/docs/secrets/postgresql/index.html

The actually effect of a renew server side is that it pushes back the expiration time of the lease.

This prevents the automatic revocation from kicking in.

Hope that helps!

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0fa3fff3-f3eb-4be8-bc9d-3ea34ea265c6%40googlegroups.com.

[Warren Gray]

So to confirm I'm understanding things correctly:

For a 'generic' backend: Vault always gives back the current secret (as decrypted from disk) when clients perform a read, along with a new lease ID.

For a backend that supports renewable leases: 

If there exists a non-revoked secret (i.e. a secret with a valid lease), 'vault read' will return the value of the secret associated with that lease and the existing lease ID.

If there is no valid lease (i.e. it has either expired or has been manually revoked), 'vault read' will behave the same as it does for a non-renewable lease, re-reading the secret from the backend.

Is this correct?

- Warren

On Thursday, April 30, 2015 at 1:29:06 PM UTC-4, Armon Dadgar wrote:

Hey Warren,

[Armon Dadgar]

Hey Warren,

Your understanding of the “generic” backend is correct, it effectively just does passthrough.

The “vault read” semantics is actually simpler that that. A read operation is always independent

of any previous leases. So multiple reads against the same key will return a new tuple of {Secret, Lease}.

A client may have multiple leases outstanding at the same time. This is by design to allow overlaps

to prevent downtime.

As an example, support key “foo/bar” returns a secret that is leased for 1 hour and is non renewable.

A client may do:

1) At T0, Read “foo/bar”, get Secret1 and Lease1 valid until T3600

2) At T1800, Read “foo/bar”, get Secret2 and Lease2 valid until T5400

3) Switch to using S2 at T1800

4) At T3600, Secret1 is revoked, but client has already transitioned to Secret2.

5) At T3600, Read “foo/bar”, get Secret3 and Lease3 valid until T7200

6) Switch to using S3 at T3600

7) ...

The overlapping leases are critical to allow a client to ensure a continuity of access in the case

of non-renewable leasing.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/83d9b37e-f2fa-42b1-b1c1-4efba9debaa9%40googlegroups.com.

[Warren Gray]

So the lease time is a guarantee by the Vault server that a credential will be valid for a given amount of time (unless revoked) and it's the responsibility of the client to renew or re-read the secret after that?

In step 3 of your reply above, "switch to using S2" implies that the consumer is updating it's internal state to replace S1 with S2, correct? 

If a renew operation is successful, then the consumer knows it can continue to use that secret. otherwise it should re-read.

Thanks,

- Warren

On Thursday, April 30, 2015 at 2:21:54 PM UTC-4, Armon Dadgar wrote:

Hey Warren,

[Armon Dadgar]

Hey Warren,

That is exactly correct!

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/03907954-5762-4510-bf47-69d90d1ed057%40googlegroups.com.

[Julian Borrey]

So just clarifying to be absolutely sure - the lease is purely for the client's use.

The server, regardless of the lease time, will always output the same value for the same key (until overwritten or deleted of course).

I could have a lease of 60 seconds on a secret, but request 1 hour after inputting the secret into the server. The secret will remain the same, still be available for retrieval and the client will have information which says that this secret is only valid for 60 seconds.

Correct?

On Thursday, April 30, 2015 at 11:54:51 AM UTC-7, Armon Dadgar wrote:

Hey Warren,

[Armon Dadgar]

Julien,

That is correct for the “generic” backend only. Any backend that is doing dynamic secret generation

will behave totally differently. They will hard revoke the credentials at the end of the lease, and each read

will generate a new response with different values.

The generic backend does not know the semantics of what it is storing, so it has a limited ability to do richer

things. Clients should still respect the lease as it allows you to reason about key rotation. e.g. if the lease is 60

seconds, I can swap the value and *know* that within 60 additional seconds +/- a grace period, all my clients

will switch to the new value. This is a critical promise between clients and Vault as it restores some sanity

to InfoSec folks and operators.

Best Regards,

Armon Dadgar

[russell....@sas.com]

Hi,

We're in the process of creating a credential management system that manages credentials by "domain" and I'm investigating what it would take to use Vault for our purposes.  My concern is the lease expiration for secrets I load into the vault (https://vaultproject.io/docs/concepts/lease.html). "All secrets in Vault are required to have a lease. Even if the data is meant to be valid for eternity, a lease is required to force the consumer to check in routinely.", is what I have questions about.  Our data is intended to be "valid for eternity"; or, at lease, valid until we decide to delete / update it.  We would use the "generic" secret backend with the "consul" storage backend.  I've created some test secrets and notice a lease duration of 2592000 (30 days) and lease_renewable set to "false".  

I've read this thread a couple of times and am still confused.  Using this secret-storage combination and what would happen if I attempted to read these test secrets after (say) 31 days?  Based on what I've read the "read" would be successful and the value returned.

Regards,

Russell

On Thursday, July 30, 2015 at 1:38:55 PM UTC-4, Armon Dadgar wrote:

Julien,

That is correct for the “generic” backend only. Any backend that is doing dynamic secret generation

will behave totally differently. They will hard revoke the credentials at the end of the lease, and each read

will generate a new response with different values.

The generic backend does not know the semantics of what it is storing, so it has a limited ability to do richer

things. Clients should still respect the lease as it allows you to reason about key rotation. e.g. if the lease is 60

seconds, I can swap the value and *know* that within 60 additional seconds +/- a grace period, all my clients

will switch to the new value. This is a critical promise between clients and Vault as it restores some sanity

to InfoSec folks and operators.

Best Regards,

Armon Dadgar

[Warren Gray]

The thing that I got hung up on was that a lease is assigned when a secret is read from Vault. When using the generic backend, leases serve a lesser purpose because the actual secret is never revoked (even if the lease expires). You can continue to read a secret from Vault, regardless of your lease status, until somebody deletes the secret itself from Vault. There's really no need to renew leases when using the generic backend unless you're rotating secrets. 

As an example, consider the MySQL backend: When you "read" a secret from Vault under MySQL, Vault actually calls out to the MySQL server and generates credentials for you. The "lease" that comes with these credentials is a period of time that Vault is guaranteeing the credentials will be valid for. To keep the credentials for longer than this period, the consumer of the credentials must "renew" the lease. If the lease is not renewed before the lease duration is up, Vault expires the lease and deletes the credentials from MySQL (thus invalidating them). In the generic backend, there's nothing to delete, so the secrets remain in Vault and can be re-read (which will generate a new lease).

Essentially, secrets stored in the generic backend will exist indefinitely, until they are deleted by an API call.

One note: if you continually use the "read" API, you'll end up with a *huge* amount of metadata stored in your backend. This is because each read creates a new lease, and the default lease duration is 30 days. Ideally, you should "renew" the lease using the token provided during the initial read until you cannot renew it (at which point you should re-read the secret). My solution to this problem was to write each secret with a "lease" value of "1s". This causes the lease to expire immediately and keeps the Vault metadata minimal. There's also a fix in the latest branch that makes the default lease duration configurable.

Hope this helps,

- Warren

[Michael Fischer]

I wonder whether most of the confusion around leases could be cleared up by calling leases TTLs instead.  Most folks are already familiar with what this means with respect to DNS, and the behavior and expectations associated with DNS TTLs seem similar for vault leases.

--Michael

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---

You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/5012f0e1-6ed5-4349-9270-edaa4186dd93%40googlegroups.com.

[russell....@sas.com]

Thanks, Warren.  Your statement in bold and the paragraph that follows are most helpful. Our model is relatively simple - an application with the appropriate authorization calls a REST api to get username-password.  We'd like to use the Vault with the generic backend to store secrets -none of our layers are lease aware.  The leases (or my lack of understanding of them in the generic case) really muddy the waters.  After reading your statements about the "huge" amount of metadata made me poke around Consul: sure enough, I do another READ and another entry shows up in the VAULT/SYS/EXPIRE/ID/SECRET/ tree!  We definitely should use your 1s suggestion.

With my new found understanding, I'll reread this and the other threads which are lease related.

Regards,

Russell

[Mic Le]

Hi Guys, 

I am trying to renew my token and I get the same error as above (lease is not renewable) 

I want to make sure my command is correct:

vault token-renew  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  3600

Please advice.

[Armon Dadgar]

Hey Mic,

You may want to start a new thread, as this isn’t related to the same topic.

How did you get that token? Not all tokens are renewable, and when it is generated

the output will indicate as such.

Best Regards,

Armon Dadgar

From: Mic Le <big...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: September 8, 2015 at 9:35:21 AM
To: Vault <vault...@googlegroups.com>>

Cc: julian...@gmail.com <julian...@gmail.com>>, w.gray...@gmail.com <w.gray...@gmail.com>>
Subject:  [vault] Re: Secret leases and renewals

Hi Guys, 

I am trying to renew my token and I get the same error as above (lease is not renewable) 

I want to make sure my command is correct:

vault token-renew  xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  3600

Please advice.

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---

You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0b60af15-502f-4cdc-b9ee-5fb917d73836%40googlegroups.com.

[Mic Le]

Hi Armon,

I have started the new thread here

Thanks !