Vault-agent not renewing database secret

[zxcvb...@gmail.com]

I've been experimenting with the vault agent and I'm certain that database secrets I'm obtaining are not being renewed.

The vault agent is successful at doing an AWS IAM auth, and it is successful at obtaining and caching a secret from the database engine.

The problem with the token expiring might be a misunderstanding on my part - currently I've defined my DB policies with both a ttl and maxttl of 86400 - the idea is to ensure that tokens are renewed at least once a day, however I may have mis-interpreted how these values are used. 

[Nick Cabatoff]

I suspect you're encountering the same issue that's described in https://github.com/hashicorp/vault/issues/6467 with Consul secrets: when you obtain a secret via Agent with auto_auth, when the Agent's auto_auth token is revoked (upon reaching max_ttl), your secrets will be revoked as well.  You should either increase your auto_auth method's max_ttl or make it a periodic token.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/0d52be5a-4a98-488c-9ca6-f7fd3e44913e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[zxcvb...@gmail.com]

Nick - thanks for the link, there was a lot of good information in that issue. However, I'm not clear how max_ttl and period interact with the max_lease_ttl value set in the hcl configuration. Also the documentation doesn't state specifically but I'm guessing that if both period and max_ttl are set, then the token can be renewed each period until it reaches maxage, is that right?

On Monday, April 1, 2019 at 7:58:35 AM UTC-4, Nick Cabatoff wrote:

I suspect you're encountering the same issue that's described in https://github.com/hashicorp/vault/issues/6467 with Consul secrets: when you obtain a secret via Agent with auto_auth, when the Agent's auto_auth token is revoked (upon reaching max_ttl), your secrets will be revoked as well.  You should either increase your auto_auth method's max_ttl or make it a periodic token.

On Sat, Mar 30, 2019 at 3:24 PM <zxcvb...@gmail.com> wrote:

I've been experimenting with the vault agent and I'm certain that database secrets I'm obtaining are not being renewed.

The vault agent is successful at doing an AWS IAM auth, and it is successful at obtaining and caching a secret from the database engine.

The problem with the token expiring might be a misunderstanding on my part - currently I've defined my DB policies with both a ttl and maxttl of 86400 - the idea is to ensure that tokens are renewed at least once a day, however I may have mis-interpreted how these values are used. 

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.

To unsubscribe from this group and stop receiving emails from it, send an email to vault...@googlegroups.com.

[Nick Cabatoff]

The token isn't renewed once it reaches maxttl; maxttl really means max.  At that point the token is revoked and Agent re-authenticates to get a new one.  When the token is revoked, so are all the leases that were created using that token.  I understand your confusion, it does seem natural that if Agent can get a new token it should be able to recreate all the secrets that expired with the old one, but that's now how it and Vault are designed.  To achieve what you want, you must increase your auto_auth token's max_ttl or make it periodic.

max_lease_ttl sets a ceiling to how big a max_ttl you can configure on any individual lease.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/4cf27128-71ab-4fc5-ab01-99b858b1d90c%40googlegroups.com.

[zxcvb...@gmail.com]

Nick - ok I've changed the auth role to be periodic and max_ttl on the database secret to be 0. I believe that max_ttl of 0 means no expiration, so it will exist until its parent token from auth expires. Next day in the logs I can see the auth token being renewed but no mention of the database secret and the later has expired. 

I double checked that the max_ttl for the database role is 0, and when I view the new database secret (obtained via vault agent and I see the same cached result if I ask for it multiple times) I see lease_duration of 24h (from default ttl) and lease_renewable is true.

Is there anything else I can check? Maybe some small detail I've overlooked?

[Nick Cabatoff]

Can we confirm that the changes to the auth role have been applied successfully?  If you lookup the auth token used by the agent, what's it's creation time?  Vault reports the creation time in epoch seconds, you can use something like https://www.epochconverter.com/ to get a human-readable value.

You don't need to set the database secret max_ttl to 0, it's the periodic token that gets a max_ttl of 0 (though that happens automatically by virtue of making it periodic.)  It's okay - and probably desirable - for your database secret to expire and be recreated, and Agent will handle that.  What we don't want is for the Agent token to expire.

To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/9458f1b0-bed7-4f2e-b4f3-875f15364263%40googlegroups.com.

[zxcvb...@gmail.com]

Nick - reason for max_ttl 0 is some of the applications I'm using are closed source commercial products and they are probably never going to have vault integration, so the database username/password I get from the secrets engine can't change for as long as the auth token exists (which is the lifetime of the AWS instances the applications run on). Overnight I found this article https://learn.hashicorp.com/vault/secrets-management/agent-caching and noticed they are giving permissions to sys/leases/* and auth/token/*. Nothing in the logs suggests there are any permission issues with renewing leases (or that an attempt to renew is being made at all) and nowhere in the documentation I've seen does it mention any considerations for vault auto-renew, but I'll try adding those policies and see if I get a different result. 

[zxcvb...@gmail.com]

Ok, I made some progress - you need to pass "-log-level=debug" as command line parameter to see lease renewal activity. I set ttl to 60 seconds to I don't want to wait to see results of my changes. It appears that both with and without the sys/leases/* and auth/token/* entries from the article, the renewal is working (without the username or password in database secret changing) with auth being periodic and secret having a max_ttl of 0. I'm going to leave the ttl at 60 for a day to see if there is anything else coming into play at the 24 hour mark. Otherwise I'm going guess that the first time I tried max_ttl 0 I didn't update the policy in vault so that is why it failed.