Automated Vault token rotation

[SunSparc]

I have recently been looking into automating the rotation of vault tokens that are being used by many servers/services that I manage. Following are a couple of ideas that I have come up with so far.

1. Tokens as Secrets using consul-template (push)

Create an application/script that lives on (or near) the vault cluster that creates new tokens periodically and then stores them in Vault as Secrets. Then each server/service can be running consul-template to update a token file via a consul-template template. Depending on the service that is using the token, the service may need to be restarted, which can be accomplished through the consul-template "command" option. This application could use root permissions and be able to create tokens with any permissions and policies it needs to.

2. Service/server periodically requests a new token (pull)

Create an application/script that lives on each server (alongside each service) that periodically hits the Vault HTTP API to create a new token using the same permissions and policies of the token that it currently has. This application could acquire a new token, place it in a token file, and restart/reload whatever services that depend on a Vault token.

I am interested in feedback on these ideas as well as other ideas on this topic.

Regards,

Jonathan

[Paul Archer]

I think the answer may be simpler than what you are thinking.
The application has to authenticate to vault to get the token in the first place (say, by using a client certificate, user/password, or AppRole). The application should just re-authenticate. That should generate a new token. (You'll have to check to see if the old token is automatically invalidated, or if you will have to do that manually.)

Paul

[SunSparc]

On Thursday, June 15, 2017 at 11:59:05 AM UTC-6, Paul Archer wrote:

I think the answer may be simpler than what you are thinking.
The application has to authenticate to vault to get the token in the first place (say, by using a client certificate, user/password, or AppRole). The application should just re-authenticate. That should generate a new token. (You'll have to check to see if the old token is automatically invalidated, or if you will have to do that manually.)

The way that we are currently doing it is by server. We launch a server from a server provider and during the launch we provide a token in the user-data/startup script/init script (etc), that is then run by the node as it initializes on first boot. Part of that initialization is installing applications via apt-get and private package repositories. As the packages are installed and come online they look for the token and proceed.

We are still deciding on a final solution for automating the launch of servers. For now it is done by hand. We have been looking at Ansible or Terraform (and others). But for now we are wanting to understand the best options for rotating the tokens. Once a server is running we want the initially provided token to expire and the server to acquire a new token that expires after a time so that each server needs to continue getting new tokens periodically.

So in our setup the applications on the servers are not authenticating to get tokens in the first place.

Jonathan

[Paul Archer]

We're at about the same stage as you (actually not as far along) in our automation. My plan is to use ansible to generate a certificate for the client machine on its behalf (ridiculously easy to do with vault), push that to the client (in wrapped form), and use it for authentication against vault. Then it will be able to renew tokens or re-auth by itself. The certs can have a short lifetime, with ansible updating them every X period of time.

You might want to dig into AppRole, too. It's more complicated than using certs for authentication, but has an element of what you are looking for in key rolling. A trusted third party, like an ansible server, would generate the secret on behalf of the client, and push it to the client, where it would have a (very) limited lifetime. The client would have to periodically go back to the trusted third party. You have to establish a strong trust relationship, but the third party can cut the client off at any time, by simply not renewing the AppRole secret_id.

Paul

[SunSparc]

On Saturday, June 17, 2017 at 11:52:52 AM UTC-6, Paul Archer wrote:

We're at about the same stage as you (actually not as far along) in our automation. My plan is to use ansible to generate a certificate for the client machine on its behalf (ridiculously easy to do with vault), push that to the client (in wrapped form), and use it for authentication against vault. Then it will be able to renew tokens or re-auth by itself. The certs can have a short lifetime, with ansible updating them every X period of time.

Interesting approach.

 

You might want to dig into AppRole, too. It's more complicated than using certs for authentication, but has an element of what you are looking for in key rolling. A trusted third party, like an ansible server, would generate the secret on behalf of the client, and push it to the client, where it would have a (very) limited lifetime. The client would have to periodically go back to the trusted third party. You have to establish a strong trust relationship, but the third party can cut the client off at any time, by simply not renewing the AppRole secret_id.

I will take a look at AppRole. Thanks for the suggestion. 

[Jeff Mitchell]

Hi there,

Paul's suggestions are good to look into, but I have a more basic question:do you need your token lifetimes to be disparate from your application lifetimes? 

In many situations, the application is regularly rolled to deal with new deployments, security updates to any underling OS/base layers, and so on. It's pretty easy (using periodic tokens) to tie an application's token lifetime close to the application's lifetime if the period is going to be similar.

The benefit to this approach is that if the application goes down early for some reason, the token disappears quickly.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/c93852d1-0ea4-42b5-bf2b-d4217449e83f%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[SunSparc]

On Thursday, June 22, 2017 at 10:12:45 AM UTC-6, Jeff Mitchell wrote:

Paul's suggestions are good to look into, but I have a more basic question:do you need your token lifetimes to be disparate from your application lifetimes? 

In many situations, the application is regularly rolled to deal with new deployments, security updates to any underling OS/base layers, and so on. It's pretty easy (using periodic tokens) to tie an application's token lifetime close to the application's lifetime if the period is going to be similar.

The benefit to this approach is that if the application goes down early for some reason, the token disappears quickly.

Currently we are doing system level tokens and all the applications on the system use the same token. However we are wanting to move toward having the tokens be application specific.

It sounds like you are suggesting that the token request/renew/rotate mechanism be written into the application code. Or perhaps used as a library. That could be a good solution, though I am not sure we want the integration to be that tight.

[Jeff Mitchell]

As a quick and dirty sort of approach, Consul Template will keep a configured Vault token renewed, so you could just run that as a sidecar process purely for Vault token renewal :-)

Best,

Jeff

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/2f29baa7-5dfe-417b-a58d-ea0dddd502e8%40googlegroups.com.

[SunSparc]

On Tuesday, June 27, 2017 at 5:08:00 AM UTC-6, Jeff Mitchell wrote:

As a quick and dirty sort of approach, Consul Template will keep a configured Vault token renewed, so you could just run that as a sidecar process purely for Vault token renewal :-)

Yes, we rely heavily on Consul-Template already. It is a very helpful little tool for us. The renewing aspect is nice and we may very well take advantage of it. Yet my task of automating token rotation still remains. I appreciate the feedback.

[SunSparc]

 This is slightly off-topic, but I was reading the docs about the renew_token option of Consul-Template. It says: "This option only applies to the top-level Vault token itself." Does top-level in this context mean "root"?

[Jeff Mitchell]

Hi,

What it means is if you are making calls via C-T that generate tokens (for instance, to pass into your applications), it will not keep those renewed. Only the token that C-T itself is using will be kept renewed.

"root" there is a bit of an overloaded term; here it's not referring to the root policy within Vault, but rather since you can create child tokens that are revoked when the parent token is revoked, that parent token is at the root of the token tree.

Best,

Jeff

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/6126191c-8e51-4e2b-98d5-fac13312f0fa%40googlegroups.com.

[Chris Hiestand]

On Wednesday, June 14, 2017 at 10:47:24 AM UTC-7, SunSparc wrote:

2. Service/server periodically requests a new token (pull)

Create an application/script that lives on each server (alongside each service) that periodically hits the Vault HTTP API to create a new token using the same permissions and policies of the token that it currently has. This application could acquire a new token, place it in a token file, and restart/reload whatever services that depend on a Vault token.

Hi Jonathan. Did you ever figure out a way to get your idea number 2 working? I had the same idea but it seems like vault does not endorse this approach. In order for the new token to not expire when the old token does, the new token would have to be an orphan token. Creating an orphan token means the old (and new token) would require sudo capabilities, which seems like too much for this use case.

[Chris Hiestand]

Upon further study it seems like this is possible using a token role with the orphan parameter set.

But another feature that you might want to have is to limit how many of these tokens can be live at once (e.g. 1 or 2 ). I don't see any way for vault to enforce that, so you'd have to do that via another vault client with the necessary politcy?