Consul "encrypt" key and TLS certificate rotation

[ss...@agari.com]

Hello,

I'm starting to roll out Vault+Consul mostly along the lines of the Terraform registry modules at

https://registry.terraform.io/modules/hashicorp/vault/aws/

https://registry.terraform.io/modules/hashicorp/consul/aws

and have folded in Gossip Encryption and TLS certificates per

https://registry.terraform.io/modules/hashicorp/consul/aws

https://russellsimpkins.blogspot.com/2015/10/consul-adding-tls-using-self-signed.html

using a single wildcard cert for both server/client auth.

I'm now thinking down the road and considering what a zero-downtime TLS certificate rotation process would look like.

It seems to me the options are

1. keep the CA key around so you can generate a new cert signed by the same CA, copy the new cert into place and do a rolling restart; in this case you have to secure the CA private key and keep track of the CA metadata like serial number etc

2. if you follow the "throw away the private key" approach suggested at the bottom of https://github.com/hashicorp/terraform-aws-vault/tree/master/modules/private-tls-cert (which, granted, is a suggestion for Vault and not Consul) then by design you cannot generate a new cert signed by the existing CA key.  In this case I think you could effect a zero-downtime rotation at the cost of a brief period without TLS by doing

2a. copy new CA/cert files into place on all Consul servers AND clients

2b. push a config change that turns off verify_incoming and verify_outgoing, do a rolling restart of consul agents

2c. push a second config change that turns verify_* back on and do another rolling restart of consul agents

Am I missing anything fundamental about (1) or (2)?  Are there other options 3, 4...?

And separately but similarly, if a situation ever came up where it was deemed necessary to rotate the "encrypt" key the only clean way to rotate that it seems would be to turn it off and then turn it back on again with the new key across two rolling restarts?

Thanks!

[ow...@avarteq.de]

"I'm now thinking down the road and considering what a zero-downtime TLS certificate rotation process would look like."

any news on that? we are currently also looking for this