Let's Encrypt renewal - restart required?

[Tom Ryan]

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version

* Used enterprise modules, if any

* JVM version and operating system version

* Search Guard configuration files

* Elasticsearch log messages on debug level

Hi,

I'm working on upgrading our 2.x cluster to 5.5.1, and searchguard-ssl therefore to 5.5.1-23, and I see that PEM certs are now usable in recent versions of searchguard-ssl (with a mention of letsencrypt in the example config). 

Will cert renewals require a restart of each Elasticsearch instance? Or will the file changing on disk cause searchguard-ssl to re-read?

If there are any other good practices and/or gotchas to think about here, I'd love to hear them.

Thanks in advance,

Tom

[Jochen Kressin]

Yes, any changes to the certificates require a restart of your node(s) in order for Search Guard to pick up the new certificates. The certs are loaded on start up of the node(s).

[Tom Ryan]

Thanks Jochen.

In that case, I'll stick with using our internal PKI. I trust it a bit more than Let's Encrypt anyway tbh. I do think it would be great ti have a way to reread/reload either the cert or searchguard-ssl itself, without doing a full node restart.

Regards,

Tom

[Jochen Kressin]

Just out of curiosity - what is your use case for auto-reloading the certificates? I understand that the Let's Encrypt certificates have an expiration date and you would need to renew them from time to time. Which would also mean a rolling restart of the cluster. However, automatically reloading the certificates would open a potential attack vector, which is why we decided against it, and rather support certificate revocation lists. How would you use an automatic certificate reload feature?

Thanks!

[Tom Ryan]

Let's Encrypt certs expire after 90 days, and are typically renewed after 60. The key, as you might expect, doesn't change.

I mostly asked because I saw Let's Encrypt mentioned in the latest example configs. After I thought about it a bit more, I decided it wasn't for me because of the immense trust it places in public DNS (which mostly cannot be avoided, but since we control all the clients, in this case it can).

But to answer your questions: reload could avoid having to rolling restart every ~60 days, and I'd expect it to either watch the files on disk or be called after successful renewal by the `certbot` renewal client.

hth,

tom

[Hector Martin]

I have a similar use case. I'm building an internal PKI but intend to issue short-lived certs (like Let's Encrypt); the idea is that by automating cert renewal and doing it often you can make sure the actual renewal process works and is tested, and it reduces the window of vulnerability if a cert is compromised. In the long run that also cuts down on the risk of "the certs expired and the cluster died all of a sudden since nobody noticed" scenarios.

The problem is that if cert renewals require a full restart, then that needs to be coordinated between nodes. If reloads can be done with no downtime, that means servers can be responsible for requesting renewal of their own cert on their own schedule. If downtime is involved then at minimum I'd have to use a fixed cron schedule to make sure nodes never restart at the same time, which is less ideal.

Why do you think cert reloads would open an attack vector? Surely if an attacker can change the certs on disk they can also restart Elasticsearch, assuming no egregious filesystem permissions issues.

[SG]

We are currently looking into that and maybe we come up with certificate hot reloading, so stay tuned ...
At the moment it is not possible.

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d692488a-434b-4ab0-a2b9-193ed7ebfc6c%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Joakim Sundqvist]

Any news on this?

[Jochen Kressin]

So far we have not decided to support certificate hot reloading. In our opinion, this would introduce major security risks and we still think that root certificate rollover should be a deliberate, manual process.

[Joakim Sundqvist]

Ok, thanks for the response.

In our case we have decided to go for letsencrypt certificates on all our servers so for us it means that we will have to restart the server every 60 days to reload the https certificate, for the transport protocol we are using a self-signed cert with an expiration date far in the future.

[SG]

Why is a restart every 60 days a problem? You can do it in a rolling manner and should not experience any downtime.
Normally you need anyhow to take down nodes down from time to time to apply patches for the OS or the JVM.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/b89eb71c-5a58-4dfc-8606-a9190a6158b9%40googlegroups.com.

[Joakim Sundqvist]

Well, it does not have to be a problem. 

We will for sure do automatic restarts to patch the servers in a rolling manner and roll the certificates on restart but sometimes the only thing changed is the certificate and then it would be nice to have a hot reload of the cert.