puppet client certificates being revoked without human intervention

[Dustin Cannon]

Hi,

I've googled this and found a couple of people asking questions about what seems to be a similar issue a few years ago but no solutions.

The problem:

Some puppet clients will check in with the master and get: "SSL_connect returned=1 errno=0 state=unknown state: sslv3 alert certificate revoked"

No human being as explicitly revoked the cert as far as we know. Bash history doesn't show any revocation.

The cert is not expired. Private key modulus of and certificate modulus match.

This started happening a couple of weeks ago and only to some clients.

Context:

We have a janky dual-master setup and are running puppet version 3.4.3 on the masters. It's janky because what's being done is that one master runs haproxy to redirect 60% of requests to another machine. That machine rsyncs /var/lib/puppet (and uses the same certificate as the other). The clients that are apparently revoked are running version 3.7.2, but we have other clients running 3.7.2 that haven't had this issue. "puppet cert list --all" shows the certs as revoked (with the correct fingerprint) but those certs serial numbers do not appear in the certificate revocation list on the masters.

I'd appreciate any ideas or help in getting to the bottom of this. The problem is easy enough to fix by regenerating the certs but that doesn't tell us why this is happening in the first place and won't prevent it from happening in the future.

Happy to give more detail as needed. Thanks in advance!