certificates being 'randomly' revoked

[st...@wtfast.com]

Hi,

I've been having issues with certificates being revoked without any human intervention or oversight; one day a node will try to do an update and it can't because its certificate is revoked.

There is definitely no one issuing 'puppet cert clean nodename' on the commandline.

puppet --version

3.4.3

any ideas? Is there some automated process that 'cleans' and revokes nodes that are 'too old'?

I'd like to have control over this and have absolutely no automated system revoking certificates at all.

Thanks.

[jcbollinger]

Puppet does not auto-revoke client certificates, but certificates do have a fixed lifetime, set when they are generated.  I don't recall Puppet's default, but lifetimes are usually at least several years.   If your lifetimes are abnormally short, or if some of your agents have been in operation a long time, though, then that may be what's happening.  You cannot disable this aspect of cryptographic certificates, but you can choose very long lifetimes.


John

[st...@wtfast.com]

These are not new nodes but not old either, only a few months. The date/time is correct. The DNS is correct. I have not manually set certificate lifetimes to be shorter than the default. However sometimes these nodes might not check in for a few days.

This was recently a big problem as the cert for the puppetdb server was revoked.

How can I get more information about the revocation?

[jcbollinger]

On Tuesday, March 18, 2014 10:25:02 AM UTC-5, st...@wtfast.com wrote:

These are not new nodes but not old either, only a few months. The date/time is correct. The DNS is correct. I have not manually set certificate lifetimes to be shorter than the default. However sometimes these nodes might not check in for a few days.

This was recently a big problem as the cert for the puppetdb server was revoked.

How can I get more information about the revocation?

You could start by giving us more information.  Specifically, the actual messages that lead you to conclude that certificates have been revoked.

You could also look at the Puppet CA's data files in /var/lib/puppet/ssl/ca, or something like that.  The inventory of current certificates and the CRL should both be there.


Is there any chance that your nodes' timekeeping is inconsistent?  That can happen with VMs, for instance.  If your nodes do not agree fairly closely with the master with respect to the current date and time of day then that can prevent successful SSL handshaking.


John

[Throwe, Jesse]

Steve,

Do you, perchance, have multiple puppet masters in play?

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/40e1eea2-50c5-435a-adcd-b6d6b3ce1912%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[st...@wtfast.com]

I don't have one of the actual messages handy right now but when it occurs I run 'puppet agent --test' and instead of doing its work it presents an error message which explains that the certificate (of the node I ran puppet on) has been revoked. This is what led me to believe that the problem was with certificate revocation.

I guess its possible that timekeeping could be an issue. However this has happened with a new puppetmaster server and a puppetdb server running on the same VMWare ESXi host. It seems a bit unlikely that the clocks would be significantly skewed. Just how much does the clock have to vary before the master decides the certificate is revoked? Also, when this problem has happened I have checked time on both master and node and they appeared to agree very closely (to my eye). But would this present itself as a "certificate revoked" error message?

[st...@wtfast.com]

No. Only one puppet master.

[st...@wtfast.com]

What would happen if I chattr +i ca_crl.pem to prevent it being updated?

Certificate revocation is something that should be manually controlled anyway.

Suppose that the Puppet error message is wrong (or at least misleading) and the problem is not revocation. If the crl.pem file is immutable and this error really happened then I would know that it really isn't a revocation, right? And if I ever do want to revoke a cert all I have to do is chattr -i

Would this break anything else in Puppet?

However looking at it now, I can see that the ca_crl.pem was in fact updated on the day I had problems with the puppetdb servers certificate being 'revoked' so perhaps there is something revoking certs? Or is this just coincidence?

Here we go:

openssl crl -in ca_crl.pem -text shows a bunch of revocations and

    Serial Number: 0C

        Revocation Date: Mar 17 18:15:36 2014 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Key Compromise

why would some automated system think the key was compromised and revoke it without any human intervention?

[jcbollinger]

Key compomise is the default revocation reason; that's what Puppet will record if no other is specified.

I remain dubious that anything within Puppet automatically revoked your certificates.  I even had a look at the code, and found no evidence that Puppet has any automated mechanism to do that.  All certificate revocations appear to come back to the test suite (you're not running that on your live master, right?) or to one of Puppet's UI termini.


John

[st...@wtfast.com]

I'm not sure about test suite or UI terminii, I'm just running Puppet from the repos at puppetlabs (its not Puppet enterprise). This is happening in production but I can't reproduce it, it just happens apparently at random.

I've set the crl file immutable, hopefully next time something tries to do this it will complain and I'll catch it.

[Cliff Wells]

On Wednesday, March 19, 2014 2:14:15 PM UTC-7, jcbollinger wrote:

On Wednesday, March 19, 2014 10:15:19 AM UTC-5, st...@wtfast.com wrote:

What would happen if I chattr +i ca_crl.pem to prevent it being updated?

Certificate revocation is something that should be manually controlled anyway.

Suppose that the Puppet error message is wrong (or at least misleading) and the problem is not revocation. If the crl.pem file is immutable and this error really happened then I would know that it really isn't a revocation, right? And if I ever do want to revoke a cert all I have to do is chattr -i

Would this break anything else in Puppet?

However looking at it now, I can see that the ca_crl.pem was in fact updated on the day I had problems with the puppetdb servers certificate being 'revoked' so perhaps there is something revoking certs? Or is this just coincidence?

Here we go:

openssl crl -in ca_crl.pem -text shows a bunch of revocations and

    Serial Number: 0C

        Revocation Date: Mar 17 18:15:36 2014 GMT

        CRL entry extensions:

            X509v3 CRL Reason Code:

                Key Compromise

why would some automated system think the key was compromised and revoke it without any human intervention?

Key compomise is the default revocation reason; that's what Puppet will record if no other is specified.

I remain dubious that anything within Puppet automatically revoked your certificates.

I'm not. We've also experienced this perhaps a dozen times over the last year and a half (most recently this morning, where the puppet master revoked it's own cert).

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate revoked for /CN=puppetmaster1.redacted.com]

Our Puppet install is maintained by two people and neither of us revoked this cert. The puppet master was built perhaps a month ago. No DNS, time, or other issues seem apparent.
 
I have not looked into the code myself, but the behavior is clearly there.

[Felix Frank]

On 10/29/2014 07:27 PM, Cliff Wells wrote:

Key compomise is the default revocation reason; that's what Puppet will record if no other is specified.

I remain dubious that anything within Puppet automatically revoked your certificates.

I'm not. We've also experienced this perhaps a dozen times over the last year and a half (most recently this morning, where the puppet master revoked it's own cert).

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [certificate revoked for /CN=puppetmaster1.redacted.com]

Our Puppet install is maintained by two people and neither of us revoked this cert. The puppet master was built perhaps a month ago. No DNS, time, or other issues seem apparent.
 
I have not looked into the code myself, but the behavior is clearly there.

To get to the bottom of this, try and make frequent backups of the CRL file (rsync can do this quite economically).
Once you notice a random revocation, inspect the history of the CRL to see just when the serial in question was added. This will hopefully help you figure out what might have happened.

HTH,
Felix

[Dustin Cannon]

Did you ever get to the bottom  of this? I'm seeing a similar issue. We have a janky dual master setup where one master rsyncs /var/lib/puppet (and uses the same certificate) from the other master and the other master runs haproxy and sends 60% of requests to the other one. We are seeing some non-expired certs randomly appear revoked. "puppet cert list -all" shows the cert as revoked. But the serial number for the supposedly revoked cert is not in /var/lib/puppet/ssl/ca/ca_crl.pem nor in /var/lib/puppet/crl.pem. Seem like this just started happening a couple of weeks ago. I know this thread is over 3 years old but not really finding much on this. This is with puppet version 3.4.3.

[David Danzilio]

I’ve seen this before as well. It appears to be documented in PUP-2931. There are a lot of issues running the Puppet CA in an active/active HA configuration like this. I suggest you pick one master to be the CA and send all CA requests to that master.