Puppet Master as agent to itself continues to generate SSL errors

[Jeff Johnson]

I've been battling this issue all week with a new puppet (open source) build out. I stood up the puppet master and configured it. I have a test agent that works correctly (external to the master). However, the master is also an agent pointing to itself. In this instance, ep1p-apux06 is the puppet master. I generated the cert using the hostname 'puppet.domain.com' to try to prevent accidental deletion of the puppet master cert when running 'puppet cert clean --all'.

I can run 'puppet agent -t' on the master and it runs without error. But 30 minutes in, and these errors show up in the log files, even though if I rerun the puppet agent, it comes back without errors.

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: Unable to fetch my node definition, but the agent run will continue:

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: The certificate retrieved from the master does not match the agent's private key.

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: Certificate fingerprint: A9:1C:29:E2:66:65:46:EB:C8:37:C7:27:24:85:9C:58:2D:24:19:C4:2C:53:7B:46:D4:D5:65:93:57:CF:52:11

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: On the master:

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]:   puppet cert clean ep1p-apux06.domain.com

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: On the agent:

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]:   1a. On most platforms: find /etc/puppet/ssl -name ep1p-apux06.domain.com.pem -delete

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]:   1b. On Windows: del "/etc/puppet/ssl/ep1p-apux06.domain.com.pem" /f

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]:   2. puppet agent -t

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: (/File[/var/lib/puppet/facts.d]) Failed to generate additional resources using 'eval_generate': SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: (/File[/var/lib/puppet/facts.d]) Could not evaluate: Could not retrieve file metadata for puppet://puppet/pluginfacts: SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:43 ep1p-apux06 puppet-agent[20602]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://puppet/plugins: SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: Using cached catalog

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: (/Stage[main]/Ntp::Config/File[ntp.conf]) Could not evaluate: Could not retrieve file metadata for puppet:///modules/ntp/ntp.conf: SSL_CTX_use_PrivateKey:: key values mismatch

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: (/Stage[main]/Ntp::Service/Service[ntpd]) Dependency File[ntp.conf] has failures: true

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: (/Stage[main]/Ntp::Service/Service[ntpd]) Skipping because of failed dependencies

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: Finished catalog run in 0.03 seconds

Jun 19 08:39:44 ep1p-apux06 puppet-agent[20602]: Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch

I can run the agent test on itself and it comes back without error:

[root@ep1p-apux06 puppet]# puppet agent -t

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Caching catalog for ep1p-apux06.domain.com

Info: Applying configuration version '1434634454'

Notice: Finished catalog run in 0.29 seconds

I've ran the 'puppet cert clean ep1p-apux06.domain.com' and removed any ssl certs that might have remained:

[root@ep1p-apux06 puppet]# puppet cert clean ep1p-apux06.domain.com

Notice: Revoked certificate with serial 13

Notice: Removing file Puppet::SSL::Certificate ep1p-apux06.domain.com at '/var/lib/puppet/ssl/ca/signed/ep1p-apux06.domain.com.pem'

Notice: Removing file Puppet::SSL::Certificate ep1p-apux06.domain.com at '/var/lib/puppet/ssl/certs/ep1p-apux06.domain.com.pem'

Notice: Removing file Puppet::SSL::CertificateRequest ep1p-apux06.domain.com at '/var/lib/puppet/ssl/certificate_requests/ep1p-apux06.domain.com.pem'

Notice: Removing file Puppet::SSL::Key ep1p-apux06.domain.com at '/var/lib/puppet/ssl/private_keys/ep1p-apux06.domain.com.pem'

[root@ep1p-apux06 puppet]# find /etc/puppet/ssl -name ep1p-apux06.domain.com.pem -delete

[root@ep1p-apux06 puppet]# find /var/lib/puppet/ssl -name ep1p-apux06.domain.com.pem -delete

And then I rerun the 'puppet agent -t' again to regenerate the cert which completes successfully (I have 'autosign = true' on the master to automate adding new clients).

puppet.conf:

[root@ep1p-apux06 puppet]# cat /etc/puppet/puppet.conf

[main]

    # The Puppet log directory.

    # The default value is '$vardir/log'.

    logdir = /var/log/puppet

    # Where Puppet PID files are kept.

    # The default value is '$vardir/run'.

    rundir = /var/run/puppet

    # Where SSL certificates are kept.

    # The default value is '$confdir/ssl'.

    ssldir = $vardir/ssl

    runinterval = 1h

    server = puppet.domain.com

    environment = production

[master]

    dns_alt_names = puppet,puppet.starkey.com,puppetmaster,puppetmaster.starkey.com

    environment_timeout = unlimited

    always_cache_features = true

    autosign = true

[agent]

    # The file in which puppetd stores a list of the classes

    # associated with the retrieved configuratiion.  Can be loaded in

    # the separate ``puppet`` executable using the ``--loadclasses``

    # option.

    # The default value is '$confdir/classes.txt'.

    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration.  An

    # extension indicating the cache format is added automatically.

    # The default value is '$confdir/localconfig'.

    localconfig = $vardir/localconfig

Unfortunately 30 minutes later, the same error pops up in the messages file.

Any thoughts on what I might be missing?

[Nathan Abbott]

Hi Puppet users group,

First time poster here.

This issue matches what I'm facing exactly, although in my circumstance we are not intentionally trying to do the  " the master is also an agent pointing to itself  " thing.
We are in this situation because someone ran 'puppet ssl clean' and 'puppet agent -t' accidently on the master itself. (as opposed to the client where they were supposed to run it.)

Were you able to resolve this issue? i expect that we will have to regenerate a master cert and re-sign all client certs also? or is there a more simple/quicker solution that anyone can think of?

Thanks in advance

-Nathan