Weird puppetmaster issue.

[Peter Berghold]

I've been banging my head against the wall with this one.  Probably a simple fix but I'm not seeing it at the moment.

I have a proof of concept setup in our lab where I have a central Foreman/Puppet server.  The Puppet server on the Foreman box has the responsibility of keeping things straight on subordinate "remote servers" that in production will be located in the same data center as the client base they will server.

To get the POC remote server set up I manually load the Puppet client software on the host (later it will come pre-loaded on a VM template) and I run the Puppet client manually.  That first run is all about the certs and getting the Foreman ENC aware of the host.  Adding a couple of config groups the the host and next we run puppet agent -t-test --server=<foreman host fqdn> and all the configuration files get loaded and then things go sideways. 

The Puppet Master daemon will not run on the host.  What I'm seeing in /var/log/messages is as follows:

Feb 18 19:48:20 <hostname> puppet-master[32593]: Could not prepare for execution: The certificate retrieved from the master does not match the agent's private key.

Feb 18 19:48:20 <hostname>puppet-master[32593]: Certificate fingerprint: BC:0C:19:83:62:F8:A6:AD:ED:85:B7:19:B6:AD:75:FE:36:62:D7:43:C9:5B:76:64:E8:A1:F5:C1:FE:1F:39:21

Feb 18 19:48:20 <hostname>puppet-master[32593]: To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

Feb 18 19:48:20 <hostname> puppet-master[32593]: On the master:

Feb 18 19:48:20 <hostname> puppet-master[32593]:   puppet cert clean <agent host FQDN>

Feb 18 19:48:20 <hostname> puppet-master[32593]: On the agent:

Feb 18 19:48:20 <hostname> puppet-master[32593]:   1a. On most platforms: find /var/lib/puppet/ssl -name <agent host FQDN> -delete

Feb 18 19:48:20 <hostname>puppet-master[32593]:   1b. On Windows: del "/var/lib/puppet/ssl/<agent host FQDN>.pem" /f

Feb 18 19:48:20 <hostname> puppet-master[32593]:   2. puppet agent -t

Here is my puppet.conf (sanitized) from my clent system

[main]

    # The Puppet log directory.

    # The default value is '$vardir/log'.

    logdir = /var/log/puppet

    # Where Puppet PID files are kept.

    # The default value is '$vardir/run'.

    rundir = /var/run/puppet

    # Where SSL certificates are kept.

    # The default value is '$confdir/ssl'.

    ssldir = $vardir/ssl

    # Allow services in the 'puppet' group to access key (Foreman + proxy)

    privatekeydir = $ssldir/private_keys { group = service }

    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

    # Puppet 3.0.x requires this in both [main] and [master] - harmless on agents

    autosign       = $confdir/autosign.conf { mode = 664 }

    show_diff     = false

    hiera_config = $confdir/hiera.yaml

### Next part of the file is managed by a different template ###

## Module:           'puppet'

[agent]

    # The file in which puppetd stores a list of the classes

    # associated with the retrieved configuration.  Can be loaded in

    # the separate ``puppet`` executable using the ``--loadclasses``

    # option.

    # The default value is '$statedir/classes.txt'.

    classfile = $vardir/classes.txt

    # Where puppetd caches the local configuration.  An

    # extension indicating the cache format is added automatically.

    # The default value is '$confdir/localconfig'.

    localconfig = $vardir/localconfig

    # Disable the default schedules as they cause continual skipped

    # resources to be displayed in Foreman - only for Puppet >= 3.4

    default_schedules = false

    report            = true

    pluginsync        = true

    masterport        = 8140

    environment       = production

    certname          = <agent host fqdn>

    server            = <puppet host fqdn>

    listen            = false

    splay             = false

    splaylimit        = 1800

    runinterval       = 1800

    noop              = false

    configtimeout     = 600

[master]

    storeconfigs = true

    storeconfigs_backend = puppetdb

    autosign       = $confdir/autosign.conf { mode = 664 }

    reports        = foreman

    external_nodes = /etc/puppet/node.rb

    node_terminus  = exec

    ca             = true

    ssldir         = /var/lib/puppet/ssl

    certname       = <agent host FQDN>

    strict_variables = false

    environmentpath  = /etc/puppet/environments

    basemodulepath   = /etc/puppet/environments/common:/etc/puppet/modules:/usr/share/puppet/modules

Just to be clear:  <puppet host FQDN> is the central Foreman/Puppet Master's FQDN not the agent FQDN that where a "remote" Puppet master is trying to start.

Thoughts?  Am I going about this wrong?

--

Peter L. Berghold                       Salty....@gmail.com

http://science-fiction.berghold.net

[jcbollinger]

On Wednesday, February 18, 2015 at 2:12:43 PM UTC-6, Salty Old Cowdawg wrote:

I've been banging my head against the wall with this one.  Probably a simple fix but I'm not seeing it at the moment.

I have a proof of concept setup in our lab where I have a central Foreman/Puppet server.  The Puppet server on the Foreman box has the responsibility of keeping things straight on subordinate "remote servers" that in production will be located in the same data center as the client base they will server.

I suspect that your issue as to do with exactly what it means (or should mean) for a remote server to be "subordinate".  More on that below.

 

To get the POC remote server set up I manually load the Puppet client software on the host (later it will come pre-loaded on a VM template) and I run the Puppet client manually.  That first run is all about the certs and getting the Foreman ENC aware of the host.  Adding a couple of config groups the the host and next we run puppet agent -t-test --server=<foreman host fqdn> and all the configuration files get loaded and then things go sideways. 

If I understand correctly, you are using the Puppet agent on your subordinate remote server to configure that server for operation (in particular, for operation as a Puppet master for other machines in the same data center).

 

The Puppet Master daemon will not run on the host.  What I'm seeing in /var/log/messages is as follows:

Feb 18 19:48:20 <hostname> puppet-master[32593]: Could not prepare for execution: The certificate retrieved from the master does not match the agent's private key.

Feb 18 19:48:20 <hostname>puppet-master[32593]: Certificate fingerprint: BC:0C:19:83:62:F8:A6:AD:ED:85:B7:19:B6:AD:75:FE:36:62:D7:43:C9:5B:76:64:E8:A1:F5:C1:FE:1F:39:21

Feb 18 19:48:20 <hostname>puppet-master[32593]: To fix this, remove the certificate from both the master and the agent and then start a puppet run, which will automatically regenerate a certficate.

Feb 18 19:48:20 <hostname> puppet-master[32593]: On the master:

Feb 18 19:48:20 <hostname> puppet-master[32593]:   puppet cert clean <agent host FQDN>

Feb 18 19:48:20 <hostname> puppet-master[32593]: On the agent:

Feb 18 19:48:20 <hostname> puppet-master[32593]:   1a. On most platforms: find /var/lib/puppet/ssl -name <agent host FQDN> -delete

Feb 18 19:48:20 <hostname>puppet-master[32593]:   1b. On Windows: del "/var/lib/puppet/ssl/<agent host FQDN>.pem" /f

Feb 18 19:48:20 <hostname> puppet-master[32593]:   2. puppet agent -t

If the master is emitting those messages at startup, it must be talking about its own cert (so the message is misleading).  I'm inclined to suppose that the problem relates to the CA(s).  The subordinate master's cert was signed by the central master's CA, but by default a master relies on its own CA (and your config explicitly specifies that the remote masters should do this).  From the subordinate master's perspective, therefore, its own certificate is untrusted (there is no trusted authority in its chain).  This is a bona fide problem, because that cert won't be trusted by the remote master's clients, either.  There have been ideas floating around for a long time about how to deal with this sort of scenario, but as far as I am aware, there are few good solutions.

Probably the easiest class of solutions to set up is the central CA.  Instead of your remote masters providing CA services to their clients, all client certificate requests would go to your central master's CA.  You can configure this directly in the agents' Puppet configuration, or you can set up CA proxies on the satellite masters to disguise the true location of the CA.

An alternative would be to put two separate copies of Puppet on your satellite masters, one with which to run the agent against the central master, and another with which to serve its own clients.  I think this is dirty, ugly, and dangerous, and I do not recommend it, but I mention it because you're likely to come up with the idea on your own, anyway.

If you don't like the central CA then I think the best thing to do -- and I don't think there's an automated way to go about it -- is to set up the remote masters with CA certs signed by the central CA, and have each master's identifying cert signed by its own CA.  I think the masters' certs will then be trusted by the central master because its own CA is in their chain of trust, and each master and its agents will certainly trust that master's cert because it is signed by the master's CA.  If you wanted machines, including masters, at different remote sites to trust each other's certs, then maybe you could accomplish it by adding the central CA's cert to their trusted CA lists.


John