Foreman, Katello, Puppet. Scalable build. Node failing following CSR.

[bren...@gmail.com]

G'day everyone.

So far, have built out:

• host001 - PostgreSQL
  
• host002 - memcached
  
• host003 - Foreman UI with Katello
  
• host004 - Puppet certificate authority
  
• host005 - Puppet compile master
  

Where:

• host003 uses host001 for persistent storage, host002 for transient values (in the event we create more Foreman nodes)
• host004 is connected to host003 using Smart Proxy
• host005 is connected to host003 using Smart Proxy

If we take a new host - node001 - and modify puppet.conf to read like

  [main]

    ca_server = host004

  [agent]

    server = host005

And execute puppet agent --test --verbose, we can successfully begin a certificate signing request.  

After we sign the request on host004, we execute puppet agent --test --verbose from node001 again, but encounter the following:

[root@node001 ~]# puppet agent --test --verbose

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Info: Retrieving pluginfacts

Error: /File[/opt/puppetlabs/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Info: Retrieving plugin

Error: /File[/opt/puppetlabs/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Error: /File[/opt/puppetlabs/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Error: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

Error: Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [self signed certificate in certificate chain for /CN=Puppet CA: host005]

[root@node001 ~]#

Does anyone have an indication as to why this may be happening?

[BJ]

Believe we've had some success.

When the Puppet master was created it has gone and created it's certificate:

[root@host005~]# puppet cert list --all

+ "host005" (SHA256)

And determined this to be self-signed by host005, being listed as the issuer, using openssl x509 -text -noout -in /etc/puppetlabs/puppet/ssl/certs/host005.pem.

After referring to the following resources over the past week:

• https://puppet.com/docs/pe/2017.2/regenerate_certs_master.html
  
• https://puppet.com/docs/pe/2017.2/compile_master_cert_regen.html
  
• https://ask.puppet.com/question/15772/how-to-setup-multiple-puppet-masters-with-a-single-ca/
  
• https://serverfault.com/questions/644673/how-to-seperate-the-ca-server-from-master-basic-setup
  
• https://blkperl.github.io/replace-puppet-ca.html

We:

1. Revoked host005's certificate 
2. Stopped all the Puppet services and httpd
3. On host004, generated certificates for host005 (puppet cert generate host005)
4. Copied the contents of /etc/puppetlabs/puppet/ssl from host004 > host005
5. Restarted httpd and all the Puppet services

After this we cleaned certs for node001; completed CSR with the Puppet certificate authority (host004); was able to pull content from the Puppet master (host005).

Still to validate with additional nodes and some further testing to verify nothing in our playpen has gotten borked.

Hopefully this proves of some use to anyone else that encounters the issue.