Additional Puppet Server Failure
Matthew Ceroni
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://aws-puppet-01.xxxxxx.com/pluginfacts: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://aws-puppet-01.xxxxxx.com/plugins: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Notice: Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished AIf I point the node at the other puppet server it runs fine.
I have verified that the certificate being used on the new puppet server is in fact signed by the same CA that generates the node certificate and the certificate used on the pre-existing puppet server. I can also connect to the new Puppet server via OpenSSL and issue a simple HTTP GET command (don't know a valid get request so end up with a 404 response, but it shows the certificate is setup correctly).
Also, I have disabled CA services on the second puppetserver as it is not providing CA capabilities.
As a test I removed the SSL certificates and started up Puppetserver fresh on the new server, that way it generated the CA certificates, etc. A node then successfully checks in, gets a certificate and can apply a manifest. This indicates I am doing something wrong in creating certs from the primary server (CA) for the new Puppet server.
Any help on what the issue is would be appreciated.
Chris Price
On Monday, February 22, 2016 at 5:01:20 PM UTC-8, Matthew Ceroni wrote:
I have the following setup.1) Existing PuppetServer version 1.1.32) New PuppetServer version 1.1.3 installation (new DC)I followed the following documentation on setting up multiple Puppet MastersEssentially I pre-generated the SSL certificate for the new puppet server from the existing puppet server, instead of letting the puppet server generate it on startup.A new node checks in and contacts server 1 (pre-existing Puppet server) for CA functions (configured via ca_server in puppet.conf). Certificate generates and node caches it. However, the puppet run (which runs against the new puppet server) generates the following error:
Warning: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://aws-puppet-01.xxxxxx.com/pluginfacts: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://aws-puppet-01.xxxxxx.com/plugins: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Info: Loading facts
Error: Could not retrieve catalog from remote server: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
Notice: Using cached catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: SSL_connect SYSCALL returned=5 errno=0 state=SSLv3 read finished A
If I point the node at the other puppet server it runs fine.
I have verified that the certificate being used on the new puppet server is in fact signed by the same CA that generates the node certificate and the certificate used on the pre-existing puppet server. I can also connect to the new Puppet server via OpenSSL and issue a simple HTTP GET command (don't know a valid get request so end up with a 404 response, but it shows the certificate is setup correctly).
Are you using the agent's certs/key when you make this connection via openssl? Can you include the syntax of the command you're running for this? Have you tried with curl? e.g.
curl --cacert $ssldir/certs/ca.pem --cert $ssldir/certs/agent-node.pem --key $ssldir/private_keys/agent-node.pem https://new-puppet-master:8140/production/nodes/agent-node
Also, are there any log messages in the puppetserver.log file? If not, perhaps try editing the 'logback.xml' config file and changing the log level from info/warn to 'debug', and then see if any log messages appear?
My best guess at the moment is that either:
* the new puppet server's cert is not signed by the same CA, or,
* the certname of the puppet server cert does not match the hostname that the agent is using to try to connect to it
Also, I have disabled CA services on the second puppetserver as it is not providing CA capabilities.
How did you go about doing this?
As a test I removed the SSL certificates and started up Puppetserver fresh on the new server, that way it generated the CA certificates, etc.
If the new puppet server instance is generating its own CA cert, then its CA is not disabled, and its CA cert will not be compatible with the CA cert from the original puppet server. The steps I'd recommend:
1. shut down puppet server on the new puppet server host
2. make sure that its CA is disabled ( https://docs.puppetlabs.com/puppetserver/latest/external_ca_configuration.html#disabling-the-internal-puppet-ca-service )
3. set up the puppet agent on the new puppet server host; configure it to connect to the old puppet server as its master
4. do an agent run on the new puppet server host; this will cause it to request a cert from the old puppet server CA
5. once you've signed that cert and are able to complete your agent runs successfully on the new puppet server host, it should have valid certs in place
6. start puppet server back up on the new puppet server host; it should find the certs that were generated by the agent and use them
7. now try running an agent on a different host, pointed at your new puppet server.
Matthew Ceroni
Also, I have disabled CA services on the second puppetserver as it is not providing CA capabilities.
How did you go about doing this?
As a test I removed the SSL certificates and started up Puppetserver fresh on the new server, that way it generated the CA certificates, etc.
If the new puppet server instance is generating its own CA cert, then its CA is not disabled, and its CA cert will not be compatible with the CA cert from the original puppet server. The steps I'd recommend:
1. shut down puppet server on the new puppet server host
2. make sure that its CA is disabled ( https://docs.puppetlabs.com/puppetserver/latest/external_ca_configuration.html#disabling-the-internal-puppet-ca-service )
3. set up the puppet agent on the new puppet server host; configure it to connect to the old puppet server as its master
4. do an agent run on the new puppet server host; this will cause it to request a cert from the old puppet server CA
5. once you've signed that cert and are able to complete your agent runs successfully on the new puppet server host, it should have valid certs in place
6. start puppet server back up on the new puppet server host; it should find the certs that were generated by the agent and use them
7. now try running an agent on a different host, pointed at your new puppet server.
A node then successfully checks in, gets a certificate and can apply a manifest. This indicates I am doing something wrong in creating certs from the primary server (CA) for the new Puppet server.
Any help on what the issue is would be appreciated.