PUPPET 6.0 : PuppetDB SSL Engine issue

[Andy Hall]

Just fixed an issue with the puppetserver ca after a 5.x to 6.x upgrade (see post "PUPPET 6.0 : CSR from master does not match the agent public key" for more details) but now experience the following issue with PuppetDB (maybe a problem with the Java KeyStore ?):

AGENT:

# puppet agent --test

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for andy-puppet6-test.london.company.com: Failed to find facts from PuppetDB at puppet:8140: Failed to execute '/pdb/query/v4/nodes/andy-puppet6-test.london.company.com/facts' on at least 1 of the following 'server_urls': https://ldn1-puppet5.london.company.com:8081

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=53837e24e8b91d10fc3a81a657b83258c0ab3f8f&version=5&certname=andy-puppet6-test.london.company.com&command=replace_facts&producer-timestamp=1538588583' on at least 1 of the following 'server_urls': https://ldn1-puppet5.london.company.com:8081

Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

MASTER:

==> /var/log/puppetlabs/puppetserver/puppetserver.log <==
2018-10-03T18:49:26.860+01:00 ERROR [qtp1255475413-70] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
    at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
    at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
    at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doWrap(SSLIOSession.java:265)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:305)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:509)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doRunTask(SSLIOSession.java:283)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:353)
    ... 9 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
    at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:362)
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:270)
    at sun.security.validator.Validator.validate(Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601)
    ... 17 common frames omitted
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
    at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:154)
    at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
    at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
    at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:357)
    ... 23 common frames omitted
2018-10-03T18:49:26.873+01:00 WARN  [qtp1255475413-70] [puppetserver] Puppet Error connecting to ldn1-puppet5.london.company.com on 8081 at route /pdb/query/v4/nodes/andy-puppet6-test.london.company.com/facts, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list
2018-10-03T18:49:26.881+01:00 ERROR [qtp1255475413-70] [puppetserver] Puppet Server Error: Could not retrieve facts for andy-puppet6-test.london.company.com: Failed to find facts from PuppetDB at puppet:8140: Failed to execute '/pdb/query/v4/nodes/andy-puppet6-test.london.company.com/facts' on at least 1 of the following 'server_urls': https://ldn1-puppet5.london.company.com:8081

Seems to be an SSL issue with PuppetDB ? Maybe the Java KeyStore ? Please note this is not a simple TCP problem - the connection from agent to master on port 8081 is fine.

[Maggie Dreyer]

If you regenerated your CA as part of fixing the issues with the master/agent connection, did you also regenerate the certificates for PuppetDB? Not having really any experience with PuppetDB, I could see thi error being cause by still using certificates issued by the old certificate authority.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/10f93c46-6fbb-484f-9a60-a3ebbf0116b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Hall]

Apologies for the late reply but do you know how to re-create the certs for PuppetDB ? Is there a specific PuppetDB group who may be able to answer this ? Thanks very much.

[Andy Hall]

Hmm perhaps I should RTFM : https://puppet.com/docs/puppetdb/6.0/maintain_and_tune.html#redo-ssl-setup-after-changing-certificates

[Eric Sorenson]

Andy, did you get this fixed?

--eric0