PuppetDB : unable to upgrade 6.5 to 6.9 => SSL errors

[Yvan Broccard]

Hi,

I'm struggling with a simple update of PuppetDB since a couple of days, without finding the problem.

I have 4 PuppetServers running Puppetserver 6.9 (puppetserver-6.9.0-1.el7.noarch). One has the CA role, the 3 others are simple masters. I have one dedicated PuppetDB server running puppetdb-6.5.0-1.

Everything is working like a charm since a couple of years. It was updated from Puppet 3, 4 and 6 without a glitch. Everything is running on CentOS 7.

Now, when I want to update PuppetDb from 6.5 to 6.9, nothing works anymore.

All nodes are complaining with these messages :

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 500 on SERVER: Server Error: Could not retrieve facts for vmlabybr06.staging.rsvgnw.local: Failed to find facts from PuppetDB at vmprdpuppet41.rsvgnw.local:8140: Failed to execute '/pdb/query/v4/nodes/vmlabybr06.staging.rsvgnw.local/facts' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

Info: Loading facts

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed to execute '/pdb/cmd/v1?checksum=5da252cdae0fc1737726e9ace846d74856395703&version=5&certname=vmlabybr06.staging.rsvgnw.local&command=replace_facts&producer-timestamp=2020-04-09T13:15:44.382Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081

Warning: Not using cache on failed catalog

Error: Could not retrieve catalog; skipping run

In the server log I get this :

2020-04-09T15:22:45.169+02:00 WARN  [qtp1002336767-143] [c.p.h.c.i.PersistentSyncHttpClient] Error executing http request

javax.net.ssl.SSLException: Received fatal alert: handshake_failure

        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)

        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1647)

        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1615)

        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1781)

        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1070)

        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:896)

        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:766)

        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)

        at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:271)

        at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:316)

        at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:503)

        at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)

        at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)

        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)

        at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)

        at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)

        at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)

        at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)

        at java.lang.Thread.run(Thread.java:748)

2020-04-09T15:22:45.171+02:00 WARN  [qtp1002336767-143] [puppetserver] Puppet Error connecting to vmctldeploy20.rsvgnw.local on 8081 at route /pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z, error message received was 'Error executing http request'. Failing over to the next PuppetDB server_url in the 'server_urls' list

2020-04-09T15:22:45.172+02:00 ERROR [qtp1002336767-143] [puppetserver] Puppet Failed to execute '/pdb/cmd/v1?checksum=0f8f2f1e474b2f551f6dc656bff34f1e43e56f6b&version=8&certname=vmlabvmt01.rsvgnw.local&command=store_report&producer-timestamp=2020-04-09T13:22:45.130Z' on at least 1 of the following 'server_urls': https://vmctldeploy20.rsvgnw.local:8081

I have checked a few things :

- Updated puppetdb-termini on the puppet-master from 6.5 to 6.9 (no change)

- added "verify_client_certificate = false" to /etc/puppetlabs/puppet/puppetdb.conf on the masters (no change)

- added full certs list to PuppetDB server /etc/puppetlabs/puppetdb/ssl/public.pem

I've read there has been a change liked to SSL in the PuppetDB 6.6 CHANGELOG.

Here is what happens when I try to connect with openssl for troubleshooting, to PuppetDB 6.5

openssl s_client -host puppetdb -port 8081 -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem

CONNECTED(00000003)

Can't use SSL_get_servername

depth=1 CN = Puppet CA: vmctldeploy10.rsvgnw.local

verify return:1

depth=0 CN = vmctldeploy20.rsvgnw.local

verify return:1

140503727654720:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42

---

Certificate chain

 0 s:CN = vmctldeploy20.rsvgnw.local

   i:CN = Puppet CA: vmctldeploy10.rsvgnw.local

---

Server certificate

-----BEGIN CERTIFICATE-----

...

-----END CERTIFICATE-----

subject=CN = vmctldeploy20.rsvgnw.local

issuer=CN = Puppet CA: vmctldeploy10.rsvgnw.local

---

Acceptable client certificate CA names

CN = Puppet CA: vmctldeploy10.rsvgnw.local

Client Certificate Types: RSA sign, DSA sign, ECDSA sign

Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1

Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1

Peer signing digest: SHA256

Peer signature type: RSA

Server Temp Key: ECDH, P-256, 256 bits

---

SSL handshake has read 2213 bytes and written 455 bytes

Verification: OK

---

The only way to go back is doing a full "revert to snaphot", as the db is migrated between 6.5 and 6.9

Any advise welcome !

Cheers

Yvan

[comport3]

"Redo SSL setup after changing certificates

If you’ve recently changed the certificates in use by the PuppetDB server, you’ll also need to update the SSL configuration for PuppetDB itself.

If you’ve installed PuppetDB from Puppet packages, you can simply re-run the puppetdb ssl-setup command. Otherwise, you’ll need to again perform the SSL configuration steps outlined in the installation instructions."

https://puppet.com/docs/puppetdb/latest/maintain_and_tune.html

[Yvan Broccard]

Hello,

Thank you for your reply. My PuppetDB is installed and managed by puppetdb puppet module, and I didn't change the certificates since its installation years ago (Still valid for one year though).

I will have a try with puppetdb ssl-setup later today.

What I have noticed with openssl, before and after the upgrade to 6.9, there are a few different Ciphers used.

echo QUIT | openssl s_client -connect puppetdb:8081 -host puppetdb -port 8081 -CAfile /etc/puppetlabs/puppet/ssl/certs/ca.pem -status -state -showcerts 2>&1

diff -u s_client.65 s_client.69
--- s_client.65 2020-04-27 16:54:53.887179070 +0200
+++ s_client.69 2020-04-27 16:59:36.347189451 +0200
@@ -16,7 +16,7 @@
SSL_connect:SSLv3/TLS write finished
SSL3 alert read:fatal:bad certificate
SSL_connect:error in error
-139851809683264:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42
+139950036502336:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42
CONNECTED(00000003)
OCSP response: no response sent
---
@@ -69,12 +69,12 @@

Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA256
Peer signature type: RSA

-Server Temp Key: ECDH, P-256, 256 bits
+Server Temp Key: DH, 1024 bits
---
-SSL handshake has read 2217 bytes and written 499 bytes
+SSL handshake has read 2411 bytes and written 539 bytes
Verification: OK
---
-New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
+New, TLSv1.2, Cipher is DHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
@@ -82,14 +82,14 @@
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
-    Cipher    : ECDHE-RSA-AES256-SHA
-    Session-ID: 5EA6F235228812A1D39268BEA73CA0538FBD9DB65BDBFE0B2A7B620D619608CF
+    Cipher    : DHE-RSA-AES128-GCM-SHA256
+    Session-ID: 5EA6F33E2ED8579BEF57A377556A369D4E2194D1E009250BCE0D972002D4D0C1
    Session-ID-ctx:  
-    Master-Key: 1B3E5AF06F394B30E32D5E957D0F9FC8270C19FCB6BE32FCB27B51E310F2C735F1C0E4AFE4DBFD98A67F53F945C34967
+    Master-Key: 851F2F19D603D607DB9410ED5E945A76AF1408AE8692D4A4AB8A46598C88F9CF82E19748B411C5C0CB33731E856B1681
    PSK identity: None
    PSK identity hint: None
    SRP username: None
-    Start Time: 1587999285
+    Start Time: 1587999550
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes

But we can clearly see the "verify" is ok on 6.9 as well

...

verify return:1

...

Yvan

[Yvan Broccard]

For the people here maybe having the same error. I was able to resolve the issue in updating Java OpenJDK from 1.7 to version 11.

Then the problem was immediatelly solved.

So, it was a cipher issue between Java and Puppetdb. With more recent version of Java all is solved.

Puppetdb ssl-setup said everything was allright.

Cheers

Yvan