PuppetDB SSL error (while trying to reach the dashboard)

[SirHopcount]

Hi All,

I am having some problems getting PuppetDB and SSL to work. I build an test environment in Vagrant without any problems but when it try and deploy it to my production environment I can't seem to get it to work with SSL.  I followed the instructions on the puppetlabs website and I made sure I could do an successful puppet run before installing PuppetDB:

(changed the hostname/company name)

# puppet agent --test
info: Caching catalog for hostname.company.local
info: Applying configuration version '1347960542'
notice: Finished catalog run in 2.04 seconds

But when I try to reach the dashboard it get an error:

The connection was interrupted
    The connection to xxx.xxx.xxx.xxx:8081 was interrupted while the page was loading.

When I check the puppetdb log file I see the following error:

2012-09-19 10:39:24,016 WARN  [qtp1281335597-66] [io.nio] javax.net.ssl.SSLException: Received fatal alert: unknown_ca
2012-09-19 10:39:49,182 WARN  [qtp1281335597-67] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain
2012-09-19 10:39:49,218 WARN  [qtp1281335597-66] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain
2012-09-19 10:39:49,251 WARN  [qtp1281335597-67] [io.nio] javax.net.ssl.SSLProtocolException: handshake alert: no_certificate

I checked the jetty.ini file and made sure both the keystore and truststore exist. It also has an key-password and trust-password set. I changed the jetty.ini file to allow http connections and then I can reach the dashboard via http without any problems. Does anybody have an idea what could be wrong ?

[Chris Price]

Hi SirHopcount,

How did you install PuppetDB?  Is there any chance that any of your certs (agent / master) have changed since the time when you installed?

I've found that sometimes the easiest way to get things fixed up when you have this problem is to do the following:

0) Stop puppetdb

1) remove PuppetDB's SSL directory entirely (usually /etc/puppetdb/ssl)

2) make sure that the agent on the puppetdb machine can run successfully against the master (puppet agent --test, sounds like you've already done this.

3) Run the puppetdb-ssl-setup script (which should be in your sbin directory)

4) Restart puppetdb

If that doesn't fix it, the next things I would doublecheck are the values in jetty.ini and that the IP addresses / DNS names for your hosts match up with what their certnames are.  Please let us know if this doesn't get you moving and we will be happy to assist further.

[SirHopcount]

Hi Chris

Sorry for the late response but I wanted to check some things first before responding. I made a new Puppetmaster and used this to install a new PuppetDB server. I successfully connected them together, I can do a successful puppet run on the PuppetDB node itself. When i check the puppetdb.log I can see the run:

2012-09-25 14:56:59,434 INFO  [command-proc-74] [puppetdb.command] [e1ef3a0e-e5ed-4cc6-a49e-2c9dd3caef7e] [replace facts] mgmt-puppetdb-01.edu.local
2012-09-25 14:57:03,787 INFO  [command-proc-74] [puppetdb.command] [6f07e5ca-d254-4186-a900-c5706d41e25c] [replace catalog] mgmt-puppetdb-01.edu.local
2012-09-25 15:08:57,233 INFO  [command-proc-74] [puppetdb.command] [b0cd223f-dc63-470f-a7ff-2702720f58c2] [replace facts] mgmt-puppetdb-01.edu.local
2012-09-25 15:08:58,736 INFO  [command-proc-74] [puppetdb.command] [bd776140-7c16-423c-8244-ff620346dbce] [replace catalog] mgmt-puppetdb-01.edu.local

But when I try and reach the dashboard from the browser I get the same error as before:

2012-09-25 15:18:20,934 WARN  [qtp1248545328-67] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain
2012-09-25 15:19:54,544 WARN  [qtp1248545328-63] [io.nio] javax.net.ssl.SSLHandshakeException: null cert chain

Firefox gives the following error:

The connection to mgmt-puppetdb-01.edu.local:8081 was interrupted while the page was loading.

So I can connect to Puppetmaster to the PuppetDB but I cannot reach the PuppetDB dashboard. I checked the connection with openssl client and this is the error i got:

openssl s_client -connect  mgmt-puppetdb-01.edu.local:8081
CONNECTED(00000003)
depth=0 CN = mgmt-puppetdb-01.edu.local
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = mgmt-puppetdb-01.edu.local
verify error:num=27:certificate not trusted
verify return:1
depth=0 CN = mgmt-puppetdb-01.edu.local
verify error:num=21:unable to verify the first certificate
verify return:1
140229444073120:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
 0 s:/CN=mgmt-puppetdb-01.edu.local
   i:/CN=Puppet CA: mgmt-puppetmaster-01.edu.local
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/CN=mgmt-puppetdb-01.edu.local
issuer=/CN=Puppet CA: mgmt-puppetmaster-01.edu.local
---
Acceptable client certificate CA names
/CN=Puppet CA: mgmt-puppetmaster-01.edu.local
---
SSL handshake has read 2373 bytes and written 178 bytes
---
New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: 5061AF7A33726FF51EF0CBFAD8AD3F4C88D2FFAC73E26BEFD2C0F3C722877211
    Session-ID-ctx:
    Master-Key: 85C3BF6C8830C349642BE7168E16F78873DAFE2FE6B60C842056BD65E0C9CE4633CF6C1558D6EEAA6EFDE5EA0BAE7CBF
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1348579196
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

Unfortunately SSL is not my area of expertise so I am stuck.. do you have any idea's ? If you need more information please let me know.

[SirHopcount]

I forgot to include the firefox SSL warning:

mgmt-puppetdb-01.edu.local:8081 uses an invalid security certificate.

The certificate is not trusted because no issuer chain was provided.

(Error code: sec_error_unknown_issuer)

I do accept the exception but that is when I get the error:

[Deepak Giridharagopal]

You're pointing your browser at puppetdb's HTTPS port (8081), but that
port is locked-down in terms of security; it demands a client SSL
certificate when making the connection. Because your browser doesn't
supply one, the connection is terminated by the daemon.

Here's the relevant section of the docs:

http://docs.puppetlabs.com/puppetdb/1/maintain_and_tune.html#monitor-the-performance-dashboard

You'll want to connect to port 8080 instead. That's the default
plain-text HTTP port, which will work fine in your browser. By
default, though, puppetdb binds that port to localhost instead of all
interfaces. So you've got a few options:

1) hit the dashboard using a browser on the same host running
puppetdb. that's not always possible/practical, though.

2) use an ssh tunnel to proxy a local port to port 8080 on the
puppetdb server: "ssh -NL 8080:localhost:8080
your.puppetdb.host". Then point your browser at
http://localhost:8080.

3) as the docs above mention, configure puppetdb to bind the plaintext
socket to something other than localhost (like 0.0.0.0). Then you
can just connect to port 8080 directly from your browser.

4) Theoretically, you could use "puppet cert generate" to make a new
cert for your browser, and configure your browser to use that
client certificate when trying to access the dashboard. i confess
to never having tried this, though, as I find client certificates
in browsers a serious PITA to configure. :P

I believe the vast majority of people use either #2 or #3.

deepak

--
Deepak Giridharagopal / Puppet Labs