SSL error when after restarting httpd. CA information missing/or wrong?

[Tom Hallam]

Hi everybody

I've been running puppet under Apache/Passenger for some time.  I restarted Apache this morning I'm getting the following error

---
[root@ecm-rhl-032 ~]# puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [self signed certificate for /C=--/ST=SomeState/L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/CN=ECM-RHL-001/emailAddress=root@ECM-RHL-001]
Info: Retrieving pluginfacts
---

If I switch back to the WEBrick server it all works...
Apache/Passenger configuration has not been changed.

Digging into SSL, it looks like the CA information is missing when I run Apache/Passenger but not when I run WEBrick.

Where does WEBrick get its Certificate information from?  How do I check what values it's using?
Where does the Puppet agent get its Certificate information from?  How do I check what values it's using?

Tom

[Tom Hallam]

It took a bit of digging - but I did workout what the issue was.  I'll give the solution first and then an overview of how I diagnosed the issue  It may be useful to others with SSL issues.

I was running Puppet on a Red Hat server.  By default Red Hat installs an SSL configuration in /etc/httpd/conf.d. The Puppet server's default configuration is also installed in /etc/httpd/conf.d and loads before the SSL configuration.  I'd moved the puppet configuration (a virtual host / site ) to /etc/httpd/sites-available so it now loads after the SSL configuration and is in the same location as other sites / virtual hosts.  Unfortunately the default Red Hat SSL configuration includes a virtual host with a self signed certificate.  Unlike other SSL configuration these are not commented out. 

SSL certificates are selected on a per IP basis.  The first one loaded by Apache for an IP takes precedence.  The Red Hat SSL certificate was therefore loading and the one in the Puppet configuration was being silently ignored.  The solution was simple: comment out the unused virtual host configuration in the default Red Hat SSL configuration.

I'll follow up later on how I debugged this issue.

Tom

[Tom Hallam]

OK, so this is a straight line version of how I debugged the issue.  There were a couple more dead ends than described here.

The first thing to notice is the error. I was getting: "certificate verify failed: [self signed certificate for /C=--/ST=SomeState/L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/CN=ECM-RHL-001/emailAddress=root@ECM-RHL-001".
This told me a couple of things:

• The certificate verification was failing.
• The identity given in the certificate was: "/C=--/ST=SomeState/L=SomeCity/O=SomeOrganization/OU=SomeOrganizationalUnit/CN=ECM-RHL-001/emailAddress=root@ECM-RHL-001".

I checked the configuration of the Puppet virtual site.  It all looked OK. I then turned to openssl (https://www.openssl.org/docs/apps/x509.html) to check the certificates.  I used the following (substituting the certificate file for cert.pem) to print out the server and CA certificates used in the configuration:

openssl x509 -in cert.pem -noout -text 

I decided to have a look at the certificates I got if I connected to the server.  Again openssl is your friend (http://stackoverflow.com/questions/7885785/using-openssl-to-get-the-certificate-from-a-server).  I ran the following with output to separate files for the WEBrick and Apache servers.

openssl s_client -showcerts -connect ecm-rhl-001:8140 </dev/null >WEBrick.certs
openssl s_client -showcerts -connect ecm-rhl-001:8140 </dev/null >Apache.certs

This gave me 2 files each with 2 certificates in it.  The first certificate is the server certificate and the second is the CA certificate. A brief inspection of the files showed that the first certificate was different whilst the second the same: ie the server certificates differed but the CA certificates were the same. I confirmed this further by splitting each certificate out to a separate file and using openssl to print text representations as per check on configured certificates above.

So where were the different certificates coming from?  Openssl (again) lets me print out and md5 for a certificate and a private key.  The md5 will match for matching certificates and keys.

openssl x509 -noout -modulus -in signed/ecm-rhl-001.pem | openssl md5
openssl rsa -noout -modulus -in private_keys/ecm-rhl-001.pem | openssl md5

This allowed me to check that the certificate and private keys matched.  It also gave me a quick way to check what keys the certificates that I retrieved from the servers matched.  All checked out OK.  I still could not locate the illusive certificate that was being returned by the Apache server.  I checked a number of possible keys and

So I went back to look at the Apache configuration and checked all of them for 'SSLCertificateFile' using grep. This showed the that "SSL Configuration" had a certificate defined in it by default.  A quick check using the previous Openssl commands showed that it matched the one returned by the Apache server.  A few comments to remove the unwanted site from the Apache configuration and everything was working again.


I've installed ssl on a number of Apache servers (debian package, ubuntu package & compiled from source) over the last 15 years or so and this is the first time the ssl configuration has included a "live" ssl virtual host definition.


Tom