Puppet Scalability - Centralised Puppet SSL Cert Issues

[John Warburton]

Hi All

I am trying to use the section on Centralised Puppet Infrastructure on the Scaling Puppet page -  http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability

No matter what I do, I always end up with the client contacting a puppet server and rejecting the configuration with a dreaded "certificate verify failed":

err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

I have started from completely fresh servers, and repeated this behavior a number of times, with clean puppet configs - you can see a very detailed working below.

I am stumped as to what to do next, but suspect a number of things:
- the example given was for Mongrel - is Passenger different?
- there are a number SSL cert chaining tickets in the issues list

My goal is to have any puppet client be able to talk to any puppet server, so that if one.s designated puppet server died, we could repoint its CNAME to another puppet server in another datacentre and the client would continue working as if nothing happened. Does anyone have a working configuration that fits this scenario?

Thanks

John

I have Solaris 10 Update 8 0.25.5 puppeteer, client and server, and Apache 2.2.15 with rack and the following gems:
fastthread (1.0.7)
passenger (2.2.14)
rack (1.1.0)
rake (0.8.7)

I start with a clean config on my puppeteer:

cornadm010# nslookup puppet.example.com
Server:         1.2.3.4
Address:        4.5.6.7#53

puppet.example.com  canonical name = cornadm010.example.com.
Name:   cornadm010.example.com

cornadm010# /opt/local/sbin/puppetmasterd --server puppet.example.com --certname puppet.example.com --certdnsname `uname -n`.example.com:puppet.example.com --genconfig --vardir=/local/puppet/var --confdir=/local/puppet/etc --pluginsync --ssl_client_header=SSL_CLIENT_S_DN --ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign /local/puppet/etc/autosign.conf --node_terminus exec --external_nodes /local/puppet/bin/node_classifier.pl | sed -e 's/genconfig = true/genconfig = false/' > /local/puppet/etc/puppetmasterd.conf

cornadm010# \rm -rf /local/puppet/etc/ssl

root@cornadm010# /opt/local/sbin/puppetmasterd --no-daemonize --verbose --config /local/puppet/etc/puppetmasterd.conf
info: Creating a new SSL key for ca
info: Creating a new SSL certificate request for ca
notice: Signed certificate request for ca
notice: Rebuilding inventory file
info: Creating a new certificate revocation list
info: Creating a new SSL key for puppet.example.com
info: Creating a new SSL certificate request for puppet.example.com
notice: puppet.example.com has a waiting certificate request
info: authstore: defaulting to no access for puppet.example.com
notice: Signed certificate request for puppet.example.com
notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at '/local/puppet/etc/ssl/ca/requests/puppet.example.com.pem'
notice: Removing file Puppet::SSL::CertificateRequest puppet.example.com at '/local/puppet/etc/ssl/certificate_requests/puppet.example.com.pem'
notice: Starting Puppet server version 0.25.5


root@engnsvr002# /opt/local/sbin/puppetmasterd --server `uname -n`.example.com --certname `uname -n`.example.com --certdnsname `uname -n`.example.com --genconfig --vardir=/local/puppet/var --confdir=/local/puppet/etc --pluginsync --ssl_client_header=SSL_CLIENT_S_DN --ssl_client_verify_header=SSL_CLIENT_VERIFY --reports store --autosign /local/puppet/etc/autosign.conf --node_terminus exec --external_nodes /local/puppet/bin/node_classifier.pl | sed -e 's/genconfig = true/genconfig = false/' > /local/puppet/etc/puppetmasterd.conf

root@engnsvr002# \rm -rf /local/puppet/etc/ssl

root@engnsvr002# /opt/local/sbin/puppetmasterd --no-daemonize --verbose --config /local/puppet/etc/puppetmasterd.conf
info: Creating a new SSL key for ca
info: Creating a new SSL certificate request for ca
notice: Signed certificate request for ca
notice: Rebuilding inventory file
info: Creating a new certificate revocation list
info: Creating a new SSL key for engnsvr002.example.com
info: Creating a new SSL certificate request for engnsvr002.example.com
notice: engnsvr002.example.com has a waiting certificate request
notice: Signed certificate request for engnsvr002.example.com
notice: Removing file Puppet::SSL::CertificateRequest engnsvr002.example.com at '/local/puppet/etc/ssl/ca/requests/engnsvr002.example.com.pem'
notice: Removing file Puppet::SSL::CertificateRequest engnsvr002.example.com at '/local/puppet/etc/ssl/certificate_requests/engnsvr002.example.com.pem'
notice: Starting Puppet server version 0.25.5


root@engnsvr002# egrep example.com /tmp/openssl.cnf
commonName = engnsvr002.example.com
nsCaRevocationUrl               = https://puppet.example.com/ca_crl.pem

root@engnsvr002# openssl req -new -nodes -key /local/puppet/etc/ssl/ca/ca_key.pem -config /tmp/openssl.cnf -out /tmp/`uname -n`.example.com.csr -passin file:/local/puppet/etc/ssl/ca/private/ca.pass


puppet@cornadm010% scp root@engnsvr002:/tmp/engnsvr002.example.com.csr .

puppet@cornadm010% touch /local/puppet/etc/ssl/index

puppet@cornadm010% egrep example.com /tmp/openssl.cnf
commonName = puppet.example.com
nsCaRevocationUrl               = https://puppet.example.com/ca_crl.pem

puppet@cornadm010% /opt/local/bin/openssl ca -config /tmp/openssl.cnf -extfile /tmp/openssl.cnf -extensions v3_ca -in engnsvr002.example.com.csr -out engnsvr002.example.com.pem -passin file:/local/puppet/etc/ssl/ca/private/ca.pass -batch
Using configuration from /tmp/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 3 (0x3)
        Validity
            Not Before: Sep  1 05:09:00 2010 GMT
            Not After : Aug 29 05:09:00 2020 GMT
        Subject:
            commonName                = engnsvr002.example.com
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                70:86:83:1E:C0:73:53:F8:3D:98:BD:58:C8:A7:49:E9:81:70:2F:C3
            X509v3 Authority Key Identifier:
                keyid:FC:86:06:92:FB:99:75:EC:58:F2:83:F7:50:77:38:6F:17:62:04:74
                DirName:/CN=ca
                serial:01

            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage:
                Certificate Sign, CRL Sign
Certificate is to be certified until Aug 29 05:09:00 2020 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

puppet@cornadm010% scp engnsvr002.example.com.pem root@engnsvr002:/tmp/engnsvr002.example.com.pem

root@engnsvr002# cp /local/puppet/etc/ssl/ca/ca_crt.pem /local/puppet/etc/ssl/ca/ca_crt.pem.orig

root@engnsvr002# cp /tmp/`uname -n`.example.com.pem /local/puppet/etc/ssl/ca/ca_crt.pem

puppet@cornadm010% cat ssl/ca/ca_crt.pem
-----BEGIN CERTIFICATE-----
MIICCTCCAXKgAwIBAgIBATANBgkqhkiG9w0BAQUFADANMQswCQYDVQQDDAJjYTAe
Fw0xMDA4MzEwMjU0MjBaFw0xNTA4MzAwMjU0MjBaMA0xCzAJBgNVBAMMAmNhMIGf
MA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCuPbG6LHp/5nIEPMFQbuiqUGHedrRc
5aKJpWOAqXvAiVXnwYP6vBl+jVlxCJG4xHVaLcIIp1lHVBweyz8VwZ/aw60/2333
6v6GsLo4UYrz9a/SWKT4JNPQABBvbY/8rU7H/Yuvop3nhXBbQVMtvqCgQDFpkpx2
KYz2zXi6MJoiMQIDAQABo3kwdzA4BglghkgBhvhCAQ0EKxYpUHVwcGV0IFJ1Ynkv
T3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwDwYDVR0TAQH/BAUwAwEB/zAd
BgNVHQ4EFgQU/IYGkvuZdexY8oP3UHc4bxdiBHQwCwYDVR0PBAQDAgEGMA0GCSqG
SIb3DQEBBQUAA4GBAEk7XQV7ohMMFjzJnd+AVc/VJaw7QAUdtjJYPthlBZKv4guO
iy9BpSLZn2ChHNh1ANBAnRGIIFzljMHN6i4MXhhzfxKk6Vz0sAg74A3dE2Ots8F4
BF4BtunVFt7fyTPw/GFf3UibTM1xRXRpHq79fM5XTiuSu71pxQDCclYP2MPH
-----END CERTIFICATE-----

engnsvr003# vi /var/puppet/confdir/ssl/certs/ca.pem
<with above>

puppet@cornadm010% grep ^ServerName /local/apache-infra/conf/httpd.conf
ServerName puppet.example.com:80

puppet@cornadm010% less /local/apache-infra/conf.d/puppetmasterd.conf
<VirtualHost *:8140>
        ServerName puppet.example.com

        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile      /local/puppet/etc/ssl/certs/puppet.example.com.pem
        SSLCertificateKeyFile   /local/puppet/etc/ssl/private_keys/puppet.example.com.pem
        SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /local/puppet/etc/ssl/ca/ca_crt.pem
        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        #SSLCARevocationFile     /local/puppet/etc/ssl/ca/ca_crl.pem
        SSLVerifyClient optional
        SSLVerifyDepth  1
        SSLOptions +StdEnvVars

cornadm010# svcadm restart apache-infra

root@engnsvr002# grep ^ServerName /local/apache-infra/conf/httpd.conf
ServerName engnsvr002.example.com:80

root@engnsvr002# less /local/apache-infra/conf.d/puppetmasterd.conf
<VirtualHost *:8140>
        ServerName engnsvr002.example.com

        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile      /local/puppet/etc/ssl/certs/engnsvr002.example.com.pem
        SSLCertificateKeyFile   /local/puppet/etc/ssl/private_keys/engnsvr002.example.com.pem
        SSLCertificateChainFile /local/puppet/etc/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /local/puppet/etc/ssl/ca/ca_crt.pem
        # If Apache complains about invalid signatures on the CRL, you can try disabling
        # CRL checking by commenting the next line, but this is not recommended.
        #SSLCARevocationFile     /local/puppet/etc/ssl/ca/ca_crl.pem
        SSLVerifyClient optional
        SSLVerifyDepth  1
        SSLOptions +StdEnvVars


root@engnsvr003# mkdir /var/puppet/confdir
root@engnsvr003# /opt/local/sbin/puppetd --confdir /var/puppet/confdir --vardir /var/puppet/confdir/var --server engnsvr002.example.com --pluginsync --report --genconfig | sed -e 's/genconfig = true/genconfig = false/' > /var/puppet/confdir/puppetd.conf

root@engnsvr003# mkdir -p /var/puppet/confdir/ssl/certs

root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize --ignorecache --no-usecacheonfailure --config /var/puppet/confdir/puppetd.conf --environment lab --debug

info: Creating a new SSL key for engnsvr003.example.com
debug: Using cached certificate for ca
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Creating a new SSL certificate request for engnsvr003.example.com
warning: peer certificate won't be verified in this SSL session
debug: Using cached certificate for ca
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for engnsvr003.example.com
debug: Finishing transaction 7818336 with 0 changes
info: Retrieving plugin
debug: Using cached certificate for ca
debug: Using cached certificate for engnsvr003.example.com
err: /File[/var/puppet/confdir/var/lib]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
debug: file_metadata supports formats: b64_zlib_yaml marshal pson raw yaml; using pson
err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://engnsvr002.example.com/plugins: SSL_connect re
turned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
debug: Finishing transaction 7755204 with 0 changes
debug: catalog supports formats: b64_zlib_yaml marshal pson raw yaml; using pson
err: Could not retrieve catalog from remote server: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run


Delete & recreate ssl dirs on 002 & 003 with no chained cert, and all is OK:
root@engnsvr003# /opt/local/sbin/puppetd --verbose --onetime --no-daemonize --ignorecache --no-usecacheonfailure --config /var/puppet/confdir/puppetd.conf --environment lab

notice: running from engnsvr002.example.com on engnsvr003.example.com
notice: //Notify[running from engnsvr002.example.com on engnsvr003.example.com]/message: defined 'message' as 'running from engnsvr002.example.com on engnsvr003.example.com'

[Patrick]

On Aug 31, 2010, at 10:47 PM, John Warburton wrote:

Hi All

I am trying to use the section on Centralised Puppet Infrastructure on the Scaling Puppet page -  http://projects.puppetlabs.com/projects/1/wiki/Puppet_Scalability

No matter what I do, I always end up with the client contacting a puppet server and rejecting the configuration with a dreaded "certificate verify failed":

err: /File[/var/puppet/confdir/var/lib]: Failed to retrieve current state of resource: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed Could not retrieve file metadata for puppet://engnsvr002.example.com/plugins: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

I have started from completely fresh servers, and repeated this behavior a number of times, with clean puppet configs - you can see a very detailed working below.

I am stumped as to what to do next, but suspect a number of things:
- the example given was for Mongrel - is Passenger different?
- there are a number SSL cert chaining tickets in the issues list

My goal is to have any puppet client be able to talk to any puppet server, so that if one.s designated puppet server died, we could repoint its CNAME to another puppet server in another datacentre and the client would continue working as if nothing happened. Does anyone have a working configuration that fits this scenario?

I've done it 2 ways.

1) Just copy the ca folder to the other servers.  (Warning, breaks certificate revocation because of duplicate serial numbers)

2) Use one server as the ca for everything, but have local servers for everything else. (Not as much reliability, but close.  You can't sign when the ca goes down, but everything else works.)

I have tried using that method, but I've had horrible luck and didn't manage to make it work.

[Ohad Levy]

There is an open bug with 0.25.x (and 2.6) which breaks certificate chaining.

this works well for the 0.24.x series, and I hope that will work again sometime in the near future with 2.6.x series.

I would recommend you at the moment to use one machine as the CA, if you can accept the fact that its a single point of failure for creating new certificates.

Ohad

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[John Warburton]

Thanks Ohad

I have updated the Wiki entry with a warning (where's the <blink> tag?) and references to the bugs on certificate chaining

I'm not 100% comfortable with a single CA, so would it be possible to do the following:

ca_server = puppet-ca.example.com

rsync the ssl dir every 5 minutes to puppet-ca2.example.com

If puppet-ca dies, I would swing the CNAME over to puppet-ca2.example.com

Thanks

John

--
John Warburton
Ph: 0417 299 600
Email: jwarb...@gmail.com