Combine multiple CA certificates into a bundle

[Alan Barrett]

I am struggling with using multiple puppet CAs. I think I am missing
information about which files are used for which purposes by different
parts of puppetmasterd, puppetd, puppetca, and the apache/passenger
frontend.

I have an old puppetmaster (and CA), which signed certificates for old
clients. I want to add a new puppetmaster (and CA) and let it sign
certificates for new clients. I want any client (old or new) to be able
to work with any master (old or new). I would prefer not to have to set
up a centralised CA hierarchy; the self-signed CA certificates would be
good enough if I could just figure out how to combine them into usable
bundles.

It seems to me that I should be able to take the $ssldir/ca/ca_crt.pem
files from the two puppetmasters, concatenate them to create a
ca-bundle.pem file, and place the bundle in some suitable place where
any client or server can use the bundle to verify certificates from any
CA. I haven't been able to figure out where that suitable place is.
Help?

--apb (Alan Barrett)

[Alan Barrett]

On Mon, 29 Nov 2010, Alan Barrett wrote:
> I am struggling with using multiple puppet CAs. I think I am missing
> information about which files are used for which purposes by different
> parts of puppetmasterd, puppetd, puppetca, and the apache/passenger
> frontend.

For example, which file on the puppetmaster ends up being copied
to $localcacert on the client? Neither of the files referred
to by the puppet.conf variables [puppetmasterd].cacert or
[puppetmasterd].localcacert seems to be copied to the client.

(Running puppet 0.25.5 on all masters and clients.)

--apb (Alan Barrett)

[Alan Barrett]

On Mon, 29 Nov 2010, Alan Barrett wrote:

> It seems to me that I should be able to take the $ssldir/ca/ca_crt.pem
> files from the two puppetmasters, concatenate them to create a
> ca-bundle.pem file, and place the bundle in some suitable place where
> any client or server can use the bundle to verify certificates from
> any CA. I haven't been able to figure out where that suitable place
> is. Help?

I have a ca-bundle.pem file which contains the concatenation of the ca.pem
files from the two CAs.

I make both cacert and localcacert on the puppetmaster refer to that file:

# extract frmo pupept.conf
[puppetmasterd]
cacert = $ssldir/certs/ca-bundle.pem
localcacert = $ssldir/certs/ca-bundle.pem

The first time I run puppetd on the client it creates
/etc/puppet/ssl/certs/ca.cert with the wrong contents (the cert from
the master that it's talking to atthe moment, not the bundle that I want).

I manually replace the client's /etc/puppet/ssl/certs/ca.cert with a
copy of the bundle.

Now the openssl tests described near the end of <http://projects.puppetlabs.com/projects/1/wiki/Multiple_Certificate_Authorities>
work. On the master, I run

cd /etc/puppet/ssl
openssl s_server -Verify 10 \
-cert ./certs/${server_fqdn}.pem \
-key ./private_keys/${server_fqdn}.pem \
-CAfile ./certs/ca-bundle.pem

On the client, I run:

cd /etc/puppet/ssl
openssl s_client -connect ${serevr_fqdn}:4433 -verify 10 \
-cert ./certs/${client_fqdn}.pem \
-key ./private_keys/${client_fqdn}.pem \
-CAfile ./certs/ca.pem # a copy of ca-bundle.pem

and the client successfully connects to the server, and they like
each other's keys and certificates.

If I keep exactly the same openssl s_server running on the pupeptmaster,
and run puppetd instead of openssl s_client on teh client side, it fails.
I run this command:

puppetd --onetime --test --debug \
--server=${server_fqdn} --masterport=4433

and it prints several messages about "debug: /File[foo]: Autorequiring
File[bar]", then this:

debug: Using cached certificate for ca
debug: Using cached certificate for ${server_fqdn}
debug: Using cached certificate_revocation_list for ca
debug: catalog supports formats [blah blah]; using pson
err: could not retrieve catalog from remote server: \
SSL_connect returned=1 errno=0 \
state=SSLv3 read server certificate B: certificate verify failed

On the server side (the openssl s_server process, not puppet), I see this:

ACCEPT
ERROR
29884:error:14094418:SSL routines:SSL3_READ_BYTES:\
tlsv1 alert unknown ca:s3_pkt.c:1052:SSL alert number 48
shutting down SSL
CONNECTION CLOSED

So, it seems that the puppetd client is doing something different from
the "openssl s_client" command used for testing. What certificate is
the puppetd client attempting to present, and how can I change that?

There's no apache+passenger in this test, but in reality I would be
using that on the server. This is puppet 0.25.5.

--apb (Alan Barrett)

[Patrick]

On Nov 29, 2010, at 10:13 AM, Alan Barrett wrote:

On Mon, 29 Nov 2010, Alan Barrett wrote:

It seems to me that I should be able to take the $ssldir/ca/ca_crt.pem

files from the two puppetmasters, concatenate them to create a

ca-bundle.pem file, and place the bundle in some suitable place where

any client or server can use the bundle to verify certificates from

any CA.  I haven't been able to figure out where that suitable place

is.  Help?

So, it seems that the puppetd client is doing something different from
the "openssl s_client" command used for testing.  What certificate is
the puppetd client attempting to present, and how can I change that?

Run this on the client for the config puppet is using:

puppetd --genconfig

Technically this won't actually use information passed to the puppet executable using flags what ever starts the service, but this usually doesn't matter.

If that is too much information try:

puppetd --genconfig | grep host | grep .pem

You will probably need to run these as root to get the correct config.

[Alan Barrett]

On Mon, 29 Nov 2010, Patrick wrote:
> > So, it seems that the puppetd client is doing something different from
> > the "openssl s_client" command used for testing. What certificate is
> > the puppetd client attempting to present, and how can I change that?
>
> Run this on the client for the config puppet is using:
> puppetd --genconfig

I use that all the time. The file names that I passed to "openssl
s_client" are identical to those reported by "puppetd --genconfig".

Whether or not the clientcrl file (ca_crl.pem) exists seems to have
something to do with the problem but I haven't figured out the details.
If I delete that file, then the puppetd client can connect, and it
downloads a fresh copy of the CRL, after which it can no longer connect.
I have configured certificate_revocation=false on the server, but it
nevertheless sends the CRL file to the client.

--apb (Alan Barrett)

[Eric Sorenson]

Have you perhaps revoked a certificate off one CA that matched the serial number issued by another? And perhaps the second CA issued that particular serial number to the puppet server itself?

No, I haven't done this myself, why do you ask? *whistles tunelessly*

https://projects.puppetlabs.com/issues/4948

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

- Eric Sorenson - N37 17.255 W121 55.738 - http://twitter.com/ahpook -

[Alan Barrett]

On Mon, 29 Nov 2010, Eric Sorenson wrote:
> Have you perhaps revoked a certificate off one CA that matched the serial number issued by another? And perhaps the second CA issued that particular serial number to the puppet server itself?

No, neither CA has ever revoked a certificate. "openssl crl -text"
reports that the ca_crl.pem file contains no revocations.

--apb (Alan Barrett)

[Ohad Levy]

Hi,

I think you are facing https://projects.puppetlabs.com/issues/3640#note-11

which would hopefully be resolved in https://projects.puppetlabs.com/issues/3770.

I must admit that I did not have time to continue troubleshooting this further...

Ohad

[Alan Barrett]

On Tue, 30 Nov 2010, Ohad Levy wrote:
> I think you are facing https://projects.puppetlabs.com/issues/3640#note-11
>
> <https://projects.puppetlabs.com/issues/3640#note-11>which would hopefully
> be resolved in https://projects.puppetlabs.com/issues/3770.
>
> I must admit that I did not have time to continue troubleshooting this
> further...

Thanks. I give up now.

--apb (Alan Barrett)