Problem with Load Balancing Puppet masters with Apache mod_proxy

[kai]

I have a single LB running Apache with mod_proxy in front of a Puppet
master. These are the LB and Puppet master configs:

<Proxy balancer://puppetmaster>
BalancerMember http://192.168.1.10:8140
</Proxy>

Listen 8140

<VirtualHost *:8140>
SSLEngine on
SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
SSLProtocol -ALL +SSLv3 +TLSv1
SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP
SSLCertificateFile /var/lib/puppet/ssl/certs/
puppetlb.example.com.pem
SSLCertificateKeyFile /var/lib/puppet/ssl/private_keys/
puppetlb.example.com.pem
SSLCertificateChainFile /var/lib/puppet/ssl/certs/ca.pem
SSLCACertificateFile /var/lib/puppet/ssl/ca/ca_crt.pem
SSLCARevocationFile /var/lib/puppet/ssl/ca/ca_crl.pem
SSLVerifyClient optional
SSLVerifyDepth 1
SSLOptions +StdEnvVars

RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

<Location />
SetHandler balancer-manager
Order allow,deny
Allow from all
</Location>

ProxyPass / balancer://puppetmaster/
ProxyPassReverse / balancer://puppetmaster/
ProxyPreserveHost On

</VirtualHost>

Ant the back-end Puppet master:

Listen 8140

<VirtualHost *:8140>
SSLEngine off

# Obtain Authentication Information from Client Request
Headers
SetEnvIf X-Client-Verify "(.*)" SSL_CLIENT_VERIFY=$1
SetEnvIf X-SSL-Client-DN "(.*)" SSL_CLIENT_S_DN=$1

RackAutoDetect On
DocumentRoot /etc/puppet/rack/puppetmaster/public/
<Directory /etc/puppet/rack/puppetmaster/>
Options None
AllowOverride None
Order allow,deny
allow from all
</Directory>
</VirtualHost>

On the puppet agent when I initially request a cert, all works well
and the Puppet master back-end receives it and I can sign it on the
master with puppetca -s hostname.
After this when i run puppetd -tv on the agent I get this error:

err: Could not retrieve catalog from remote server: SSL_connect
returned=1 errno=0 state=SSLv3 read server certificate B: certificate
verify failed. This is often because the time is out of sync on the
server or client

Also, when I run this on the LB and the master:

tcpdump -s 1024 -l -A port 8140 -i eth1

I see that headers are not set at all:

X-SSL-Subject: (null)
X-SSL-Client-DN: (null)
X-Client-Verify: NONE

Any idea what I am missing?

[kai]

Puppet version 2.7.14 on Ubuntu.
My puppet master config:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates

[master]
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

And the puppet agent config:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates

[agent]
server = puppetlb.example.com
runinterval = 1800
splay = false
configtimeout = 1200
noop = false
autoflush = true
report = false

[Jeff McCune]

Is the same CA is being used to issue the lb certificate and issue the
agent certificate?

Could you paste the output of `puppet cert print puppetlb.example.com`
and again for the agent you're seeing the error on? `puppet cert
print <agent_certname>`

-Jeff

> --
> You received this message because you are subscribed to the Google Groups "Puppet Users" group.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
>

[david....@gmail.com]

Yeah,

Good question: Sorry for the interjection. I would like to create a none puppet generated CA and intermediate CA for my puppet master. I tried but failed. Does anyone have a procedure or has anyone done this?

Thanks,
Dave Garvey

--
David Garvey

[david....@gmail.com]

If puppet is to be enterprise than I would think we should be able to use a CA generated for the organization?

--
David Garvey

[david....@gmail.com]

You would have to sync serial number too?

--
David Garvey

[david....@gmail.com]

Well everything would have to be synced...

--
David Garvey

[kai]

Jeff,

So the way I solved this is by exporting the /var/lib/puppet/ssl directory from the Puppet CA and mounting it on the LB, then making sure that the Puppet Masters have "certname = puppetlb.example.com" in their puppet.conf files.
I wonder if there's better way to do this, as the LB actually terminates the SSL traffic.

[Jeff McCune]

On Fri, Jun 15, 2012 at 6:27 AM, kai <kaiv...@gmail.com> wrote:

Jeff,

So the way I solved this is by exporting the /var/lib/puppet/ssl directory from the Puppet CA and mounting it on the LB, then making sure that the Puppet Masters have "certname = puppetlb.example.com" in their puppet.conf files.
I wonder if there's better way to do this, as the LB actually terminates the SSL traffic.

You don't need the full CA available on the load balancer.  You only need a signed certificate and the private key associated with that certificate.

What I normally do when I help customers set this up is log into the CA system and run `puppet cert generate puppetlb.example.com --dns_alt_names all,other,names,this,loadbalancer,could,be,accessed,by

I then securely transfer the signed certificate and private key file for puppetlb.example.com from the CA to the load balancer and simply use that.

You also need the CA certificate (but _not_ the private key) so the load balancer can verify the authenticity of connecting agents.

These three files correspond to these Apache configuration options:

SSLCertificateFile SSLCertificateKeyFile SSLCertificateChainFile

Hope this helps,

-Jeff

[kai]

If the LB does not have all the signed agent's certificates, how will it know which agent is valid. All the signed certs are stored on the CA which is behind the LB.
I'll try and figure out how to just copy the signed certificate and the private key associated with that certificate from the CA to the LB and update the little article I wrote at http://kaivanov.blogspot.com/2012/06/load-balancing-puppet-with-apache.html

Thanks for your pointers  Jeff I appreciate it!!

[Jeff McCune]

On Fri, Jun 15, 2012 at 11:51 AM, kai <kaiv...@gmail.com> wrote:

If the LB does not have all the signed agent's certificates, how will it know which agent is valid. All the signed certs are stored on the CA which is behind the LB.

The same way your web browser knows https://www.puppetlabs.com is valid without having Puppet Labs' certificate.  There's a trusted third party involved, the certification authority (CA).  Your web browser trusts the CA that issued our certificate, not our certificate directly.

You might want to check out http://en.wikipedia.org/wiki/Trusted_third_party and http://en.wikipedia.org/wiki/X.509

 

I'll try and figure out how to just copy the signed certificate and the private key associated with that certificate from the CA to the LB and update the little article I wrote at http://kaivanov.blogspot.com/2012/06/load-balancing-puppet-with-apache.html

Thanks for your pointers  Jeff I appreciate it!!

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/cKaK0eVhyOYJ.

[kai]

I get it now! Since the CA signed the agents cert the LB knows that the agent cert is valid because the LB has the CA cert and key to validate with. So, what is the point of the CA storing all the signed agent certs?

[Jeff McCune]

On Fri, Jun 15, 2012 at 1:27 PM, kai <kaiv...@gmail.com> wrote:

I get it now! Since the CA signed the agents cert the LB knows that the agent cert is valid because the LB has the CA cert and key to validate with. So, what is the point of the CA storing all the signed agent certs?

Yes, but one more detail; the LB doesn't need the CA [private] key.  It only needs the [public] cert to verify the authenticity of the agent.

Trusted third party is all built on top of public key cryptography.  The idea is that two parties can securely exchange information without knowing anything "private" about one another.  They need only the public keys.

SSH is a good example of a public key system that isn't trusted third party.  You upload your public key to a server and that server is then able to identify you.

With SSL certificates, the server doesn't have a copy of the client's public key.  So we need some other way to verify the client is who they say they are.  This is where the third party comes into the picture.  The third party (the CA) uses it's private key to digitally sign the public key of the client.  This is a certificate.  The CA's private key is not transferred to anyone, but the signed public key is transfered back.

When the client connects to a sever, it presents it's signed public key.  The server uses the public key of the CA (NOT the private key) to verify the public key of the client is actually signed by the CA.  At this point trust is established.

So...  You've got it, except the LB should not have the CA's private key.

-Jeff

[kai]

I understand that only the CA cert needs to be copied on the LB and not the private key, as the private key is just for signing the agents certificates. Just wanted to note that the CA also needs SSLCARevocationFile, for revocation to work it seems.
The only other concept that is not clear to me is why the CA keeps all the signed agents certs? Is there any need for that.

[Jeff McCune]

On Mon, Jun 18, 2012 at 7:06 AM, kai <kaiv...@gmail.com> wrote:

I understand that only the CA cert needs to be copied on the LB and not the private key, as the private key is just for signing the agents certificates. Just wanted to note that the CA also needs SSLCARevocationFile, for revocation to work it seems.
The only other concept that is not clear to me is why the CA keeps all the signed agents certs? Is there any need for that.

Puppet keeps the signed certificates as a record of what's been signed.  The signed agent certificates are for informational purposes only and aren't required to be kept around.

-Jeff

[kai]

Jeff, thank you very much for taking the time to answer all my questions. I really appreciate it. This thread had helped me a lot in my journey to mastering Puppet.
Thank you again!