multiple puppetmasters (w/ Passenger) behind load balancer

[CraftyTech]

Hello Gang,

     I'm working on scaling my puppet solution, and I'm deploying multiple masters w/ passenger that are going sit behind a load balancer.  If anyone is using these type of setup, would you share how you deal with the SSL certs?  I've been following Bode's Blog (http://bodepd.com/wordpress/?p=7), and it's not working to good for me.  For instance I generate the ssl cert from the 1st master, and I copy it over to the 2nd puppetmaster:

puppet1>scp private_keys/puppetmaster2.pem root@puppet2:/var/lib/puppet/ssl/private_keys/puppetmaster2.pem
puppet1>scp ca/signed/puppetmaster2.pem root@puppet2:/var/lib/puppet/ssl/certs/puppetmaster2.pem
puppet1>scp ca/ca_crt.pem root@puppet2:/var/lib/puppet/ssl/certs/ca.pem

I get the following error:

"Starting puppetmaster: Could not prepare for execution: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key"

Does anyone have a better system for handling the generation and distribution of SSL certs?

Please advise.

Thanks,

[CraftyTech]

I'm using puppet 0.25.5

[Patrick]

On Feb 8, 2011, at 1:09 PM, CraftyTech wrote:

Hello Gang,

     I'm working on scaling my puppet solution, and I'm deploying multiple masters w/ passenger that are going sit behind a load balancer.  If anyone is using these type of setup, would you share how you deal with the SSL certs?  I've been following Bode's Blog (http://bodepd.com/wordpress/?p=7), and it's not working to good for me.  For instance I generate the ssl cert from the 1st master, and I copy it over to the 2nd puppetmaster:

If you can, post the puppet.conf from "puppetmaster".  I'd guess your problem is there.

[CraftyTech]

Here's the  puppet.conf that i'm distributing to the puppetmasters:

[main]

    vardir = /var/lib/puppet

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    reports= log, foreman

    clientyamldir = /var/lib/puppet/yaml/node

    pluginsync = true

    external_nodes = /etc/puppet/external_node.rb

    node_terminus = exec

    manifest  = /etc/puppet/manifests/site.pp

    modulepath = /apps/iadmin/puppet/modules

[puppetmasterd]

    certname=puppetmaster2.dev.domain.com

    ca= false

    ssl_client_header = SSL_CLIENT_S_DN

    ssl_client_verify_header = SSL_CLIENT_VERIFY

    templatedir = /apps/iadmin/puppet/templates

    clientyamldir = /var/lib/puppet/yaml/node

    reportdir = /var/lib/puppet/reports

    storeconfigs = true

    dbadapter = mysql

    dbuser = app_puppet

    dbpassword = xxxxxxxxxxxxxx

    dbserver = puppetdb.glb.domain.com

    dbsocket = /sql/mysql/mysql.sock

    rrddir = /var/lib/puppet/rrd

    rrdinterval = $runinterval

    rrdgraph = true

[puppetd]

    report = true

    clientbucketdir = /backup/puppetClientBucketDir

    ca_server = puppetmaster1.dev.domain.com

    #node_name = facter

    listen = true

    ssldir = /etc/puppet/ssl

    #certname = puppetmaster2

My main idea is to have one puppetmaster per datacenter syncing with the main puppetmaster.  I'm going to make sure that each datacenter resolves to their local puppetmaster via load balancer appliance (bigIP). The main puppetmaster also hosts the mysql database, which is dual replicated with another server, and which is also accessed via virtual name/ Load balance.  I know it's a little ambitious but ironing out the SSL certs situation would certainly put the last nail on the coffin.

Thanks,

[Patrick]

On Feb 8, 2011, at 9:02 PM, CraftyTech wrote:

> Here's the puppet.conf that i'm distributing to the puppetmasters:
>

> [puppetmasterd]
> certname=puppetmaster2.dev.domain.com

>
>
>
> My main idea is to have one puppetmaster per datacenter syncing with the main puppetmaster. I'm going to make sure that each datacenter resolves to their local puppetmaster via load balancer appliance (bigIP). The main puppetmaster also hosts the mysql database, which is dual replicated with another server, and which is also accessed via virtual name/ Load balance. I know it's a little ambitious but ironing out the SSL certs situation would certainly put the last nail on the coffin.
>
> Thanks,

So, you tell the puppetmaster the certname is "puppetmaster2.dev.domain.com", but you're generating a cert named "puppetmaster2". You should either change one or the other.

[CraftyTech]

Thanks for your reply.  At this juncture, I'm doing it by the book and it's still failing for me.  Can someone spot what I'm doing wrong here?:

puppetmasterd --no-daemonize --verbose --certname puppet-test01.dev.domain.com

"Could not prepare for execution: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key"

Puppet.conf:

[main]

    vardir = /var/lib/puppet

    logdir = /var/log/puppet

    rundir = /var/run/puppet

    reports= log, foreman

    clientyamldir = /var/lib/puppet/yaml/node

    pluginsync = true

    external_nodes = /etc/puppet/external_node.rb

    node_terminus = exec

    manifest  = /etc/puppet/manifests/site.pp

    modulepath = /apps/iadmin/puppet/modules

[puppetmasterd]

    certname = puppet-test01.dev.domain.com

    ca= false

    ssl_client_header = SSL_CLIENT_S_DN

    ssl_client_verify_header = SSL_CLIENT_VERIFY

    templatedir = /apps/iadmin/puppet/templates

    clientyamldir = /var/lib/puppet/yaml/node

    reportdir = /var/lib/puppet/reports

    storeconfigs = true

    dbadapter = mysql

    dbuser = app_puppet

    dbpassword = CoNf1Gpupp3T

    dbserver = puppetdb.glb.domain.com

    dbsocket = /sql/mysql/mysql.sock

    rrddir = /var/lib/puppet/rrd

    rrdinterval = $runinterval

    rrdgraph = true

[puppetd]

    report = true

    clientbucketdir = /backup/puppetClientBucketDir

    ca_server = puppetmaster1.dev.domain.com

    #node_name = facter

    listen = true

    ssldir = /etc/puppet/ssl

    certname = puppet-test01.dev.domain.com

Generated from Main Puppetmaster with:

sudo /usr/sbin/puppetca --generate --certdnsnames puppet:puppet.glb.domain.com  puppet-test01.dev.domain.com

The Files are there:

puppet-test01 puppet # ls -ltr /etc/puppet/ssl/private_keys/

total 4

-rw------- 1 puppet root 887 Feb  9 10:08 puppet-test01.dev.domain.com.pem

puppet-test01 puppet # ls -ltr /etc/puppet/ssl/certs

total 8

-rw-r--r-- 1 puppet root 960 Feb  9 10:09 puppet-test01.dev.domain.com.pem

-rw-r--r-- 1 puppet root 765 Feb  9 10:11 ca.pem

[Patrick]

On Feb 9, 2011, at 7:27 AM, CraftyTech wrote:

> Thanks for your reply. At this juncture, I'm doing it by the book and it's still failing for me. Can someone spot what I'm doing wrong here?:
>
> puppetmasterd --no-daemonize --verbose --certname puppet-test01.dev.domain.com
> "Could not prepare for execution: Retrieved certificate does not match private key; please remove certificate from server and regenerate it with the current key"
>
>

> Generated from Main Puppetmaster with:
> sudo /usr/sbin/puppetca --generate --certdnsnames puppet:puppet.glb.domain.com puppet-test01.dev.domain.com
>
>
> The Files are there:
> puppet-test01 puppet # ls -ltr /etc/puppet/ssl/private_keys/
> total 4
> -rw------- 1 puppet root 887 Feb 9 10:08 puppet-test01.dev.domain.com.pem
> puppet-test01 puppet # ls -ltr /etc/puppet/ssl/certs
> total 8
> -rw-r--r-- 1 puppet root 960 Feb 9 10:09 puppet-test01.dev.domain.com.pem
> -rw-r--r-- 1 puppet root 765 Feb 9 10:11 ca.pem

With your current config file, "there" should be /var/lib/puppet/ssl, not /etc/puppet/ssl. Check by running this on puppet-test1:
puppetmasterd --genconfig | grep 'ssldir ='

It will probably return "ssldir = /var/lib/puppet/ssl" telling you the ssl directory is in "/var/lib/puppet/ssl".

[CraftyTech]

The SSLDIR is where it should be:

puppetmasterd --genconfig | grep 'ssldir ='

    ssldir = /etc/puppet/ssl

The crazy thing is that running puppetd with the same cert works fine.  Only puppetmaster is the issue.  I run it to sync up with the main puppetmaster.  

[CraftyTech]

These are the files that I copy:

puppetmaster1>cd /etc/puppet/ssl

puppetmaster1>scp private_keys/puppet-test01.pem root@puppet-test01:/etc/puppet/ssl/private_keys/puppet-test01.pem

puppetmaster1>scp ca/signed/puppet-test01.pem root@puppet-test01:/etc/puppet/ssl/certs/puppet-test01.pem

puppetmaster1>scp ca/ca_crt.pem root@puppet-test01:/etc/puppet/ssl/certs/ca.pem

To use Passenger (I'd need also ca_crt.pem and ca_crl.pem), so I also copy:

puppetmaster1>scp /etc/puppet/ssl/ca/ca_crt.pem root@puppet-test01:/etc/puppet/ssl/ca/ca_crt.pem 

puppetmaster1>scp /etc/puppet/ssl/ca/ca_crl.pem root@puppet-test01:/etc/puppet/ssl/ca/ca_crl.pem 

I don't get why it works for the client side, and not the servers side......

All ideas welcome :)

[Patrick]

The only line I see that's different that I think matters is this:

ca_server = puppetmaster1.dev.domain.com

Try putting that line in the puppetmasterd section.

[CraftyTech]

If I just leave it with

[puppetmasterd]

certname = puppet-test01.dev.domain.com

ca = falsa

it works......  it just breaks after I put in the rest of the options....  I need to work my way through all the options....

[Monachus]

I run two 0.25.5 puppetmasters behind an haproxy load balancer with
both production and QA puppetmasters on different ports on each host.
These resolve to puppet.arces.net and puppet-qa.arces.net on our load
balancer and go to host-a and host-b on 8140 for production and 8150
for QA. It all works great. We evolved up to this over the last 3.5
years going from Webrick to Mongrel to some intermediate stuff and
finally to passenger. Our puppetmasters are in the same datacenter,
so the manifest/module directory is shared via NFS. It sounds exactly
like what you're trying to do, except for the NFS part. If you're
replicating the contents of var and etc between systems, then it
shouldn't matter that it's not NFS. My puppetmasterd block in
puppet.conf looks like this:

[puppetmasterd]
vardir = /var/www/localhost/puppet/var
confdir = /var/www/localhost/puppet/etc
modulepath = $confdir/modules
bucketdir = $vardir/bucket
clientbucketdir = $vardir/clientbucket
fileserverconfig = $confdir/fileserver.conf
ssldir = $vardir/ssl
yamldir = $vardir/yaml
manifestdir = $confdir/manifests
logdest = syslog
bindaddress = 127.0.0.1
user = puppet

storeconfigs = true
dbadapter = mysql

dbuser = xxxxxxxxxx
dbname = xxxxxxxxxxx
dbpassword = xxxxxxxxxx
dbserver = xxxxxxxxxx
reports = puppet_dashboard

ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

We have a single key/cert for the master named "puppet.arces.net" (or
puppet-qa.arces.net for the QA one). I don't designate a cert name
anywhere - I just have a cert generated for the puppetmasters that
matches the hostname that the clients use to connect to the load
balancer, not a cert name for the hosts themselves. This is shared
via the NFS mount, but it could just as easily be on both servers.

My suggestion is to get it working behind the F5 with only one node.
When you have that working, copy the vardir to the other node and
switch your config to use only that node. If it still works, activate
both nodes behind the VIP and set your session persistence to bind by
clientip so you don't have flipflop behavior between requests from the
clients. You should be good to go.

Hope that helps.

Adrian Goins
Arces Network, LLC
http://www.arces.net

[CraftyTech]

Thanks Adrian !! I'll certainly try it out in QA.  On the other hand, for those who followed this thread, I've found problem, and corrected it.  It turns out, that space after any of the values, would basically break your config.  I just edited the file, and removed all extra spaces, and everything is back to normal.  I needed to have to going so that I can deploy puppetmasters via a puppet class.  Thanks kc7zzv and Adrian.

Cheers,

Henry