puppetmasterd screwing the SSL setup

[Marco Marongiu]

Dear puppeteers

I am trying to build a tree hierarchy of puppetmasters. The architecture
is aimed to distribute the load among a number of datacenters, while
keeping the puppetmasters in sync by means of puppet itself.

The architecture I am trying to build is:

- one "main puppetmaster";
- many "distribution servers", that will be client of the main
puppetmaster, and masters to other clients
- plain clients

Unfortunately puppetmasterd gets in the way (maybe thinking it's so
smart?), screwing up the SSL setup. This was discussed yesterday on IRC;
Volcano suspects that there something in the certificates is at the root
of the problem, and that's why I added a certdnsnames directive, but
with no result so far.

I am testing this setup on VirtualBox VMs on my desktop (which is
actually a luck since I can use snapshot and rewind back and forth to
different working states). The main puppetmaster is called
mastertest.oslo.osa and has address 192.168.56.108; the distribution
server is called distserver.oslo.osa and has address 192.168.56.111.
Both are on each other's /etc/hosts file.

First, I configure distserver as a plain puppet client of mastertest. A
couple of runs of puppetd --test will bring it up to speed, and it will
work as expected.

Then, on mastertest, I create a node file for distserver, which will
define it as a distribution server, and run puppetd again.
/etc/puppet/puppet.conf is rewritten so that it contains the following
certdnsnames in the puppetmasterd section:

certdnsnames="distserver.oslo.osa:distserver"

while the server directive is the puppetd section is:

server=mastertest.oslo.osa

Eventually, after the new puppet.conf is already in place, puppetmasterd
starts, and screws up the SSL setup:

> Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL key for ca
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sun Jul 05 12:44:33 UTC 2015
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Expiring the certificate cache of ca
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Removing file Puppet::SSL::Certificate ca at '/var/lib/puppet/ssl/certs/ca.pem'
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Retrieved certificate does not match private key
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Creating a new SSL certificate request for ca
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Signed certificate request for ca
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Rebuilding inventory file
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate_revocation_list for ca, good until
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for ca, good until Sat Jul 11 12:00:38 UTC 2015
> Jul 13 14:00:38 distserver puppetmasterd[2861]: Using cached certificate for distserver.oslo.osa, good until Sat Jul 11 09:25:03 UTC 2015
> Jul 13 14:00:38 distserver puppetmasterd[2888]: Reopening log files

(note the "Removing file" line...)

Now, next time I run puppetd --test, all I get is:

> Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog from remote server: undefined method `closed?' for nil:NilClass
> Jul 13 14:01:08 distserver puppetd[3212]: Not using cache on failed catalog
> Jul 13 14:01:08 distserver puppetd[3212]: Could not retrieve catalog; skipping run

Needless to say, if I stop puppetmasterd and put the old, "client" files
back in place into /var/lib/puppet, this machine starts working again as
a client.

On mastertest (which has a nginx reverse proxy to four puppetmasterd
instances, again for scalability) I see that the request from distserver
is wrong. In fact, for distserver I have:

> 192.168.56.111 - - [13/Jul/2010:15:30:09 +0200] "-" 400 0 "-" "-"

while for working clients (e.g.: mastertest itself) I have something like:

> 192.168.56.109 - - [13/Jul/2010:15:30:28 +0200] "GET /production/catalog/mastertest.oslo.osa?facts_format=b64_zlib_yaml&facts=LONG_BASE64_STRING_HERE

I honestly can't understand what is going on here...

Is there a way to make this all work as intended?

Thanks in advance

Ciao
--bronto

[Patrick Mohr]

Basically, the puppet packages you are using (and I suspect most others) assume that the client and the server on a given machine are part of the same PKI. It also might be assuming a couple of other things, but my experiments never got that far.

> Is there a way to make this all work as intended?

WARNING: This fix is almost as destructive as rm -Rf /var/lib/puppet

I think everything will just work if you set puppetd and puppetmaster to have a different "ssldir" like this:

[main]
#remove the ssldir entry from here.

[puppetmasterd]
ssldir=/var/lib/puppet/ssl_server

[puppetd]
ssldir=/var/lib/puppet/ssl_client

I won't say this is working as intended. The normal way is to make a real PKI that includes all the servers, but this is probably much easier, and will probably do what you want.

[Garrett Honeycutt]

On Jul 13, 6:54 am, Marco Marongiu <brontoli...@gmail.com> wrote:
> Dear puppeteers
>
> I am trying to build a tree hierarchy of puppetmasters. The architecture
> is aimed to distribute the load among a number of datacenters, while
> keeping the puppetmasters in sync by means of puppet itself.
>
> The architecture I am trying to build is:
>
> - one "main puppetmaster";
> - many "distribution servers", that will be client of the main
>   puppetmaster, and masters to other clients
> - plain clients
>

<snip>

> Thanks in advance
>
> Ciao
> --bronto

I suggest staying away from a hierarchical puppetmaster approach as it
adds unneeded complexity. At a former employer we solved the problem
of dealing with multiple datacenters or POP's by having a puppetmaster
(or load balanced puppetmasters) in each POP. We then had a
configuration file in our VCS that mapped POP's to branches in our
VCS. The puppetmasters would check the contents of that file and if it
changed they would do an svn switch to the new branch/tag. Through
this architecture we were able to easily update the code used by the
puppetmasters. This also gave us the ability to horizontally scale
puppetmasters as appropriate at each POP.

Working on having the code available on the Puppet Forge this week.

-g

[Marco Marongiu]

Hello

Thanks to the ones who replied.

Patrick Mohr ha scritto:

> Basically, the puppet packages you are using (and I suspect most
> others) assume that the client and the server on a given machine are
> part of the same PKI. It also might be assuming a couple of other
> things, but my experiments never got that far.

Hm, I see...

>> Is there a way to make this all work as intended?
>
>
> WARNING: This fix is almost as destructive as rm -Rf /var/lib/puppet
>
>
> I think everything will just work if you set puppetd and puppetmaster
> to have a different "ssldir" like this:
>
> [main] #remove the ssldir entry from here.
>
> [puppetmasterd] ssldir=/var/lib/puppet/ssl_server
>
> [puppetd] ssldir=/var/lib/puppet/ssl_client

That did the trick, thanks.

> I won't say this is working as intended. The normal way is to make a
> real PKI that includes all the servers, but this is probably much
> easier, and will probably do what you want.
>

I'm not sure if you're suggesting to use a single machine to host a CA
for the whole infrastructure. In case, would that scale?

Thanks

Ciao
--bronto

[Ohad Levy]

On Wed, Jul 14, 2010 at 8:50 PM, Marco Marongiu <bront...@gmail.com> wrote:

Hello

I'm not sure if you're suggesting to use a single machine to host a CA
for the whole infrastructure. In case, would that scale?

It would scale, but it would be a single point of failure for adding new puppet clients.

I would say this is your best setup at the moment (if you want to keep the revoke option), while a chained ca is possible (I'm doing it) its currently broken with the latests version of puppet.

as a side note, you might want to look at my multi puppetmaster module (a bit out of date, but maybe useful for a multi puppetmaster setup) - http://github.com/ohadlevy/puppet-multipuppetmaster

Ohad

[Marco Marongiu]

Ohad Levy ha scritto:

>
>
> On Wed, Jul 14, 2010 at 8:50 PM, Marco Marongiu <bront...@gmail.com
> <mailto:bront...@gmail.com>> wrote:
>
> Hello
>
>
> I'm not sure if you're suggesting to use a single machine to host a CA
> for the whole infrastructure. In case, would that scale?
>
> It would scale, but it would be a single point of failure for adding new
> puppet clients.

Yep. OK

> I would say this is your best setup at the moment (if you want to keep
> the revoke option), while a chained ca is possible (I'm doing it) its
> currently broken with the latests version of puppet.

I see. Thanks for the warning ;)

> as a side note, you might want to look at my multi puppetmaster module
> (a bit out of date, but maybe useful for a multi puppetmaster setup)
> - http://github.com/ohadlevy/puppet-multipuppetmaster

I gave it a glance, will look into it better tomorrow (early wakeup this
morning, ouch...). I understand it is aimed to a hiearchical setup as
the one I am building, is it?

Thanks again

Ciao
--bronto

[Cesar Avalos]

I had the same problem and just fixed with Patrick solution.  I just wanted to say thanks everybody.
--
Att,

Cesar Avalos

[Marco Marongiu]

Cesar Avalos ha scritto:

> I had the same problem and just fixed with Patrick solution. I just
> wanted to say thanks everybody.

Hello

Just as a round-up, You may want to add a [puppetca] section to your
puppet.conf, setting the same ssldir as you did in [puppetmasterd], so
that you don't have to specify the ssldir on the command line each time
you use the puppetca command.

Ciao
--bronto

--
Marco Marongiu
System Administrator - Technical Author - Perl Programmer