A Design Question

[Paul Lathrop]

Hi all,

I'm preparing to deploy a Puppet infrastructure at my workplace, and
was wondering what you all think the best design would be. We have 2
datacenters, one on each coast (in the US). Should each DC have it's
own puppetmaster? Should I separate the CA function from the
puppetmaster function? Each DC has hundreds of systems, so I expect to
need Mongrel + load balancing, but what should I do if/when the load
grows beyond one machine in a DC?

I'd love to collect this kind of "wisdom" and add to the wiki, but I
can't seem to find it all now...

--Paul

[Larry Ludwig]

Yea I have that same question.

The book touches on this somewhat.

One option could be using rsync between two puppetmasters. I don't
see how this could not work.

[Andrew Shafer]

Paul,

The design depends on several factors concerning what your needs are
today and possibly what they will be tomorrow.

I think you would have at least a puppetmaster for each DC in most
cases. You can have puppetmasters configuring puppetmasters.

You can scale puppetmasters horizontally, just be sure you have a
workable strategy to federate the certificate handling.

<shamelessplug>
We would love to help you work through the design.
http://reductivelabs.com/services.html
</shamelessplug>

:)

Regards,
Andrew

[Paul Lathrop]

Andrew,

I think the SSL federation is the part I can't wrap my brain around.
Does anyone here have pointers on that subject?

--Paul

[Mark Foster]

Larry Ludwig wrote:
> Yea I have that same question.
>
> The book touches on this somewhat.
>
> One option could be using rsync between two puppetmasters. I don't
> see how this could not work.
>
>
>

I would not recommend this.

> On Apr 11, 1:44 pm, "Paul Lathrop" <p...@tertiusfamily.net> wrote:
>
>> Hi all,
>>
>> I'm preparing to deploy a Puppet infrastructure at my workplace, and
>> was wondering what you all think the best design would be. We have 2
>> datacenters, one on each coast (in the US). Should each DC have it's
>> own puppetmaster? Should I separate the CA function from the
>> puppetmaster function? Each DC has hundreds of systems, so I expect to
>> need Mongrel + load balancing, but what should I do if/when the load
>> grows beyond one machine in a DC?
>>
>> I'd love to collect this kind of "wisdom" and add to the wiki, but I
>> can't seem to find it all now...
>>
>>

I would keep them separate unless you happen to have have reliable
shared storage across the datacenters. You can scale out each
datacenter's puppetmaster cluster(s) using local shared storage.
The shared storage is key, so make sure it is redundant. I use NFS but
there are other options.

--
Mark Foster - Sr. Systems Engineer
BitPusher - premier managed services provider
http://www.bitpusher.com/

[Digant C Kasundra]

--On Friday, April 11, 2008 10:44:34 AM -0700 Paul Lathrop
<pa...@tertiusfamily.net> wrote:

Having multiple puppetmasters works fine for us. We have two puppetmasters
and a third machine that acts solely as the puppet CA. The puppetmasters
pull directly from subversion so there is no need to rsync between them for
us. Our reports are sent either by email or stored in a mysql database so
that doesn't need syncing either. The only thing we haven't solved is the
file bucket, so currently if you need to pull something out of the file
bucket, you would need to go find which server has that file. Luckily,
we've never actually need to pull anything out of the file buckets.

-- DK

--
Digant C Kasundra <dig...@stanford.edu>
Technical Lead, ITS Unix Systems and Applications, Stanford University

[Luke Kanies]

On Apr 11, 2008, at 1:50 PM, Paul Lathrop wrote:

> Andrew,
>
> I think the SSL federation is the part I can't wrap my brain around.
> Does anyone here have pointers on that subject?

It's either hard or it doesn't matter. The only bit that matters is
the serial number, and really, it only matters if you want to revoke
certificates.

If you just use the same CA cert everywhere, without worrying about
federation, it works but you get different hosts having the same serial.

If you want to make sure that doesn't happen, you can have a single
host do all CA operations (puppet has --cahost and --caport for this;
or maybe --caserver? I can't remember).

Or, you can set up a complicated scheme to always keep the serials in
sync.

--
This book fills a much-needed gap. -- Moses Hadas
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com

[Paul Lathrop]

On Fri, Apr 11, 2008 at 11:56 AM, Luke Kanies <lu...@madstop.com> wrote:
>
> On Apr 11, 2008, at 1:50 PM, Paul Lathrop wrote:
>
> > Andrew,
> >
> > I think the SSL federation is the part I can't wrap my brain around.
> > Does anyone here have pointers on that subject?
>
>
> It's either hard or it doesn't matter. The only bit that matters is
> the serial number, and really, it only matters if you want to revoke
> certificates.
>
> If you just use the same CA cert everywhere, without worrying about
> federation, it works but you get different hosts having the same serial.

I think that it won't matter for us. Machines will never travel
between datacenters, so I can't see any danger of having the same
serial number in two different datacenters. Am I understanding that
right?

Thanks for all your input everyone!

--Paul

[Nigel Kersten]

On Fri, Apr 11, 2008 at 11:56 AM, Luke Kanies <lu...@madstop.com> wrote:
>

> On Apr 11, 2008, at 1:50 PM, Paul Lathrop wrote:
>
> > Andrew,
> >
> > I think the SSL federation is the part I can't wrap my brain around.
> > Does anyone here have pointers on that subject?
>
>
> It's either hard or it doesn't matter. The only bit that matters is
> the serial number, and really, it only matters if you want to revoke
> certificates.
>
> If you just use the same CA cert everywhere, without worrying about
> federation, it works but you get different hosts having the same serial.
>
> If you want to make sure that doesn't happen, you can have a single
> host do all CA operations (puppet has --cahost and --caport for this;
> or maybe --caserver? I can't remember).
>
> Or, you can set up a complicated scheme to always keep the serials in
> sync.

It's ca_server and ca_port

I don't like the idea of keeping multiple CAs in sync, and so it seems
much cleaner to have a separate CA and then multiple puppet servers
for non-CA operations.

Given the flexibility of Pound, you could even set up redirection so
if your CA mongrel cluster goes down, it redirected to another
emergency CA server if signing new certs is critical for your
environment.

--
Nigel Kersten
Systems Administrator
MacOps

[Ohad Levy]

Hi,

In our environment, we have many sites and hundreds of hosts in each site, for us one of the most important points was to be able to failover to another puppet server if your local one fails.
a very simplified explanation of our environment is:

custom facts that detect from which site you are coming from.
site specific directories (when sites needs special configuration rather than the "standard" one).
a remotefile function which have the site fact as a variable in the source path.

than for the ssl stuff, we've create a server which control all puppet masters we have, we called it the puppeteer :)
it has a self sign ca, and it signs all puppet masters below it.
than, when a server is requesting a certificate from the puppetmaster (not the puppeteer) it gets a chained certificate (puppteer + puppet master).
than in our kickstart phase, after the certificate request, we add the needed entry into puppet/ssl/certs/ca.pem.
once this works, every puppet client can query every puppet server even if it didnt sign its key.

ah, one small thing, you will have to take care for the hostname matching, so most likely if a puppet server fails, you would need to change its puppet.sitefqdn to puppet.site2fqdn.

anyone else is using similar setup?
Would it be a good idea to have a SRV record for puppet ?

Ohad