Redundant Puppet Master Servers

[Eric Singer]

Hi,
I'm the UNIX systems administrator at the company I work for and we're
in the planning stage of setting up LDAP and Puppet servers. For
high availability we're current thinking about dedicating two of our
lower end Linux servers for the LDAP implementation and I'm thinking
both servers should be Puppet Master servers as well. One server
being primary and the other is seconday which is ready to pick up
incase the other fails.

LDAP doesn't appear to need any special cluster failover software to
perform this kind of primary with secondary backup setup, but I can't
find any mention of high availability setups with the Puppet Master
server.

Thank You,

Eric Singer

[James Turnbull]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The only documentation I know of is:

http://reductivelabs.com/trac/puppet/wiki/PuppetScalability

It's not really what you're after but someone on the list is bound to do
something similar. Peoples?

Regards

James Turnbull

- --
Author of:
* Pulling Strings with Puppet
(http://www.amazon.com/gp/product/1590599780/)
* Pro Nagios 2.0
(http://www.amazon.com/gp/product/1590596099/)
* Hardening Linux
(http://www.amazon.com/gp/product/1590594444/)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJT44C9hTGvAxC30ARAinVAKCsirDS1efGFOSghO82hsOUeNjqxACg0gvi
wMTKw3OmufR0cuzJQBnW6Vc=
=uXQJ
-----END PGP SIGNATURE-----

[Nigel Kersten]

On Mon, Dec 22, 2008 at 4:54 AM, James Turnbull <ja...@lovedthanlost.net> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Eric Singer wrote:
> Hi,
> I'm the UNIX systems administrator at the company I work for and we're
> in the planning stage of setting up LDAP and Puppet servers.   For
> high availability we're current thinking about dedicating two of our
> lower end Linux servers for the LDAP implementation and I'm thinking
> both servers should be Puppet Master servers as well.   One server
> being primary and the other is seconday which is ready to pick up
> incase the other fails.
>
> LDAP doesn't appear to need any special cluster failover software to
> perform this kind of primary with secondary backup setup, but I can't
> find any mention of high availability setups with the Puppet Master
> server.

The only documentation I know of is:

http://reductivelabs.com/trac/puppet/wiki/PuppetScalability

It's not really what you're after but someone on the list is bound to do
something similar.  Peoples?

If you want to do failover with puppet servers and you are using environments, there's a major gotcha that I really should add to that page...

If a puppet client connects to server A, downloads the compiled manifest, and then starts requesting files via the puppet:/// protocol, and in the middle of all these short lived requests the server switches over to Server B, that server doesn't necessarily know what environment the client should be using, as that is stored in a file on the server.

The only feasible solution is to somehow share those state files between servers. This may or may not be feasible in your environment (no pun intended...)

 

--
Nigel Kersten
Systems Administrator
Tech Lead - MacOps

[RijilV]

2008/12/22 James Turnbull <ja...@lovedthanlost.net>

Eric Singer wrote:
> both servers should be Puppet Master servers as well.   One server
> being primary and the other is seconday which is ready to pick up
> incase the other fails.
>

The only documentation I know of is:

http://reductivelabs.com/trac/puppet/wiki/PuppetScalability

It's not really what you're after but someone on the list is bound to do
something similar.  Peoples?

Regards

James Turnbull

We set this up at my last job, though not in an active-standby setup (active-active-active at the core, and active-active at each site).  Its not terribly difficult if you setup your CA correctly.  You'll want whatever servers are responsible for issueing client certs to be all signed by the same root authority, and place all of those certs in the ca.pem on every puppet master.  If you sign all of your puppet master certs with the same key, you just have to place that key in all of your clients.    We ended up generating our own puppet master certs with openssl because we needed additional names in the Server Alternate Names, but you can just use the one puppetmaster creates if you're okay with the SAN it uses.  You will also likely need to switch from webrick to mongrel and something that supports SSL cert chains (apache), unless that got fixed in webrick since 6 months ago.

Since puppet doesn't have a concept of producers and consumers (or master and slaves) already, I would suggest against working yourself into that mindset with respect to scalability - view it as a clustered service and go from there.  

If you don't want to deal with all the certs and don't mind the active/standby idea, you could just copy all the certs from puppetA to puppetB, and flip an alias/cname when puppetA goes down.  its not a great solution because the cert name on puppetB isn't really valid so you won't be able to easily test against that server unless you make a host entry.

.r'

[RijilV]

2008/12/22 RijilV <rij...@riji.lv>

Since puppet doesn't have a concept of producers and consumers (or master and slaves) already, I would suggest against working yourself into that mindset with respect to scalability - view it as a clustered service and go from there.  

Er what I ment to write was 'since puppetmaster doesn't have...'   You just have manifests and there isn't a notion of a 'root' server that holds some type of special manifest that gets synced out to other puppet servers.  You have to do that syncin' on your own, unlike openlap.

.r'

[Adam Jacob]

On Dec 22, 2008, at 4:49 AM, Eric Singer wrote:

>
> Hi,
> I'm the UNIX systems administrator at the company I work for and we're
> in the planning stage of setting up LDAP and Puppet servers. For
> high availability we're current thinking about dedicating two of our
> lower end Linux servers for the LDAP implementation and I'm thinking
> both servers should be Puppet Master servers as well. One server
> being primary and the other is seconday which is ready to pick up
> incase the other fails.

You can probably do this with DRBD and Heartbeat. You want to put
your Puppetmaster's configs, SSL certificates, manifests, etc all on
the DRBD. Let heartbeat swap the DRBD, VIP, and Puppet between the
two as usual.

Adam

[Eric Singer]

Thank you for all the replies. The failover cluster approach seems
like it's going to be the best answer for us. Although these server
are low end, we do have HBAs we can put in them and present LUNs from
our shared storage. The VIP, volumes and mounts could just failover
to the other server.

[Digant C Kasundra]

> If you want to do failover with puppet servers and you are using
> environments, there's a major gotcha that I really should add to that
> page...
>
>
> If a puppet client connects to server A, downloads the compiled
> manifest, and then starts requesting files via the puppet:///
> protocol, and in the middle of all these short lived requests the
> server switches over to Server B, that server doesn't necessarily know
> what environment the client should be using, as that is stored in a
> file on the server.
>
>
> The only feasible solution is to somehow share those state files
> between servers. This may or may not be feasible in your environment
> (no pun intended...)

How can the state files be shared between servers?

--
Digant C Kasundra <dig...@stanford.edu>
Technical Lead, ITS Unix Systems and Applications, Stanford University

[Nigel Kersten]

On Tue, Jan 6, 2009 at 2:13 PM, Digant C Kasundra <dig...@stanford.edu> wrote:

> If you want to do failover with puppet servers and you are using
> environments, there's a major gotcha that I really should add to that
> page...
>
>
> If a puppet client connects to server A, downloads the compiled
> manifest, and then starts requesting files via the puppet:///
> protocol, and in the middle of all these short lived requests the
> server switches over to Server B, that server doesn't necessarily know
> what environment the client should be using, as that is stored in a
> file on the server.
>
>
> The only feasible solution is to somehow share those state files
> between servers. This may or may not be feasible in your environment
> (no pun intended...)

How can the state files be shared between servers?

The simplest way is probably to just use an NFS share for that location I reckon.

I was thinking about other options, but I haven't tested this extensively, so am not sure if there are concurrency issues.

[Adam Jacob]

On Jan 6, 2009, at 2:13 PM, Digant C Kasundra wrote:
>>
>> If a puppet client connects to server A, downloads the compiled
>> manifest, and then starts requesting files via the puppet:///
>> protocol, and in the middle of all these short lived requests the
>> server switches over to Server B, that server doesn't necessarily
>> know
>> what environment the client should be using, as that is stored in a
>> file on the server.
>>
>>
>> The only feasible solution is to somehow share those state files
>> between servers. This may or may not be feasible in your environment
>> (no pun intended...)
>
> How can the state files be shared between servers?

DRBD.

Adam

[Jeffrey Hulten]

I would agree. With DRBD and uCARP almost any service can be made to
work as a active-passive cluster. It is not for the faint of heart
tho... Of course if you are managing systems with Puppet you probably
aren't faint of heart. ;)

Please note... This is not a solution for load balancing. Only one
machine at a time can mount the partition...

[Adam Jacob]

On Jan 7, 2009, at 11:10 AM, Jeffrey Hulten wrote:

>
> I would agree. With DRBD and uCARP almost any service can be made to
> work as a active-passive cluster. It is not for the faint of heart
> tho... Of course if you are managing systems with Puppet you probably
> aren't faint of heart. ;)
>
> Please note... This is not a solution for load balancing. Only one
> machine at a time can mount the partition...

You could even pull this off with DRBD and good old Heartbeat with a
custom init handler.

Adam

[Digant C Kasundra]

> > How can the state files be shared between servers?
>
> DRBD.

Considering a load-balanced environment, it seems this might not be optimal. Perhaps the puppetmaster need the ability to store the state information in a database.

[Adam Jacob]

On Jan 8, 2009, at 1:00 PM, Digant C Kasundra wrote:
>>> How can the state files be shared between servers?
>>
>> DRBD.
>
> Considering a load-balanced environment, it seems this might not be
> optimal. Perhaps the puppetmaster need the ability to store the
> state information in a database.

A document database, a-la CouchDB, would be great here.

Adam

[Aaron Lippold]

Hi,

Has anyone done this across multiple site, let's say, 13 sites, with a
primary-primary, backup-backup at each site?

How far does DRDB and Heartbeat take you across sites?

A

[Jeffrey Hulten]

DRBD is really intended for local networks.

Is there anything preventing a puppetmaster being a node of a
different puppetmaster? You could pass config that way.