High Availability of Puppet server for separate geographical location

[shyam sundar Keshari]

Hi Team,

I have to configure puppet server in Primary-Secondary mode for 2
distributed location .

Site A is already running 1 Puppet server .Now I need to configure
another puppet secondary server at site B ,so that

all client at location B ,only connect to that server .And site A
puppet client only connect to site A .

But mine requirement is of that suppose site A server goes down then
Site B server handle all client request and

same if Site B server down then server A handles all .

Kindly guide me to implement this scenario .

Thanks in Advance

Shyam

[jcbollinger]

For the most part, this is outside Puppet's scope.  You need to stand up some kind appliance or service in front of your Puppet servers to monitor when one goes down, so as to redirect traffic to the other when needed.  That cannot be part of Puppet, else it would go down with Puppet if ever Puppet went down.

With an external failover system in place, there would still be a few things to arrange on the Puppet side.  In order for site B to properly serve site A's nodes, it must have

• site A's SSL certificate, or at least a certificate that site A's nodes will trust;
  
• ENC and/or node declarations for site A's nodes, matching those on site A;
• modules (and standalone classes, if any) matching those on site A;
  
• external data (if applicable) appropriate for site A's nodes; and
  
• stored configs for site A, if applicable.
  

I see two reasonably good approaches here:

1. the site A and B masters are maintained as a mirrored pair.  This has the effect that from Puppet's perspective, physical sites A and B are meaningless partitions of a single large, federated site.  A shared, central certificate authority would be required (and that would need to be the site A master if you want to avoid issuing new certificates to all the site A nodes).
   
2. A mirror of site A's master is maintained as a hot spare at site B, and vice versa.  Each spare could run on the same (possibly virtual) hardware as the local master (but on a different port), or it could run on different hardware and merely be geographically colocated.  This would maintain a logical distinction between sites A and B, and would be overall easier to set up and maintain.

If you consider option 2, however, you should soon realize that in that case you don't get any particular advantage from putting the spare masters on the same machine as the local site's main master.  Indeed, it might be easier to set them up as ordinary nodes (for the site in which they are actually located), and even to let the local site's masters have responsibility for maintaining the spares' puppet master config, each as a mirror of the other site's master.


John

[R.I.Pienaar]

> - site A's SSL certificate, or at least a certificate that site A's
> nodes will trust;
> - ENC and/or node declarations for site A's nodes, matching those on
> site A;
> - modules (and standalone classes, if any) matching those on site A;
> - external data (if applicable) appropriate for site A's nodes; and
> - stored configs for site A, if applicable.

SRV goes a long way towards solving some of this, it is possible to share
a CA between masters too which solves some of that. Gets a lot harder when
you involve PuppetDB.

http://docs.puppetlabs.com/guides/scaling_multiple_masters.html

>
> I see two reasonably good approaches here:
>

> 1. the site A and B masters are maintained as a mirrored pair. This has

> the effect that from Puppet's perspective, physical sites A and B are
> meaningless partitions of a single large, federated site. A shared,
> central certificate authority would be required (and that would need to be
> the site A master if you want to avoid issuing new certificates to all the
> site A nodes).

> 2. A mirror of site A's master is maintained as a hot spare at site B,

> and vice versa. Each spare could run on the same (possibly virtual)
> hardware as the local master (but on a different port), or it could run on
> different hardware and merely be geographically colocated. This would
> maintain a logical distinction between sites A and B, and would be overall
> easier to set up and maintain.
>
> If you consider option 2, however, you should soon realize that in that
> case you don't get any particular advantage from putting the spare masters
> on the same machine as the local site's main master. Indeed, it might be
> easier to set them up as ordinary nodes (for the site in which they are
> actually located), and even to let the local site's masters have
> responsibility for maintaining the spares' puppet master config, each as a
> mirror of the other site's master.
>
>
> John
>

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to puppet-users...@googlegroups.com.
> To post to this group, send email to puppet...@googlegroups.com.
> Visit this group at http://groups.google.com/group/puppet-users?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

[Martin Langhoff]

On Mon, Apr 29, 2013 at 9:55 AM, shyam sundar Keshari
<shya...@gmail.com> wrote:
> I have to configure puppet server in Primary-Secondary mode for 2
> distributed location .

I am in a similar situation. Not liking the options available, I am
building "puppetgit"
https://groups.google.com/forum/?fromgroups#!topic/puppet-users/OilxMytnD_k

cheers,



m
--
martin....@gmail.com
- ask interesting questions
- don't get distracted with shiny stuff - working code first
~ http://docs.moodle.org/en/User:Martin_Langhoff

[Felipe Salum]

Is Puppetlabs planning some easy solution for this ?

The real problem I see is that you can't separate the Puppet CA from the Puppet Master. So even that you can have multiple puppet masters, the CA must run in one of them. So if that server goes down your multiple puppet master setup is screwed as well.

I would love to see a Puppet CA high availability solution out of the box.

[John Warburton]

On 9 May 2013 05:57, Felipe Salum <fsa...@gmail.com> wrote:

Is Puppetlabs planning some easy solution for this ?

I run 12 puppet servers around the world. They work in a multiple puppet master solution where any client from any location can work with any puppet server in any location with dns_alt_names. We have an easy/simple solution:

One puppet server is designated puppet-ca.example.com. All client's configuration files look like this:

    ca_server = puppet-ca.example.com

A second puppet server is randomly chosen to be puppet-ca2.example.com. A rsync job runs every minute on puppet-ca2 to only suck down the .../etc/ssl/ca directory from puppet-ca

If puppet-ca becomes unavailable, we move the puppet-ca CNAME to puppet-ca2. That lag is acceptable to us. You may chose to use other load balancing options like an F5

John

[Felipe Salum]

Yes it works perfectly, I run 2 puppet servers and one of them is the CA using a CNAME as well for the puppet master/ca.

After it is set and working it is wonderful, however until you figure out that you need to remove the ssldir, then run puppet with dns_alt_names, then sign with dns_alt_names in the other side, then start the apache2/passenger, then do a lot of more other tricks until you get out of the certificate error messages it is a headache :)

Easy solution would be something that would be easier to setup, or maybe it is just me trying to complicate things. :)

[John Warburton]

On 9 May 2013 10:59, Felipe Salum <fsa...@gmail.com> wrote:

Easy solution would be something that would be easier to setup, or maybe it is just me trying to complicate things. :)

I suppose all HA solutions are difficult - they have to suit your site's definition of HA and meet cost/complexity needs. Sometime a one pager set of instructions is good enough, but Puppet Labs aren't going to be able to meet everyone's HA needs

I was lucky that this was my second puppet deployment and I had a design that was HA from the start - so built it all in from the start. Keep plugging away

John

[Martin Langhoff]

On Thu, May 9, 2013 at 12:06 AM, John Warburton <jwarb...@gmail.com> wrote:
> I suppose all HA solutions are difficult

Nah. A service correctly designed to be resilient can be HA with
trivial investment.

DNS is a good example. It may have blemishes but nobody stresses about
its availability. Setup as many tiers of redundancy as you want,
easily.

Puppet has no need to be centralized -- a git-based puppet setup can
handle it just fine.

I am writing some tooling for git+puppet (search for ppg in recent
posts to this list), and it's trivial to add N-tiers of redundant
servers...

[Martin Langhoff]

On Thu, May 9, 2013 at 10:42 AM, Martin Langhoff
<martin....@gmail.com> wrote:
> I am writing some tooling for git+puppet (search for ppg in recent
> posts to this list), and it's trivial to add N-tiers of redundant
> servers...

Heh, so trivial in fact that you can use round-robin DNS and it'll just work :-)

I looked into adding the "feature" to ppg... then realized that git
plays very well with round-robin DNS.

[Ramin K]

On 5/9/2013 7:42 AM, Martin Langhoff wrote:
> On Thu, May 9, 2013 at 12:06 AM, John Warburton <jwarb...@gmail.com> wrote:
>> I suppose all HA solutions are difficult
>
> Nah. A service correctly designed to be resilient can be HA with
> trivial investment.
>
> DNS is a good example. It may have blemishes but nobody stresses about
> its availability. Setup as many tiers of redundancy as you want,
> easily.
>
> Puppet has no need to be centralized -- a git-based puppet setup can
> handle it just fine.
>
> I am writing some tooling for git+puppet (search for ppg in recent
> posts to this list), and it's trivial to add N-tiers of redundant
> servers...

Hubris, today thy name is Martin. :-)

I'd argue that people have stressed about DNS availability for just
under three decades and that we are currently enjoying the fruits of
that labor. Personally, I have yet to work at a company where DNS has
not caused a significant outage. I do agree that the tools are there to
build resilience, but implementation matters and it is a hard problem to
solve in a non trivial environment.

Your ppg tooling does look interesting, but there is a large trade off
in functionality. You could use some other systems to collect data and
push it into a repo which is then queried in order to generate data for
clients... starting to sound an awful lot like centralized Puppet to me
and all the problems therein regardless of transport.

Ramin

[Martin Langhoff]

On Thu, May 9, 2013 at 2:31 PM, Ramin K <ramin...@badapple.net> wrote:
> Hubris, today thy name is Martin. :-)

Fair enough. I am happy about the tool I am writing (almost finished!)
but, as the followup post makes clear, it isn't about the designe of
ppg. It is about the design of git.

> I'd argue that people have stressed about DNS availability for just
> under three decades and that we are currently enjoying the fruits of that
> labor. Personally, I have yet to work at a company where DNS has not caused
> a significant outage.

I am really surprised at your statement. Of course mishaps can happen,
or someone can mess up configuration DNS royally. But setting up a
primary and secondary setup is trivial.

SMTP and LDAP are also examples where resilience was baked into the
design. With those two, the quality of implementation, and
complications in setup make for a lot more breakage.

Compare to HTTP, databases etc where there's a whole industry of tools
to make things somewhat reliable.

Maybe we are talking about different things.

> Your ppg tooling does look interesting, but there is a large trade
> off in functionality

What is the loss of functionality you see? Anything that you use in practice?

(Reading here https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ
I can't spot anything major, but I may be missing something...)

[Ramin K]

On 5/9/2013 1:51 PM, Martin Langhoff wrote:
> On Thu, May 9, 2013 at 2:31 PM, Ramin K <ramin...@badapple.net> wrote:
>> Hubris, today thy name is Martin. :-)
>
> Fair enough. I am happy about the tool I am writing (almost finished!)
> but, as the followup post makes clear, it isn't about the designe of
> ppg. It is about the design of git.

This is where I think we diverge. :-) As someone with a fair amount of
operational experience it's not about the design of git, it's about the
implementation created on top of git. Or Puppet.

>> I'd argue that people have stressed about DNS availability for just
>> under three decades and that we are currently enjoying the fruits of that
>> labor. Personally, I have yet to work at a company where DNS has not caused
>> a significant outage.
>
> I am really surprised at your statement. Of course mishaps can happen,
> or someone can mess up configuration DNS royally. But setting up a
> primary and secondary setup is trivial.
>
> SMTP and LDAP are also examples where resilience was baked into the
> design. With those two, the quality of implementation, and
> complications in setup make for a lot more breakage.
>
> Compare to HTTP, databases etc where there's a whole industry of tools
> to make things somewhat reliable.
>
> Maybe we are talking about different things.

Not different things, but perspectives.

I'd agree that your simple primary/secondary name server is easy to
setup and it'll probably work just fine. However it supports a very
limited number of use cases and traffic levels.

My experience with DNS and administrating it in various incarnations
since the bad old days of Bind 4 informs me that it can be incredibly
fragile. It is only the implementation of the current DNS system that is
reasonably resilient or at least able to localize failure. Certainly
some designs and technology are better than others, but implementation
always matters.

The same goes for just about system/protocol you'd care to name.

>> Your ppg tooling does look interesting, but there is a large trade
>> off in functionality
>
> What is the loss of functionality you see? Anything that you use in practice?
>
> (Reading here https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/7ZpAMrMb2NQ
> I can't spot anything major, but I may be missing something...)
>
> cheers,
> m

Masterless Puppet with git as a distribution method does have some
things going for it as a design. You are giving up things like collected
resources and standard reporting which may or may not matter you.
Additionally you're building a distribution system of some sort even if
it's just git and ssh where you'd need to decide how to deal with the
failure of the parts.

In any case I'd like to see more discussion on highly available Puppet
regardless of way it's implemented.

Ramin

[Felipe Salum]

I'm trying to make a manifest to auto setup Puppet High Availability, but it is the chicken-egg issue here. As for your secondary/tertiary/etc puppetmasters, you need to copy the private key and certificate used by your puppet1 server in order for it to accept the requests coming from puppet.yourcompanydomain.com or whatever you choose there.

Also you need to maintain the /var/lib/puppet/ssl/ca in sync between all puppetmasters so you can easily failover if puppet1 is down to any other.

Since Puppet is about automation, when I mentioned something easier to setup I was expecting a puppet module to implement it, not manual changes. As the implementation is not very straightforward it is not easy to automate it IMHO.

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/Ze5QFJ95y3E/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to puppet-users+unsubscribe@googlegroups.com.

[Martin Langhoff]

On Fri, May 10, 2013 at 1:52 PM, Ramin K <ramin...@badapple.net> wrote:
> reasonably resilient or at least able to localize failure. Certainly some
> designs and technology are better than others, but implementation always
> matters.

Of course. I think we're saying the same thing, at the end of the day.

> Masterless Puppet with git as a distribution method does have some
> things going for it as a design. You are giving up things like collected
> resources and standard reporting which may or may not matter you.

Thanks for these notes.

I am definitely supporting reporting -- finishing the implementation
today/Monday.

Reading the thread I mentioned before, ISTM that resources can be
trivially added with the same machinery.

> Additionally you're building a distribution system of some sort even if it's
> just git and ssh where you'd need to decide how to deal with the failure of
> the parts.

Absolutely, and that's at the center of my design

- It is a pull design (clients pull from a "gold" server, or through
"local proxy" servers)

- If the pull fails, it's a soft error; we soldier on. config changes
that have been fetched earlier still apply. importantly, config
changes that were fetched and have scheduling data have their
scheduled rollout still apply.

For example, I make a commit with a change at midday today, and
schedule it for midnight. I publish it on the gold server. Clients
fetch it, but don't apply it yet. Gold server goes down at 10pm --
this is a soft problem: clients will apply the changes
published&distributed earlier when their time comes.

- Communication back to the gold server is store-and-forward, so that
midnight puppet run collects reports (and perhaps resources in the
future), and they are spooled locally until the gold server is ready
to receive them.

> In any case I'd like to see more discussion on highly available
> Puppet regardless of way it's implemented.

+100

[Erik Dalén]

On 10 May 2013 19:52, Ramin K <ramin...@badapple.net> wrote:

        In any case I'd like to see more discussion on highly available Puppet regardless of way it's implemented.

We are using SRV records for running multiple puppetmasters and selecting a site local but allowing fallback to others in case it is down.

We have 6 puppetmasters for the production environment running in this way currently. Each normally handling 500-1000 nodes. The git repository is push replicated to each one of them.

But only one is CA, it is backed up. If it would crash we are fine with having a outage on installing new nodes until we have restored that part to another node. But we have looked into some solutions for maybe making it more resilient though.

For PuppetDB we have two service nodes and a master and hot standby for the postgres database.

--
Erik Dalén

[Mason Turner]

We have a similar setup, minus the SRV records (although that looks quire interesting, gotta get off of 2.7). And we push SVN checkouts instead of git, but that's not a big difference.

I have been thinking about the CA, and how to make it more available. My first thought is, do we have to save the generated client certs at all? I brought this up a few weeks ago and the general answer was "there is no technical reason to keep the certs", so I am considering deleting them immediately. Now I don't have to worry about backing up the puppetca!

Next, and this is where my SSL weakness  will shine, could you have all of your HA-puppetmasters run as CAs, too, and then have multiple CA certs on trusted list on the puppet masters? Something like this:

1. foo-east01 comes up, and gets an auto-signed vert from pm-east01.

2. pm-east01 hit by asteroid, so foo-east01 automatically fails over to foo-west01

3. pm-west01 knows to trust the pm-east-01 signed cert.

4. We stand up a pm-east0.new1, generate a new vert for it and append said cert to the trusted list for all clients/PMs.

5. foo-east01 starts using pm-east01.new again

6. foo-east02 comes up, gets a cert from pm-east01.new

(This is starting to feel like a certificate rotation strategy in some weird way).

One thing I wonder is if I'll actually be a little more secure. Instead of having to have a single CA with a huge FW configuration (we have a lot of independent networks across the 'net), each PM/CA has only a very specific FW ruleset.

 

[Martin Langhoff]

On Tue, May 14, 2013 at 7:35 AM, Erik Dalén <erik.gus...@gmail.com> wrote:
> We are using SRV records for running multiple puppetmasters and selecting a
> site local but allowing fallback to others in case it is down.
> We have 6 puppetmasters for the production environment running in this way
> currently. Each normally handling 500-1000 nodes. The git repository is push
> replicated to each one of them.

Interesting - do you use dashboard or anything similar to track the
state of the nodes?

[Felipe Salum]

If you don't need to backup your puppetca, how do you carry over to a standby puppetca server your client signed certificates and revocation list in case of failure in the production puppetca ?

--
You received this message because you are subscribed to a topic in the Google Groups "Puppet Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/puppet-users/Ze5QFJ95y3E/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to puppet-users...@googlegroups.com.

[shyam sundar Keshari]

Hi ,

Thanks all for your knowledge sharing on mine Query .So after all I planned to make Individual Puppet master for each locations .

Thanks Team