Best practice for Puppet CA servers in multiple Data Centres - upgrading to v6

[chris]

Hi Guys,

so we've got a few data centres spread across the world and are looking to upgrade from Puppet v4 to Puppet v6.

At the moment we just have the one CA in the original DC (fast growing company).

I like the idea of having a separate CA in each DC and having the "local" machine use that - simples .. ;)

However, I'd like to know if there are any sane alternatives as I'll need to persuade the rest of the team/mgrs.

Is it  possible/sane to just build a CA in each DC but have it not active and then rsync the certs across every hour/day  from the active CA & bring it up if (ie when)  the main CA/DC goes away.

Are there any other sensible ideas out there?

Ideally, what is the recommended best practice by Puppet (we are on the  FOSS version, so I can't ask them).

FWIW, we use Foreman to keep an eye on stuff & I believe(?) it could be tricky to have multiple CAs talking to it ??

(I know nothing about how the foreman - puppet cxn works).

Cheers

Chris

[Luke Bigum]

It depends on how often you build "new" machines, or if you think you'd need to bootstrap new Puppet Agents if your DCs were cut off from one another.  I get away with 1 CA for your entire estate and with multiple redundant compile masters at each DC.  That way you don't need to sync certificates around, you'll only need to contact the CA the first time an Agent checks in.  This is simplicity but with a point of failure.  You're probably going to have one PuppetDB anyway (or postgres cluster in one location)?

To do it properly though, I think you would need each Puppet Server to have it's own intermediate CA, all signed from a common root CA of yours:

https://puppet.com/docs/puppetserver/5.2/intermediate_ca_configuration.html

[chris]

Hi Luke,

That's very interesting; thanks.

We do have 2 non-CA puppetmasters in each DC, so you are saying that client servers will continue to be able to call in, but we won't be able to setup any new ones?

We do only have one puppetdb & foreman in  the main DC.

Intermediate Certs looks a bit fiddly but might be an option.

Just to clarify, using these would mean we could also standup new client-servers in the other DCs if the main DC goes down?

Cheers

Chris

[Luke Bigum]

On Wednesday, 18 September 2019 05:12:49 UTC+1, chris wrote:

Hi Luke,

That's very interesting; thanks.

We do have 2 non-CA puppetmasters in each DC, so you are saying that client servers will continue to be able to call in, but we won't be able to setup any new ones?

Yes, and to make doubly sure I just shut down my own CA / Signing Master, and an Agent in a satellite DC was able to check in with the local Compiling Master fine (because the Agent already has a Puppet cert).  I find DNS SRV records useful for managing this:

https://puppet.com/docs/puppetserver/5.1/scaling_puppet_server.html#using-dns-srv-records

Obviously this approach won't work if you're spinning up many short lived VMs or disposable infrastructure.

We do only have one puppetdb & foreman in  the main DC.

PuppetDB is a different matter...  In theory an Agent should be able to run without it, except for if the Compiling Master needs to go to PuppetDB to realise any exported resources.  From memory the Agents will complain about pushing their Facts into PuppetDB, but this itself does not stop the run - I have seen catalog compilations work with PuppetDB offline, but it wasn't perfect.  Last time I tried PuppetDB maintenance in hours the after-affects annoyed all of my team, so I didn't have the luxury of finding out exactly what was reliant on PuppetDB, nor what config options I could use to lessen the impact.  Since we use exported resources a lot and these are stored in PuppetDB, it makes sense that any catalog reliant on realising exported resources would fail.

Intermediate Certs looks a bit fiddly but might be an option.

Just to clarify, using these would mean we could also standup new client-servers in the other DCs if the main DC goes down?

No, if you've got one CA / Signing Master, any new agent (fresh install) would send it's CA signing requests to your Signing Master, also sometimes called a Master of Masters.  If you had a critical need you could turn one of your existing masters in a DC into a CA, and then fix up the certs later - basically destroy and re-add all the Agents once the main DC was back online.

[chris]

Hi Luke,

thanks a lot for this information, it will be very useful.

Sorry I didn't reply earlier...

I was particularly interested in this bit as it seems it would back up my preference to tell the boss we need to make each DC independent with it's own CA master :)

Much simpler in my opinion.

Cheers

Chris