Send master B's reports to PuppetDB on a master A?

[Ryan Anderson]

I have a need to send reports from a puppet master B in datacenter B to puppetdb on master A in datacenter A. Both are using puppet open source 3.7.1 and puppetdb 2.2 (master A) or puppetdb-terminus (master B).

I have done all steps here: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_master.html. However, this page says nothing about using SSL certs so that puppetdb-terminus on master B can connect to https port 8081 on master A. I get errors like this:

Warning: Error 400 on SERVER: Could not retrieve facts for masterB.example.com: Failed to find facts from PuppetDB at masterA.example.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [unable to get local issuer certificate for /CN=masterA.example.com]

The separate page on setting up master-less puppet agents to send puppetdb reports touches on this: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html

The most promising solution here looks like setting up an apache SSL proxy that redirects https 8081 to localhost:8080 mentioned here: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html#option-a-set-up-an-ssl-proxy-for-puppetdb. However, I know little about configuring apache this way, and an example config isn't provided. It even says
More detailed instructions for setting up this proxy will be added to this guide at a later date". The 2.3 instruction lacks this also. Any ideas?

[Martin Alfke]

Hi Ryan,

On 28 Apr 2015, at 01:34, Ryan Anderson <ryan.c....@gmail.com> wrote:

> I have a need to send reports from a puppet master B in datacenter B to puppetdb on master A in datacenter A. Both are using puppet open source 3.7.1 and puppetdb 2.2 (master A) or puppetdb-terminus (master B).

This is easily possible when using a common CA between both masters.
1. Master of Masters (CA, Modules for PuppetDB and Puppet Masters)
2. Puppet DB
3. Puppet Master A (catalog compile and file serving)
4. Puppet Master B (catalog compile ans file serving)

>
> I have done all steps here: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_master.html. However, this page says nothing about using SSL certs so that puppetdb-terminus on master B can connect to https port 8081 on master A. I get errors like this:
> Warning: Error 400 on SERVER: Could not retrieve facts for masterB.example.com: Failed to find facts from PuppetDB at masterA.example.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed: [unable to get local issuer certificate for /CN=masterA.example.com]
>
> The separate page on setting up master-less puppet agents to send puppetdb reports touches on this: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html
>
> The most promising solution here looks like setting up an apache SSL proxy that redirects https 8081 to localhost:8080 mentioned here: https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html#option-a-set-up-an-ssl-proxy-for-puppetdb. However, I know little about configuring apache this way, and an example config isn't provided. It even says
> More detailed instructions for setting up this proxy will be added to this guide at a later date". The 2.3 instruction lacks this also. Any ideas?
>

The apache documentation for mod_proxy has some examples on how to set up an https -> http proxy.

hth,

Martin

[Ken Barber]

> I have a need to send reports from a puppet master B in datacenter B to
> puppetdb on master A in datacenter A. Both are using puppet open source
> 3.7.1 and puppetdb 2.2 (master A) or puppetdb-terminus (master B).
>
> I have done all steps here:
> https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_master.html.
> However, this page says nothing about using SSL certs so that
> puppetdb-terminus on master B can connect to https port 8081 on master A. I
> get errors like this:
> Warning: Error 400 on SERVER: Could not retrieve facts for
> masterB.example.com: Failed to find facts from PuppetDB at
> masterA.example.com:8081: SSL_connect returned=1 errno=0 state=SSLv3 read
> server certificate B: certificate verify failed: [unable to get local issuer
> certificate for /CN=masterA.example.com]

This means that the sender's configured CA is different to the CA that
issued the destination masterA.example.com certificate on your
PuppetDB node. On your master, you will have a particular CA
configured to sign certificates, however PuppetDB when installed tries
to use the local puppet agent's CA on that node you installed it on
(by running puppetdb-ssl-setup, which just moves certificates into a
place PuppetDB can get to them).

What is your CA topology between the two datacentres? Are they meant
to be different?

> The separate page on setting up master-less puppet agents to send puppetdb
> reports touches on this:
> https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html
>
> The most promising solution here looks like setting up an apache SSL proxy
> that redirects https 8081 to localhost:8080 mentioned here:
> https://docs.puppetlabs.com/puppetdb/2.2/connect_puppet_apply.html#option-a-set-up-an-ssl-proxy-for-puppetdb.
> However, I know little about configuring apache this way, and an example
> config isn't provided. It even says
> More detailed instructions for setting up this proxy will be added to this
> guide at a later date". The 2.3 instruction lacks this also. Any ideas?

I'm not sure you need a proxy per se, it depends on your exact needs.
Either way, you still end up having to deal with certificates.

ken.

[Ryan Anderson]

My two masters are each also their own CA to minimize traffic and firewall rules between them. Based on your responses, this difference in CA's appears be the crux of the issue....which seems obvious now.

Given that masters A & B are their own CA, how can I send puppetdb reports over port 8081 https from B to A?

[jcbollinger]

On Tuesday, April 28, 2015 at 7:49:00 AM UTC-5, Ryan Anderson wrote:

My two masters are each also their own CA to minimize traffic and firewall rules between them. Based on your responses, this difference in CA's appears be the crux of the issue....which seems obvious now.

Given that masters A & B are their own CA, how can I send puppetdb reports over port 8081 https from B to A?

Dunno whether you can, but you shouldn't.  For all intents and purposes, the CA defines the logical boundaries of the site.  With two distinct CAs, you have two logically distinct sites.  You can aggregate reports and other information about those sites at a higher level, but you are asking for trouble if you somehow put data pertaining to one site into the other's database.


John

[Ken Barber]

> My two masters are each also their own CA to minimize traffic and firewall
> rules between them. Based on your responses, this difference in CA's appears
> be the crux of the issue....which seems obvious now.
>
> Given that masters A & B are their own CA, how can I send puppetdb reports
> over port 8081 https from B to A?

It certainly is tricky to do this right with two root CA certificates,
one answer is to use intermediate CA's (this way the trust is global,
defined by a root CA), but this means you are left on your own to do
PKI: https://docs.puppetlabs.com/puppet/latest/reference/config_ssl_external_ca.html.
In a perfect world the PKI facility we should could still sign
certificates with an intermediate, but alas this doesn't seem to be
the case. Even if you have your own PKI, I think we still come back to
similar problems as I'm about to describe below.

In the past others (including myself) have done tricks to support a
single root CA between two services, but this is hackish and you end
up with a split CRL list and serial number collision issues (since
afaik, we assign them sequentially, but its been awhile since I've
looked at this). I've even gone so far as to switch the serial number
assignment to be random instead of sequential, but this required
custom patches. Others have just hacked around different starting
numbers for their serials in each service, but either way you're still
left with split CRL data, which might be a problem if you really on
using revocation regularly - perhaps in a split DC solution where you
don't care about master failover between sites this doesn't matter.
Anyway I'm conjecturing, in reality these kinds of ideas are hackish
and not for the feint hearted.

As far as what people do today and the best solution to avoid a single
CA service, I'm not across it. I'd probably see if others have any
suggestions on the matter. Today, I think the official/maintained
solution is a global CA service.

Of course you can go back to your proxy idea, but this won't scale
very well ultimately, as you'll need to add more of these proxies for
each DC you add just to establish trust. But it would work, I guess
:-).

ken.