Jira (PDB-4487) PuppetDB Terminus with Masterless Fails on SSL Errors

[Craig Watson (JIRA)]

Craig Watson created an issue

PuppetDB / PDB-4487 PuppetDB Terminus with Masterless Fails on SSL Errors Issue Type: Task Affects Versions: PDB 6.5.0 Assignee: Unassigned Components: PuppetDB Created: 2019/08/22 9:08 AM Priority: Blocker Reporter: Craig Watson We are running Puppet masterless, connecting to a standalone PuppetDB cluster (6.5.0) with a shared PostgreSQL backend. SSL termination is handed by the load balancer, which is using a managed SSL certificate (from Google, signed by LetsEncrypt). routes.yaml:   --- apply: catalog: terminus: compiler cache: puppetdb resource: terminus: ral cache: puppetdb facts: terminus: facter cache: puppetdb_apply puppetdb.conf:     [main] server_urls = https://puppetdb.[redacted]:443 soft_write_failure = true puppet.conf:   [main] storeconfigs = false report = true reports = puppetdb     We have upgraded a test node to Puppet 6.8.0 and also upgraded the PuppetDB Termini to 6.5.0:   $ yum list puppet-agent puppetdb-termini Installed Packages puppet-agent.x86_64 6.8.0-1.el7 @puppet6 puppetdb-termini.noarch 6.5.0-1.el7 @puppet We now have issues where the PuppetDB terminus is complaining about a lack of TLS-related files (CRL, CA, certificate, key) which would normally exist on a mastered Puppet installation, but don't on ours as they're not used. So far, I have added: localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false To try and work around and stub out the SSL, but I am now left with: Error: Failed to initialize SSL: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem' Error: Run `puppet agent -t` Error: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem' I have tried rolling a temporary/throwaway local CA with no success.   Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Craig Watson (JIRA)]

Craig Watson updated an issue

PuppetDB / PDB-4487 PuppetDB Terminus with Masterless Fails on SSL Errors

Change By: Craig Watson

We are running Puppet masterless, connecting to a standalone PuppetDB cluster (6.5.0) with a shared PostgreSQL backend. SSL termination is handed by the load balancer, which is using a managed SSL certificate (from Google, signed by LetsEncrypt). routes.yaml:

{code:java}

--- apply: catalog: terminus: compiler cache: puppetdb resource: terminus: ral cache: puppetdb facts: terminus: facter

cache: puppetdb_apply{code} puppetdb.conf:     {code:java}

[main] server_urls = https://puppetdb.[redacted]:443

soft_write_failure = true{code} puppet.conf:   {code:java}

[main] storeconfigs = false report = true

reports = puppetdb{code}

We have upgraded a test node to Puppet 6.8.0 and also upgraded the PuppetDB Termini to 6.5.0:

{code:java}

$ yum list puppet-agent puppetdb-termini Installed Packages puppet-agent.x86_64 6.8.0-1.el7 @puppet6

puppetdb-termini.noarch 6.5.0-1.el7 @puppet{code}

We now have issues where the PuppetDB terminus is complaining about a lack of TLS-related files (CRL, CA, certificate, key) which would normally exist on a mastered Puppet installation, but don't on ours as they're not used.

{{ So far, I have added: }}

{code :java } localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false{code}

To try and work around and stub out the SSL, but I am now left with:

{code:java}

Error: Failed to initialize SSL: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem' Error: Run `puppet agent -t`

Error: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem'{code}

I have tried rolling a temporary/throwaway local CA with no success.

Add Comment

[Craig Watson (JIRA)]

}$ yum list puppet-agent puppetdb-termini

{code}localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false{code} To try and work around and stub out the SSL, but I am now left with the following for each interaction with PuppetDB (facts, catalog, report) : {code :java }Error: Failed to initialize SSL: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem'

Error: Run `puppet agent -t` Error: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem'{code} I have tried rolling a temporary/throwaway local CA with no success.

The Puppet run itself succeeds, but nothing is sent at all to PuppetDB

Add Comment

[Craig Watson (JIRA)]

Craig Watson updated an issue

PuppetDB / PDB-4487 PuppetDB Terminus with Masterless Fails on SSL Errors Change By: Craig Watson

We are running Puppet masterless, connecting to a standalone PuppetDB cluster (6.5.0) with a shared PostgreSQL backend. SSL termination is handed by the load balancer, which is using a managed SSL certificate (from Google, signed by LetsEncrypt). routes.yaml:

{code}---

apply: catalog: terminus: compiler cache: puppetdb resource: terminus: ral cache: puppetdb facts: terminus: facter cache: puppetdb_apply{code} puppetdb.conf:

{code}[main]

server_urls = https://puppetdb.[redacted]:443 soft_write_failure = true{code} puppet.conf:

{code}[main]

storeconfigs = false report = true reports = puppetdb{code} We have upgraded a test node to Puppet 6.8.0 and also upgraded the PuppetDB Termini to 6.5.0:

{code}$ yum list puppet-agent puppetdb-termini

Installed Packages puppet-agent.x86_64 6.8.0-1.el7 @puppet6 puppetdb-termini.noarch 6.5.0-1.el7 @puppet{code} We now have issues where the PuppetDB terminus is complaining about a lack of TLS-related files (CRL, CA, certificate, key) which would normally exist on a mastered Puppet installation, but don't on ours as they're not used. So far, I have added: {code}localcacert = /etc/pki/tls/certs/ca-bundle.crt certificate_revocation = false{code} To try and work around and stub out the SSL, but I am now left with the following for each interaction with PuppetDB (facts, catalog, report):

{code}Error: Failed to initialize SSL: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem'

Error: Run `puppet agent -t` Error: The private key is missing from '/etc/puppetlabs/puppet/ssl/private_keys/[fqdn].pem'{code} I have tried rolling a temporary/throwaway local CA with no success. The Puppet run itself succeeds, but nothing is sent at all to PuppetDB

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors Craig Watson was this working when using puppet 6.5.0, and is now broken after upgrading the test node to 6.8.0?

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors

Assuming this isn't a configuration error, and it only affects the 6.5.0 terminus, I have a feeling this was caused by switching to the non-deprecated HTTP Pool method connection. We now explicitly look up an ssl_context from Puppet. This likely needs to be guarded with a config option, either existing or new. https://github.com/puppetlabs/puppetdb/commit/8714ba9288da588e9fd197902332dd7c04635f10#diff-768747907b90c39ab6f16fcb3320897a

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4487

PuppetDB Terminus with Masterless Fails on SSL Errors

Change By: Austin Blatt Issue Type: Task Bug

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors From Craig Watson in Slack, "last confirmed working versions were 6.5.0 for the agent and (IIRC) 6.4.0 for the terminus"

Add Comment

[Craig Watson (JIRA)]

Craig Watson commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors

Thanks Austin Blatt and Josh Cooper - I’ve had a dive upstream into the SSL code, and it does seem like the change to `ssl_context` is the cause, pulling this data in invokes `Puppet::SSL::Verifier` which attempts a full validation of the local certificate and CA. The wider implication here is that its directly incompatible with masterless deployments that lack these certificates. As it’s not technically caused by PuppetDB, I’m happy for this ticket to be closed/migrated around as necessary. If there’s anything I can do to help with diagnosis, feel free to reach out

Add Comment

[Craig Watson (JIRA)]

Craig Watson commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors

So, I have worked/hacked around the problem in this PR: https://github.com/puppetlabs/puppetdb/pull/3053 Essentially, I have added the option to build our own SSLContext instead of looking up Puppet's own - this still performs validation of the certificate provided by PuppetDB against a CA, so should still satisfy security requirements. This has been tested and verified to work on my 6.8.0 node. My knowledge of testing/documentation rules for PuppetDB are non-existent, so I'm happy to either take guidance on this and evolve my PR, or I'm equally as happy for my fork's commit to be cherry-picked out somehow and used by someone with enough skill to write adequate docs/tests

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt assigned an issue to Austin Blatt

PuppetDB / PDB-4487

PuppetDB Terminus with Masterless Fails on SSL Errors

Change By: Austin Blatt Assignee: Austin Blatt

Add Comment

[Austin Blatt (JIRA)]

Austin Blatt updated an issue

PuppetDB / PDB-4487 PuppetDB Terminus with Masterless Fails on SSL Errors

Change By: Austin Blatt Fix Version/s: PDB 6.6.0

Add Comment

[Louis Coilliot (JIRA)]

Louis Coilliot commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors Hello, I've tested with puppetdb-termini-6.6.0-1.el7.noarch released yesterday and it's not better. With puppetdb-termini-6.6.0 : ``` /opt/puppetlabs/puppet/bin/puppet apply --modulepath=/deploy/environments/dev/modules -e 'include base' Error: Failed to initialize SSL: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem'

Error: Run `puppet agent -t`

Error: Cached facts for centos7-sandbox-acl.c.int-lz1-dtp.internal failed: Failed to find facts from PuppetDB at puppet:8140: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem' Error: Failed to initialize SSL: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem'

Error: Run `puppet agent -t`

Error: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem' Error: Could not run: Could not retrieve facts for centos7-sandbox-acl.c.int-lz1-dtp.internal: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem' Wrapped exception: The CA certificates are missing from '/etc/puppetlabs/puppet/ssl/certs/ca.pem ``` With puppetdb-termini-6.4.0 (after 2 `yum downgrade puppetdb-termini`) : ``` /opt/puppetlabs/puppet/bin/puppet apply --modulepath=/deploy/environments/dev/modules -e 'include base' Warning: Puppet::SSL::Host is deprecated and will be removed in a future release of Puppet. (location: /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/ssl/host.rb:235:in `initialize') Error: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known) Error: Cached facts for centos7-sandbox-acl.c.int-lz1-dtp.internal failed: Failed to find facts from PuppetDB at puppet:8140: Failed to open TCP connection to puppet:8140 (getaddrinfo: Name or service not known) Notice: Compiled catalog for centos7-sandbox-acl.c.int-lz1-dtp.internal in environment dev in 0.30 seconds Notice: Applied catalog in 1.60 seconds ``` The apply is OK although I still have a non blocking error related (I guess) to another issue (https://tickets.puppetlabs.com/browse/PUP-9314 : puppetdb-termini for puppet standalone try to connect to master)

Add Comment

[Craig Watson (JIRA)]

Craig Watson commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors

Louis Coilliot Can you add this to `puppetb.conf` and retry? ``` verify_client_certificate = false ```

Add Comment

[Louis Coilliot (JIRA)]

Louis Coilliot commented on PDB-4487

Re: PuppetDB Terminus with Masterless Fails on SSL Errors

I confirm this solution is correct. All warnings and errors have disappeared and I can now set the `cache: puppetdb_apply` for facts in routes.yaml. The puppetdb SSL proxy certificate must be signed by a CA trusted in /etc/pki/tls/certs/ca-bundle.crt, so for a custom private SSL certificate and CA you need to update the trusted CA list (see : update-ca-trust on CentOS, update-ca-certificates on Debian). Or set an other target for the localcacert parameter. I agree that it should be mentioned in the doc here : https://puppet.com/docs/puppetdb/6.6/connect_puppet_apply.html#option-a-set-up-an-ssl-proxy-for-puppetdb Thanks a lot for your understanding and reactivity.

Add Comment