Jira (PUP-7814) HTTPS file sources with non-puppet-trusted certs can't be used

[Eric Sorenson (JIRA)]

Eric Sorenson created an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used Issue Type: Bug Assignee: Unassigned Created: 2017/08/02 2:51 PM Priority: Normal Reporter: Eric Sorenson In PUP-1072 we added support for http and https URL schemes for file resources' source attributes: file { '/tmp/myfile': source => 'https://myserver/myfile' } Somehow we failed to realize that the implementation of this feature requires that the remote webserver present a certificate that is trusted by the puppet agent's SSL configuration. In practice this is extremely unlikely and difficult to configure, and therefore the utility of this feature is pretty limited. In the code, this is because the request ultimately uses the same routines (in Puppet::Util::HttpProxy.get_http_object) that the agent itself uses, where the SSL verification mode is hardcoded but the certificate store is not. There should be some way to permit retrieval of these files. A couple of options that occured to me as I was tracing this down: 1. make the ssl verification mode configurable or disabled by default for file-source retrieval (equivalent to curl -k) 2. use the system-wide trust store instead of puppet's store 3. expose the certificate trust and verification mode as parameters on the file type (this is truly awful) ... surely someone else has better ideas ... Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Team: Platform Core

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used Our HTTP connection factory doesn't expose the ability for a client to request which trust anchors to use. The idea Jeremy Barlow and I had discussed previously was to allow the client to request a connection that 1) only trusts the puppet CA or 2) in addition trusts the certs in the puppet-agent cacerts bundle. If it's the latter, then the connection factory can call X509Store.set_default_paths to initialize the cert store. Marking this ticket as blocked on PUP-5069. Also, there's a similar issue when the puppetmaster/server's report processor tries to make an HTTPS connection (see PUP-7737).

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper updated an issue

Puppet / PUP-7814

HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Sub-team: Coremunity

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Charlie Sharpsteen CS Priority: Needs Priority

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Adam Bottchen CS Impact: PS is seeing this in the field and support has had a few calls associated with it.    The solution in the past has been to update the keystore to include the additional cert.  This can be automated with existing Puppet modules to help it survive upgrades.  That said, it would be better to not force customers to take this extra step. CS Severity: 3 - Serious CS Business Value: 3 - $$$$ CS Frequency: 2 - 5-25% of Customers

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Adam Bottchen CS Priority: Needs Priority Reviewed

CS Impact: PS is seeing this in the field and support has had a few calls associated with it.    The solution in the past has been to update the keystore to include the additional cert.  This can be automated with existing Puppet modules to help it survive upgrades.  That said, it would be better to not force customers to take this extra step. CS Severity: 3 - Serious CS Business Value: 3 - $$$$ CS Frequency: 2 - 5-25% of Customers

Add Comment

[Thomas Kishel (JIRA)]

Thomas Kishel commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used Large customer affected by this issue

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

Here's another (unsupported) workaround: remote_file { '/tmp/myfile': source => 'https://myserver/myfile', verify_peer => false, } See lwf/remote_file. Also supports proxies.

Add Comment

[Louis Mayorga (JIRA)]

Louis Mayorga commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

This is meant to be supported from the beginning on adding the URL feature for the file type. If you have to go to exec to solve these kind of issues then puppet is not the right one.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

It's not quite clear from the description, but this also affects HTTPS urls that are from servers that use certificates issued by the Puppet CA. The only CAs that are trusted are those in the CA bundle in the puppet-agent package: /opt/puppetlabs/puppet/ssl/cert.pem A simple re-production is to use a HTTPS source to download a copy of the Puppet CA cert from the Puppet Server: cat <<'EOF' > /etc/puppetlabs/code/environments/production/manifests/site.pp node default { file { '/tmp/ca.pem': ensure => file, source => "https://${server_facts['servername']}:8140//puppet-ca/v1/certificate/ca?environment=production", } } EOF This fails on Puppet 6 with: # puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Caching catalog for jvsf81owg61pufd.delivery.puppetlabs.net Info: Applying configuration version '1561753379' Error: /Stage[main]/Main/Node[default]/File[/tmp/ca.pem]: Could not evaluate: Could not retrieve file metadata for https://jvsf81owg61pufd.delivery.puppetlabs.net:8140/puppet-ca/v1/certificate/ca?environment=production: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) Notice: Applied catalog in 0.07 seconds And on Puppet 5 with: # puppet agent -t Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Info: Retrieving locales Info: Caching catalog for vwirqi7pylvyl6b.delivery.puppetlabs.net Info: Applying configuration version '1561753402' Error: /Stage[main]/Main/Node[default]/File[/tmp/ca.pem]: Could not evaluate: Could not retrieve file metadata for https://vwirqi7pylvyl6b.delivery.puppetlabs.net:8140/puppet-ca/v1/certificate/ca?environment=production: SSL_connect returned=1 errno=0 state=error: certificate verify failed Notice: Applied catalog in 0.05 seconds

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen updated an issue

Puppet / PUP-7814

HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Charlie Sharpsteen Affects Version/s: PUP 6.5.0 Affects Version/s: PUP 5.5.14

Add Comment

[John (JIRA)]

John commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used I had a similar issue, however the ticket PUP-8889 Puppet Agent : cannot add certificates for HTTPS Franck Jouvanceau added a comment - 2018/06/06 5:04 AM helped a great deal, and now I confirm the solution described on the ticket works as a charm.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

I tested this and confirmed the following when using an https file source: Connections to non-puppet SSL servers, such as source => 'https://github.com/path/to', succeed if the root CA cert is in CA bundle in the puppet-agent package. It resides in /opt/puppetlabs/puppet/ssl/{cert.pem,certs}, and comes from https://github.com/puppetlabs/puppet-ca-bundle. Connections to non-puppet SSL servers fail if the CA cert is only in the system CA bundle, such as /etc/pki/ca-trust/source/anchors. This is common for enterprise customers that have a root CA not issued from a trusted third party like VeriSign, etc. Connections to puppet SSL servers fail since puppet's CA cert is not in /opt/puppetlabs/puppet/ssl/cert.pem. This is the issue Charlie Sharpsteen mentioned above. The current workaround is to add the missing root CA to /opt/puppetlabs/puppet/ssl/cert.pem or the /opt/puppetlabs/puppet/ssl/certs directory. In the latter case, the /opt/puppetlabs/puppet/bin/c_rehash command must be executed so openssl can load the cert. As mentioned elsewhere, this needs to be done after each puppet-agent upgrade. I propose we do the following: Add a puppet setting for the system cert store directory. By default, the setting would be nil. If set to a directory, then call Net::HTTP#ca_path= with the directory containing certificates in PEM format, such as /etc/ssl/certs. The cert rehashing process should occur outside of puppet, such as when the ca-certificates package is updated. This would fix case 2 above. When using https file sources, always trust the puppet CA and the puppet-agent CA bundle. This would fix case 3 above.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

In my opinion, adding stuff to /opt/puppetlabs/puppet/ssl is currently an unsupported workaround. I think it would make a great official solution to this problem, but it needs: Documentation. Persistence through upgrades. Test coverage.

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

Oh sorry, I'm not suggesting we continue with the workaround. The two proposed actions above would eliminate the need to do that, as puppet would be able to load CA certs from additional locations instead of trying to "push" CA certs into locations puppet already knows about.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

Aha, that makes a lot of sense. to that.

Add Comment

[Reid Vandewiele (JIRA)]

Reid Vandewiele commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

This would be specific to the File resource provider's use of Net::HTTP, and would not extend to any of the Puppet agent's core PKI trust configuration, yes? That is, we wouldn't end up being able to trust and download a catalog from a "master" with a cert signed by Let's Encrypt or anything like that....

Add Comment

[Josh Cooper (JIRA)]

Josh Cooper commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

Correct. Connections to puppet infrastructure (compiler, CA, report servers) would only trust the CA bundle that the agent downloads from the CA. Only file resources with "https" sources would trust the CA bundle and the system cacert store.

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-7814

HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Austin Boyd Zendesk Ticket IDs: 32650 Zendesk Ticket Count: 1

Add Comment

[Austin Boyd (JIRA)]

Austin Boyd updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Austin Boyd Zendesk Ticket IDs: 32650 ,35349 Zendesk Ticket Count: 1 2

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper

In [PUP-1072] we added support for http and https URL schemes for {{file}} resources' {{source}} attributes:

{code}

file { '/tmp/myfile':   source => 'https://myserver/myfile' }

{code} ** UPDATED ** 1. The above works provided the CA certificate that issued the server's certificate is in the [CA bundle distributed|https://github.com/puppetlabs/puppet-ca-bundle] in puppet-agent packages. There are workarounds to add certificates to puppet's cert bundle, but those workarounds are lost when the puppet-agent package is updated. 2. Puppet now trusts the puppet CA[~chuck] So this ticket is about providing a configuration option to spect

Somehow we failed to realize that the implementation of this feature requires that the remote webserver present a certificate that is trusted by the puppet agent's SSL configuration. In practice this is extremely unlikely and difficult to configure, and therefore the utility of this feature is pretty limited.

In the code, this is because the request ultimately uses the same routines (in {{Puppet::Util::HttpProxy.get_http_object}}) that the agent itself uses, where the SSL verification mode is hardcoded but the certificate store is not.

There should be some way to permit retrieval of these files. A couple of options that occured to me as I was tracing this down: 1.  make the ssl verification mode configurable or disabled by default for file-source retrieval (equivalent to {{curl -k}}) 2. use the system-wide trust store instead of puppet's store 3. expose the certificate trust and verification mode as parameters on the file type (this is truly awful) ... surely someone else has better ideas ...

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used Change By: Josh Cooper

In [PUP-1072] we added support for http and https URL schemes for {{file}} resources' {{source}} attributes: {code} file { '/tmp/myfile':   source => 'https://myserver/myfile' } {code} * * UPDATED * *

1. The above works provided the CA certificate that issued the server's certificate is in the [CA bundle distributed|https://github.com/puppetlabs/puppet-ca-bundle] in puppet-agent packages. There are workarounds to add certificates to puppet's cert bundle, but those workarounds are lost when the puppet-agent package is updated.

2. As of Puppet now trusts 6.15.0 puppet will trust the puppet CA in addition to puppet's CA bundle when retrieving file metadata and content via https file sources. So the issue [~chuck] described with {{source => "https://${server_facts['servername']}:8140//puppet-ca/v1/certificate/ca?environment=production"}} will be resolved. So this 3. This ticket is about providing a configuration option ) creating a puppet setting referring to spect a directory or file containing system CA certs, and b) modifying the {{Puppet::SSL::SSLProvider#create_system_context}} method so that the system CA certs are added to the X509 store via {{OpenSSL::X509::Store#add_path}} (for a directory) or {{OpenSSL::X509::Store#add_file}} (for a file). *Original*

Somehow we failed to realize that the implementation of this feature requires that the remote webserver present a certificate that is trusted by the puppet agent's SSL configuration. In practice this is extremely unlikely and difficult to configure, and therefore the utility of this feature is pretty limited. In the code, this is because the request ultimately uses the same routines (in {{Puppet::Util::HttpProxy.get_http_object}}) that the agent itself uses, where the SSL verification mode is hardcoded but the certificate store is not. There should be some way to permit retrieval of these files. A couple of options that occured to me as I was tracing this down: 1.  make the ssl verification mode configurable or disabled by default for file-source retrieval (equivalent to {{curl -k}}) 2. use the system-wide trust store instead of puppet's store 3. expose the certificate trust and verification mode as parameters on the file type (this is truly awful) ... surely someone else has better ideas ...

Add Comment

[Paul Mansfield (Jira)]

Paul Mansfield commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used discovered this bug came back https://tickets.puppetlabs.com/browse/PA-3185

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814

HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Acceptance Criteria: It should be possible to retrieve file metadata and content from the following sources: 1. From the puppetserver (already works) 2. From a server whose CA cert is contained in the puppet-agent CA bundle (already works) 3. From a server whose CA cert is not in the puppet-agent CA bundle, but is in a file or directory system location, eg /etc/pki/tls/cert.pem, and puppet is configured to use that location. If referring to a directory, it is up to the user to rehash the directory (new). To support this: 1. Add a new puppet setting of type {{file_or_directory} 2. Default to nil 3. Update {{Puppet::SSL::SSLProvider#create_system_context}} to call {{OpenSSL::X509::Store#add_file}} or {{add_path}} if the setting is not-nil and the file or directory exists, respectively. 4. Update spec/integration/http/client_spec to ensure it makes an HTTPS connection to a test server when the CA cert is not referenced by {{SSL_CERT_FILE}}, but is referenced by the new setting.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Sprint: Platform Core KANBAN

Add Comment

[Josh Cooper (Jira)]

Josh Cooper assigned an issue to Josh Cooper

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Assignee: Josh Cooper

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Acceptance Criteria: It should be possible to retrieve file metadata and content from the following http(s) sources:

1. From the puppetserver (already works) 2. From a server whose CA cert is contained in the puppet-agent CA bundle (already works) 3. From a server whose CA cert is not in the puppet-agent CA bundle, but is in a file or directory system location, eg /etc/pki/tls/cert.pem, and puppet is configured to use that location. If referring to a directory, it is up to the user to rehash the directory (new). To support this: 1. Add a new puppet setting of type {{file_or_directory} 2. Default to nil 3. Update {{Puppet::SSL::SSLProvider#create_system_context}} to call {{OpenSSL::X509::Store#add_file}} or {{add_path}} if the setting is not-nil and the file or directory exists, respectively. 4. Update spec/integration/http/client_spec to ensure it makes an HTTPS connection to a test server when the CA cert is not referenced by {{SSL_CERT_FILE}}, but is referenced by the new setting.

Add Comment

[Melissa Stone (Jira)]

Melissa Stone commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used Merged into master with https://github.com/puppetlabs/puppet/commit/299b808a40e720c2d3e6cf44abe291fce36c7f5f

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-7814

Re: HTTPS file sources with non-puppet-trusted certs can't be used

Passed CI in 9b5bd2ad8b180efa4c143798a1a2b362928f1f1c

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814

HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Release Notes: New Feature Release Notes Summary: Puppet agents can be configured to trust additional CA certificates when connecting to HTTPS servers whose issuing CA certificate is neither the puppet CA nor in the CA cert bundle in the puppet-agent package. This can be accomplished by setting "ssl_trust_store" to a file containing CA cert(s) in PEM format concatenated together. For example, "/etc/pki/tls/cert.pem". Note the setting only applies when puppet makes an HTTPS request to non-puppet REST endpoints, such as when applying a file resource with an "https://" source parameter or running "puppet module install" to install modules from the forge. Puppet REST requests, such as retrieving a catalog, will only trust the puppet CA regardless of the ssl_trust_store setting. Previously the only way to trust additional CA certificates was to manually add the certificate to puppet-agent's CA bundle, however, those changes were lost as soon as the puppet-agent package was next updated.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Fix Version/s: PUP 6.17.0

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Josh Cooper Acceptance Criteria:

It should be possible to retrieve file metadata and content from the following http(s) sources: 1. From the puppetserver (already works) 2. From a server whose CA cert is contained in the puppet-agent CA bundle (already works)

3. From a server whose CA cert is not in the puppet-agent CA bundle, but is in a file or directory system location, eg /etc/pki/tls/cert.pem, and puppet is configured to use that location. If referring to a directory, it is up to the user to rehash the directory file (new). To support this: 1. Add a new puppet setting of type {{ file_or_directory :file } }

2. Default to nil 3. Update {{Puppet::SSL::SSLProvider#create_system_context}} to call {{OpenSSL::X509::Store#add_file}}

or {{add_path}} if the setting is not-nil and the file or directory exists, respectively .

4. Update spec/integration/http/client_spec to ensure it makes an HTTPS connection to a test server when the CA cert is not referenced by {{SSL_CERT_FILE}}, but is referenced by the new setting.

Add Comment

[Claire Cadman (Jira)]

Claire Cadman updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: Claire Cadman Labels: doc_reviewed

Add Comment

[zendesk.jira (Jira)]

zendesk.jira updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: zendesk.jira Labels: doc_reviewed jira_escalated

Add Comment

[zendesk.jira (Jira)]

zendesk.jira updated an issue

Puppet / PUP-7814 HTTPS file sources with non-puppet-trusted certs can't be used

Change By: zendesk.jira Zendesk Ticket Count: 2 3 Zendesk Ticket IDs: 32650,35349 ,43200

Add Comment