Jira (PUP-5349) only the ca server is veryfing puppet client certificates

[cristi falcas (JIRA)]

cristi falcas created an issue

Puppet / PUP-5349 only the ca server is veryfing puppet client certificates Issue Type: Bug Affects Versions: PUP 3.8.3 Assignee: Unassigned Components: Puppet Server Created: 2015/10/09 2:58 PM Priority: Normal Reporter: cristi falcas It seems that only the ca server is verifying the puppet certificates for clients. We revoked the certificate for one of our servers and copied the new crl.pem to a second master. But it looks like the second master doesn't care about this at all. Bellow are 2 runs from the same client to those 2 masters: [root@v-is-puppet-02 sdt_user]# puppet agent -t --server puppet.company.net Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect returned=1 errno=0 state=SSLv3 read server session ticket A: sslv3 alert certificate revoked Info: Retrieving pluginfacts [root@v-is-puppet-02 sdt_user]# puppet agent -t --server v-so-repo-01.company.net Info: Retrieving pluginfacts Info: Retrieving plugin Add Comment

This message was sent by Atlassian JIRA (v6.4.11#64026-sha1:78f6ec4)

[Christopher Price (JIRA)]

Christopher Price commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates Did you restart the server after copying over the new CRL? Also, what path did you copy the CRL file to?

Add Comment

[Christopher Price (JIRA)]

Christopher Price assigned an issue to cristi falcas

Puppet / PUP-5349

only the ca server is veryfing puppet client certificates

Change By: Christopher Price Assignee: cristi falcas

Add Comment

[Eric Sorenson (JIRA)]

Eric Sorenson commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates cristi falcas I'm sorry I don't understand what you expect to happen. Which is the certificate that you revoked? Does the CRL on the agent have an entry for the revoked master? Why would the master connect to the other master?

Add Comment

[Leif M (Jira)]

Leif M updated an issue

Puppet / PUP-5349

only the ca server is veryfing puppet client certificates

Change By: Leif M Affects Version/s: PUP 6.3.0

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Leif M (Jira)]

Leif M commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates I've re-opened this ticket as I seem to be encountering the same issue with Puppet server 6.3.3. We have 3 Puppet servers behind a load balancer + our Puppet CA, which is not itself behind the LB. When we revoke a certificate on the CA, the host with the revoked certificate is no longer able to execute agent runs against the CA server, but agent runs continue to work against the LB or any of the individual servers behind it. After revoking a host's certificate, the puppet CA server rejects its agent runs. I've confirmed that agent runs work against the CA before the cert is revoked. $ /opt/puppetlabs/bin/puppet agent -t --server puppet-ca.example.com Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.069 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: No more routes to puppet Info: Retrieving pluginfacts Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.049 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': No more routes to fileserver Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.069 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: No more routes to fileserver Info: Retrieving plugin Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.056 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: /File[/opt/puppetlabs/puppet/cache/lib]: Failed to generate additional resources using 'eval_generate': No more routes to fileserver Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.058 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: /File[/opt/puppetlabs/puppet/cache/lib]: Could not evaluate: Could not retrieve file metadata for puppet:///plugins: No more routes to fileserver Info: Loading facts Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.055 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: Could not retrieve catalog from remote server: No more routes to puppet Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run Error: Request to https://puppet-ca.example.com:8140/puppet/v3 failed after 0.063 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Wrapped exception: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown Error: Could not send report: No more routes to report But agent runs continue to work when executed against the load balancer or any of our 3 other puppet servers. $ /opt/puppetlabs/bin/puppet agent -t --server puppet-server.example.com Info: Using configured environment 'ENVIRONMENT'

Info: Retrieving pluginfacts Info: Retrieving plugin

Info: Retrieving locales Info: Loading facts Info: Caching catalog for a-host.example.com Info: Applying configuration version '1589922090' Notice: Applied catalog in 4.97 seconds I have tried restarting the puppetserver service on the CA as well as the other individual Puppet server I've been using to test. I've tried copying crl.pem from the CA to the server I am using to test and placing it in both of these locations (since I found it in both locations on the CA): /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem /etc/puppetlabs/puppet/ssl/crl.pem I then restarted the puppetserver service on the server; this didn't have any effect. Up until this point I was naively working under the assumption that the masters were checking in with the CA regarding cert validity, but that doesn't seem to be the case.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Does the load-balancer terminate the SSL connection? If yes, then the CA's CRL needs to be copied to where ever the LB looks for the file, and possibly reloaded. If no, then it can't check the validity of the client's certificate since all the load-balancer sees is encrypted traffic. So then each puppetserver terminates the SSL connection, and the CRL from the CA needs to be copied to each puppetserver. Older versions of puppetserver need to be explicitly HUP'ed or restarted, but newer versions don't require that: https://puppet.com/docs/puppetserver/latest/crl_refresh.html Can you provide the CRLs from a puppetserver that accepts connections from the client and a CRL from the CA that rejects connections?

Add Comment

[Leif M (Jira)]

Leif M commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

I can provide the CRL/serial number if necessary, but I don't think there is a problem with the CRL, because the CA server is rejecting agents appropriately when agents with revoked certificates try to run against the CA. SSL termination is on the puppet servers, not the LB. As I mentioned, agent runs work against the LB and against the servers directly, and for that reason I've just been testing directly against one of the individual puppet servers. As I mentioned, I found the CRL in two different locations under /etc/puppetlabs on the CA, so I went ahead & tried to copy it to those same two locations on the puppet server that I am using to test, but it didn't have any effect. Those locations are: /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem /etc/puppetlabs/puppet/ssl/crl.pem It could just be that the CRL needs to be dropped somewhere else on the other puppet servers. I know that the puppetserver service isn't supposed to need a restart to pick up CRL updates, but at this point it seemed like a reasonable thing to try. Thanks for looking at this.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

The CA writes to /etc/puppetlabs/puppet/ssl/ca/ca_crl.pem when revoking a cert. Puppetserver watches for changes to that file and copies it to /etc/puppetlabs/puppet/ssl/crl.pem which is what puppetserver and the local puppet agent read from. Did you disable the CA service for the puppetservers that are not the CA, see https://puppet.com/docs/puppetserver/5.3/external_ca_configuration.html#disabling-the-internal-puppet-ca-service

Add Comment

[Rob Braden (Jira)]

Rob Braden commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Leif M Any updates on this issue? Were you able tor resolve your issue?

Add Comment

[Leif M (Jira)]

Leif M commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Hey Rob - This is still an issue for us. Our other masters have the CA service disabled; only the CA has CA services enabled.   Our other puppetservers, including the one I am specifically testing against: /etc/puppetlabs/puppetserver/services.d/ca.cfg: # To enable the CA service, leave the following line uncommented #puppetlabs.services.ca.certificate-authority-service/certificate-authority-service # To disable the CA service, comment out the above line and uncomment the line below puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service   And on the single puppetserver serving as the CA: /etc/puppetlabs/puppetserver/services.d/ca.cfg: # To enable the CA service, leave the following line uncommented puppetlabs.services.ca.certificate-authority-service/certificate-authority-service # To disable the CA service, comment out the above line and uncomment the line below #puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service   Must all puppetservers have the CA service enabled in order for certificate revocation to work properly?

Add Comment

[Maggie Dreyer (Jira)]

Maggie Dreyer commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

No, shouldn't need the CA enabled for the CRL to work. That part is handled by Jetty.

Add Comment

[Leif M (Jira)]

Leif M commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Hey Maggie – Thanks! Initially, that ticket seemed like the opposite issue to what I'm experiencing – my non-CA puppet masters didn't seem to care about the CRL at all no matter where I put it. But I believe this issue has led me to the solution. My non-CA puppet masters did not have the ssl-crl-path }}specified in {{{{/etc/puppetlabs/puppetserver/conf.d/webserver.conf}}. After taking the following steps, I was able to get one of my Puppet masters to respect revoked certs in the CRL: 1. Copy the most recent crl.pem from puppet-ca1:/etc/puppetlabs/puppet/ssl/crl.pem => puppet-master1:/etc/puppetlabs/puppet/ssl/crl.pem My understanding is that the Puppet agent should handle updating this file on clients (such as non-CA masters) per the crl_refresh_interval, but it should be noted that the default crl_refresh_interval is *none* - https://puppet.com/docs/puppet/latest/configuration.html#crl_refresh_interval - meaning that this file will not be updated on clients/masters unless this default configuration is specifically modified.** 2. Modify /etc/puppetlabs/puppetserver/conf.d/webserver.conf to include ssl-crl-path: {{ ...}} {{ ssl-ca-cert: /etc/puppetlabs/puppet/ssl/certs/ca.pem}} {{ ssl-crl-path: /etc/puppetlabs/puppet/ssl/crl.pem}} {{ idle-timeout-milliseconds: 30000}} {{ ...}} 3. Restart puppetserver service on puppet-master1 to apply change to webserver.conf. Attempting to perform a puppet agent run against puppet-master1 now fails with a certificate error, as it should: {{ [root@test-machine ~]# /opt/puppetlabs/bin/puppet agent -t --server puppet-master1}} {{ Error: Request to https://puppet-master1:8140/puppet/v3 failed after 0.09 seconds: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown}}

{{ Wrapped exception:}} {{ SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown}}

{{ ...}} So it looks like there are two issues that were preventing this from working as I would expect: Missing ssl-crl-path in our puppet masters' webserver.conf files. This is quite likely our fault, as the configuration of our webserver.conf files is coming from a template in a homebrew Puppet module. The default configuration may well contain this path, though I don't know for certain. Default crl_refresh_interval of none means that CRL updates on the CA will not propagate to masters unless this refresh interval is specifically modified. I would suggest that this may not be an ideal default, because clients with revoked certificates will still be able to run Puppet against masters with out-of-date CRLs. This could pose a security risk, depending on what sort of configuration is being applied.**   In my ignorance, I had originally believed that non-CA Puppet masters were verifying client certificates with the CA via its API. I don't know why I believed that. But this imagined behavior would be preferable in at least one way – there would be no propagation delay for cert revocation. As soon as a cert is revoked, it would cease to work, because that revocation info lives on one server. OTOH, it creates a single point of failure. Perhaps a system that checks w/ the CA but uses a local CRL as a fallback would be the ideal.

Add Comment

[Maggie Dreyer (Jira)]

Maggie Dreyer commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Glad you got it working, and thanks for the feedback about how to make the system better. Hopefully what you found might help some of the others on here.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-5349

Re: only the ca server is veryfing puppet client certificates

Default crl_refresh_interval of none The default was chosen because we didn't want to force all agents to refresh in a .y release. In the future, we will change the default. In the meantime, if there are specific doc changes that you'd recommend please let us know. The best thing is to leave input at the bottom of our docs pages, as that goes directly to our docs team. As things are working, I'm going to close this.

Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)