Jira (PUP-10639) renew cached Puppet CA on Puppet client

30 views
Skip to first unread message

Andreas Paul (Jira)

unread,
Aug 20, 2020, 7:41:04 AM8/20/20
to puppe...@googlegroups.com
Andreas Paul created an issue
 
Puppet / New Feature PUP-10639
renew cached Puppet CA on Puppet client
Issue Type: New Feature New Feature
Assignee: Unassigned
Created: 2020/08/20 4:40 AM
Priority: Normal Normal
Reporter: Andreas Paul

Puppet Version: all
Puppet Server Version: irrelevant
OS Name/Version: irrelevant**

Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.

Desired Behavior:

  • add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
  • add Puppet client config settings to specify intervals for this CA check
  • if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

Actual Behavior:

 

Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired localy cached CA file)

Add Comment Add Comment
 
This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)
Atlassian logo

Andreas Paul (Jira)

unread,
Aug 20, 2020, 7:42:03 AM8/20/20
to puppe...@googlegroups.com
Andreas Paul updated an issue
Change By: Andreas Paul
*Puppet Version:* all
*Puppet Server Version:* irrelevant
*OS Name/Version:* irrelevant


Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.

*Desired Behavior:*
* add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
* add Puppet client config settings to specify intervals for this CA check
* if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

*Actual Behavior:*


Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired localy locally cached CA file)

Andreas Paul (Jira)

unread,
Aug 20, 2020, 7:42:03 AM8/20/20
to puppe...@googlegroups.com
Andreas Paul updated an issue
*Puppet Version: all * all
*Puppet Server Version:
irrelevant * irrelevant
*OS Name/Version:* 
* irrelevant ***

Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.



*Desired Behavior:*

* add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
* add Puppet client config settings to specify intervals for this CA check
* if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

*Actual Behavior:*

 

Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired localy cached CA file)

Andreas Paul (Jira)

unread,
Aug 20, 2020, 10:04:04 AM8/20/20
to puppe...@googlegroups.com
Andreas Paul updated an issue
*Puppet Version:* all
*Puppet Server Version:* irrelevant
*OS Name/Version:* irrelevant


Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.

*Desired Behavior:*
* add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
* add Puppet client config settings to specify intervals for this CA check
* if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

*Actual Behavior:*

Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired locally cached CA file)

Josh Cooper (Jira)

unread,
Aug 20, 2020, 1:58:03 PM8/20/20
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-10639
 
Re: renew cached Puppet CA on Puppet client

Updating the local CA file when the file doesn't exist yet (new agent) is a weak link in puppet's security model, because it's the only time that puppet agents don't authenticate the server (VERIFY_NONE). To avoid MTIM attacks, puppet has a ca_fingerprint command line argument which can be used to verify the contents of the CA file over an untrusted channel. The argument accepts the SHA256 digest of the CA file, which can be computed on the CA server as:

# openssl dgst -sha256 -r /etc/puppetlabs/puppet/ssl/certs/ca.pem | cut -f1 -d' '
6e09bc3e6778d6b497d233a49f9906b95adc212468d9d3e260398f093badaf03

Then on the agent, pass the CA fingerprint when bootstrapping the agent (or running puppet agent -t. Puppet will reject the file if the digests don't match:

# puppet ssl bootstrap --ca_fingerprint FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF
Error: CA bundle with digest (SHA256) 6E:09:BC:3E:67:78:D6:B4:97:D2:33:A4:9F:99:06:B9:5A:DC:21:24:68:D9:D3:E2:60:39:8F:09:3B:AD:AF:03 did not match expected digest FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF:FF

So if we allow agents to refresh an expired CA certificate, then we have two problems:

1. If ca_fingerprint is not set (the default), then it expands the window of opportunity when an agent could be vulnerable to MITM.
2. If ca_fingerprint is set, then won't match the new CA file.

The first issue could be handled by ignoring the expiration date of the CA certificate for the TLS connection that downloads the new CA certificate. All other errors (like bad signature would continue to fail the connection). This could be done by passing a custom Puppet::SSL::Verifier.

The second issue could be handled by only applying the ca_fingerprint to the very first CA file download. Once we have a trust anchor, then we can use that CA to download the next once (ignoring the expiration date issue above), preserving the chain of trust.

Josh Cooper (Jira)

unread,
Jul 26, 2022, 5:07:01 PM7/26/22
to puppe...@googlegroups.com
Josh Cooper updated an issue
 
Change By: Josh Cooper
Team: Phoenix
This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)
Atlassian logo

Tim Meusel (Jira)

unread,
Jan 5, 2023, 4:36:02 AM1/5/23
to puppe...@googlegroups.com
Tim Meusel commented on New Feature PUP-10639
 
Re: renew cached Puppet CA on Puppet client

Hi,
as a first step, what about a refresh interval, like we have it for the CRL, to update a not expired CA cert? The agent can establish the usual HTTPS connection and if that works, download a new CA cert. That would enable people to use short-lived certificated and automate their rollover.

Josh Cooper (Jira)

unread,
Jan 5, 2023, 12:22:03 PM1/5/23
to puppe...@googlegroups.com
Josh Cooper commented on New Feature PUP-10639

Yep, that makes sense to me.

David Piekny (Jira)

unread,
Feb 28, 2023, 2:23:01 PM2/28/23
to puppe...@googlegroups.com
David Piekny updated an issue
 
Change By: David Piekny
Epic Link: PE-34949

Josh Cooper (Jira)

unread,
Apr 20, 2023, 1:33:01 PM4/20/23
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
*Puppet Version:* all
*Puppet Server Version:* irrelevant
*OS Name/Version:* irrelevant

Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.

#puppet slack channel discussion: [https://puppetcommunity.slack.com/messages/C0W298S9G/p1597402361432800]

*Desired Behavior:*
* add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
* add Puppet client config settings to specify intervals for this CA check
* if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

*Actual Behavior:*

Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired locally cached CA file)


Grooming:
* Create new puppet setting ca_refresh_interval, similar to crl_refresh_interval. It should by default. TBD the default setting
* The agent should use the {{If-Modified-Since}} HTTP header so that it only downloads the CA if it's been modified on the server.
* If the CA bundle is not updated, it should continue using its local CA.
* If the CA bundle is updated, the agent should use the new CA bundle for all future HTTPS connections.
* If the CA bundle is updated, then we must update the CRL, as there may be a new intermediate CA and we need a corresponding CRL to verify the full chain.
* Need to understand what happens if the CRL nextUpdate time expires. Do agent runs fail?
* Need unit tests for Puppet::SSL::StateMachine similar to what's done in spec/unit/ssl/state_machine_spec.rb for CRL refreshing.

Josh Cooper (Jira)

unread,
Apr 20, 2023, 1:35:02 PM4/20/23
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Story Points: 5

Josh Cooper (Jira)

unread,
Apr 20, 2023, 1:35:02 PM4/20/23
to puppe...@googlegroups.com
Josh Cooper updated an issue
*Puppet Version:* all
*Puppet Server Version:* irrelevant
*OS Name/Version:* irrelevant

Currently the Puppet agent/client does cache the Puppet CA file it first received forever. When this CA expires and get renewed on the configured `ca_server` or Puppetserver the cache still does not get updated with the new CA.

#puppet slack channel discussion: [https://puppetcommunity.slack.com/messages/C0W298S9G/p1597402361432800]

*Desired Behavior:*
* add sane defaults that would recheck if the cached local CA does differ from the one used by the configured `ca_server` (e.g. 1 week or 2 weeks before CA expiration date)
* add Puppet client config settings to specify intervals for this CA check
* if the local cached CA file is expired check if the configured `ca_server` uses a different one and replace the local  cached CA file

*Actual Behavior:*

Currently the client does use the cached CA file forever even when the CA file has expired or the used CA file of the configured `ca_server` or Puppetserver was renewed.

This results in completely failed Puppet runs, which will never recover without external/manual intervention (removing the expired locally cached CA file)

Grooming:
* Create new puppet setting ca_refresh_interval, similar to crl_refresh_interval. It should by default. TBD the default setting
* The agent should use the {{If-Modified-Since}} HTTP header so that it only downloads the CA if it's been modified on the server.
* If the CA bundle is not updated, it should continue using its local CA.
* If the CA bundle is updated, the agent should use the new CA bundle for all future HTTPS connections.
* If the CA bundle is updated, then we must update the CRL, as there may be a new intermediate CA and we need a corresponding CRL to verify the full chain.
* What should the behavior be if the local CA cert is expired? Fail like we do now or always attempt to download a new CA bundle (even if the next update time hasn't occurred yet).
*
Need to understand what happens if the CRL nextUpdate time expires. Do agent runs fail?

* Need unit tests for Puppet::SSL::StateMachine similar to what's done in spec/unit/ssl/state_machine_spec.rb for CRL refreshing.

Josh Cooper (Jira)

unread,
May 17, 2023, 2:24:01 PM5/17/23
to puppe...@googlegroups.com
Josh Cooper assigned an issue to Josh Cooper
Change By: Josh Cooper
Assignee: Josh Cooper

Josh Cooper (Jira)

unread,
May 18, 2023, 3:42:03 PM5/18/23
to puppe...@googlegroups.com
Josh Cooper updated an issue
Change By: Josh Cooper
Sprint: Phoenix 2023-06-07
This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages