Jira (PUP-11349) Puppet agent ssl_trust_store setting on Windows does not work

[Chris Southall (Jira)]

Chris Southall created an issue

Puppet / PUP-11349 Puppet agent ssl_trust_store setting on Windows does not work Issue Type: Bug Affects Versions: PUP 6.21.0 Assignee: Unassigned Components: Windows Created: 2021/11/13 11:25 AM Priority: Normal Reporter: Chris Southall Puppet Version: Puppet agent 6.x and 7.x Puppet Server Version: Tested on PE and open source OS Name/Version: Windows Server 2019 and Windows 10 Customer has a bunch of stuff staged in Artifactory. The Artifactory server is using SSL certificates from a local, private CA that is not trusted by the Puppet agent. On Linux we configure “puppet.conf” in the [agent] section with “ssl_trust_store = /etc/pki/…long-path…/ca-certs.pem” and all is well. I’m trying to configure Windows the same way and it fails the SSL verification. I confirmed the custom PEM file contains the trusted CA certificates and run puppet agent, but it fails the SSL validation when trying to download an "archive" resource. I know the contents of my custom PEM file are good.  If I append the custom .pem file to the default 'C:\Program Files\Puppet Labs\Puppet\puppet\ssl\cert.pem' I can run the agent without an SSL failure and all resources apply, but I’d prefer the puppet agent to use our custom trust file instead of modifying the bundled version, as we do with the Linux agents. Puppet version on this Windows agent was 7.12.0, but I also see this on 6.x agents. Desired Behavior: Windows puppet agent uses the SSL trust store file specified by "ssl_trust_store" in "puppet.conf" Actual Behavior: The Windows puppet agent ignores the "ssl_trust_store" entry and fails SSL validation to servers using certificates issued by a private CA. Example output from a Windows agent: PS C:\> puppet config print --section agent ssl_trust_store C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem PS C:\> puppet agent -t Info: Using environment 'production' …truncated… Info: Applying configuration version '1636583399' Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/winstall/manifests/product.pp, line: 85) Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/winstall/manifests/product.pp, line: 85) Wrapped exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) Error: /Stage[main]/Winstall/Winstall::Product[VMware Remote Console]/Archive[VMware Remote Console]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: /etc/puppetlabs/code/environments/production/modules/winstall/manifests/product.pp, line: 85) (corrective) ...subsequest resources are skipped due to the failure... Notice: Applied catalog in 5.94 seconds Add Comment

This message was sent by Atlassian Jira (v8.13.2#813002-sha1:c495a97)

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-11349 Puppet agent ssl_trust_store setting on Windows does not work

Change By: Ciprian Badescu Labels: needs_repro

Add Comment

[Ciprian Badescu (Jira)]

Ciprian Badescu updated an issue

Puppet / PUP-11349 Puppet agent ssl_trust_store setting on Windows does not work

Change By: Ciprian Badescu Team: Night's Watch

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work This is surprising. We explicitly test the ssl_trust_store setting on Ubuntu and Windows in https://github.com/puppetlabs/puppet/blob/0fc096f07ecc1b4a36ae6eabe5716b3f03bd9c4e/spec/integration/application/agent_spec.rb#L420-L469

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

Chris Southall does Puppet[:ssl_trust_store] contain multiple PEM files? Do they have unix or windows line endings? How is each cert delimited?

Add Comment

[Chris Southall (Jira)]

Chris Southall updated an issue

Puppet / PUP-11349

Puppet agent ssl_trust_store setting on Windows does not work

Change By: Chris Southall Attachment: Export-PEM.ps1

Add Comment

[Chris Southall (Jira)]

Chris Southall commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work The "ssl_trust_store" has a single file entry, as follows:

PS C:\> puppet config print --section agent ssl_trust_store C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem

I'm using a PowerShell script to generate the file from the Windows certificate store.  The idea was for Puppet to create the file and to refresh it any time we add/remove certificates from the Windows store.  I attached the PowerShell script to the ticket as "Export-PEM.ps1".

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

Chris Southall powershell < 6 will write a BOM when using -Encoding UTF8, which I'm fairly certain openssl does not handle. Since the certs are base64 encoded try using -Encoding ASCII instead. See https://stackoverflow.com/questions/5596982/using-powershell-to-write-a-file-in-utf-8-without-the-bom

Add Comment

[Chris Southall (Jira)]

Chris Southall commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

The UTF8 encoded output of the script was an issue, so thanks for pointing that out.  I updated the PowerShell script to use ASCII encoding I got the following results that seem to confirm there is still a problem. Beginning with the default configuration of the "ssl_trust_store" value (none) and the default Puppet CA trust file that does not contain my local CA.  A simple catalog with an "archive" resource errors out with an SSL failure, which is expected:

PS C:\> puppet config print --section agent ssl_trust_store

PS C:\> & puppet apply --modulepath=${mods} -e "include local" -t Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for win10.southall.local in environment production in 0.18 seconds Info: Using environment 'production' Info: Applying configuration version '1638492869' Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3) Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3)

Wrapped exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

Error: /Stage[main]/Local/Archive[C:\Windows\Temp\VMRC.exe]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3) Notice: Applied catalog in 0.28 seconds     Now I configured the agent to use the custom ssl_trust_store and generate the trust file using the updated PowerShell script with ASCII output:   PS C:\> puppet config set --section agent ssl_trust_store C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem

PS C:\> puppet config print --section agent ssl_trust_store C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem

PS C:\> .\Users\chris.SOUTHALL\Desktop\Export-PEM.ps1 C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs\ca-trust.pem PS C:\> dir C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem Directory: C:\ProgramData\PuppetLabs\puppet\etc\ssl\certs Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 12/2/2021 6:55 PM 95514 ca-trust.pem   With the newly generated trust file in place I expect the SSL connection to work, but the Puppet agent still fails: PS C:\> & puppet apply --modulepath=${mods} -e "include local" -t Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for win10.southall.local in environment production in 0.20 seconds Info: Using environment 'production' Info: Applying configuration version '1638492948' Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3) Error: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3)

Wrapped exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

Error: /Stage[main]/Local/Archive[C:\Windows\Temp\VMRC.exe]/ensure: change from 'absent' to 'present' failed: Could not set 'present' on ensure: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate) (file: C:/Users/chris.SOUTHALL/puppet/modules/local/manifests/init.pp, line: 3) Notice: Applied catalog in 0.28 seconds   Without changing the Puppet agent configuration, if I simply overwrite the default trust file with my newly exported custom trust file, the agent is able to apply the catalog.  This seems to indicate the the trust file is valid, but not being used by the Puppet agent per the "ssl_trust_store" setting: PS C:\> copy C:/ProgramData/PuppetLabs/puppet/etc/ssl/certs/ca-trust.pem 'C:\Program Files\Puppet Labs\Puppet\puppet\ssl\cert.pem' PS C:\> & puppet apply --modulepath=${mods} -e "include local" -t Info: Loading facts Info: Loading facts Info: Loading facts Notice: Compiled catalog for win10.southall.local in environment production in 0.20 seconds Info: Using environment 'production' Info: Applying configuration version '1638492995' Notice: /Stage[main]/Local/Archive[C:\Windows\Temp\VMRC.exe]/ensure: download archive from https://vr.southall.local/sw/VMware-VMRC-12.0.1-18113358.exe to C:\Windows\Temp\VMRC.exe without cleanup Notice: Applied catalog in 7.69 seconds PS C:\>   As far as I can tell, the the SSL configuration for the Windows puppet agent is coming from 'C:\Program Files\Puppet Labs\Puppet\bin\environment.bat', which is sourced when the agent starts.  While that is fine as a default, if the "ssl_trust_store" is defined in "puppet.conf" I expect it to be the effective value.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

The ssl_trust_store setting only works with puppet's http client. The archive module uses raw Net::HTTP, so you'd need to patch the type/provider.

Add Comment

[Chris Southall (Jira)]

Chris Southall commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

That makes sense.  I'll take a look at that module and see what I can do.  Thanks for pointing this out.

Add Comment

[Chris Southall (Jira)]

Chris Southall commented on PUP-11349

Re: Puppet agent ssl_trust_store setting on Windows does not work

Pull request submitted for the "puppet-archive" module to address this issue. https://github.com/voxpupuli/puppet-archive/pull/467

Add Comment

This message was sent by Atlassian Jira (v8.20.2#820002-sha1:829506d)