Location of Cached SSL root certificate (maybe)

[Antony Nelson]

On the first of October all of my older computers started to get an error with the SSL certificate of our munki installation.

Download error -1202: The certificate for this server is invalid. You might be connecting to a server that is pretending to be ‚ '<our fqdn>'

SSL error detail: (-9814, 'Chain had an expired cert')

Headers: None

ERROR: Could not retrieve managed install primary manifest.

Newer machines (installed in last few days) work just fine,  and contacting the server directly via a web browser shows it to have a valid certificate with a valid certificate chain. Installing munki over the top of the existing munki installation doesn't fix the problem. 

I have searched keychain for any invalid certificates on the affected computers and this doesn't appear to be where the expired certificate is located.

My hypothesis is that munki has a cached root certificate which needs to be removed for the replacement certificate to be accepted.

Is this plausible? (if not what is the probable cause)

how can I fix it?

Thanks kindly for any help

[Antony Nelson]

ok I have got a little further with this but I am still stuck.

The offending certificate is in the keychain access listed under "System Roots" Doh! (good old screen blindness)

The certificate is "DST Root CA X3" which expired on the 30 Sep 2021 at 15:01:15 

The certificate was replaced with "ISRG Root X1" which is the root certificate that is shown when accessing the server using a web browser. 

The facility to remove root certificates through the GUI has been removed from Catalina so instructions along the lines of right click and select delete do not work.

Via the command line

sudo security delete- certificate -c  "DST Root CA X3" / System/Library/Keychains/SystemRootCertificates.keychain

results in the error message 

security: SecKeychainItemDelete: UNIX[Operation not permitted]

The suggested work around 

saving the root certificate as a .cer (right click and export via keychain access) then running: 

sudo security add-trusted-cert -d -r deny -k "/Library/Keychains/System.keychain" certname.cer

Changes the message in the munki error log from 

SSL error detail: (-9814, ’Chain had an expired cert')

to

SSL error detail: (-9807, ‘Invalid certificate chain’)

It changes the certificate from being "certificate has expired" to " This certificate is marked as not trusted for all users" in keychain access but munki purists in rejecting the certificate.

sudo security add-trusted-cert -d -r trustRoot -k "/Library/Keychains/System.keychain" certname.cer

sudo security add-trusted-cert -d -r unspecified -k "/Library/Keychains/System.keychain" certname.cer

Result in 

SSL error detail: (-9814, ’Chain had an expired cert')

Setting “trustRoot” removes the expiration notice from keychain access and I suspect would revalidate the certificate to other applications.

Setting “trustAsRoot” results in the error 

SecTrustSettingsSetTrustSettings: One or more parameters passed to a function were not valid.

But I suspect would not be appropriate.

This is pure speculation but it is starting to look like a significant issue with munki not using Apples API for certificate verification and apple removing the ability to remove a certificate because it's not necessary if you use Apples certificate verification API.

Once again any help would be greatly appreciated.

[Gregory Neagle]

I’m sorry I cannot be of more help since I know very little about SSL/TLS.

I can assure you that Munki _does_ use Apple’s APIs for certificate verification.

Safari might work for you possibly because Safari might have an additional CA store it uses beyond the default system store.

Curious what you see if you attempt to use `nscurl` to access your server.

-Greg

-- 
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/616d63b6-d48b-43bb-a4c2-1425cc45580en%40googlegroups.com.

[Daniel Moore]

Sounds like you might have been affected by the expiration of a root certificate in older OS versions. See  https://eclecticlight.co/2021/09/21/el-capitan-and-older-mac-os-x-are-about-to-have-a-security-certificate-problem/.

On Oct 6, 2021, at 8:57 AM, Antony Nelson <antony...@gmail.com> wrote:

On the first of October all of my older computers started to get an error with the SSL certificate of our munki installation.

--
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/0af04aa8-75b3-4645-ac5e-2931c497cc5dn%40googlegroups.com.

[Antony Nelson]

Hi Greg, 

the information about munki using Apple’s APIs for certificate verification was useful, 

nscurl https://munki.<fqdn>/manifests/<serialnumber>

returns the manifest as expected.

It looks like people are starting to realise it may be part of a fairly major problem for anyone using Let’s Encrypt certificates in MacOS for https access outside of a web browser. 

I'm hoping that performing an unnecessary update of the servers ssl certificate will get around the problem.

[Antony Nelson]

It's slightly miss leading since I'm using Catalina but this article does have a note at the bottom which is  relevant.

I gather the "DST Root CA X3" expiration affects all OS versions as the failed certificate is stored in a security database. 

The most relevant details can be found here

https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/   

but I don't think there is a client side fix for OSX as the security database isn't user editable.

a new server certificate seems like the only solution. I will confirm if this fixes things when I have tried it.

[Antony Nelson]

I re-newed certificate from lets encrypt failed to fix the issue,  but a new certificate from a different provider did.