Munki 5.5 w/ keychain client cert causing NSURLErrorNetworkConnectionLost

342 views
Skip to first unread message

Shawn Maddock

unread,
Jul 1, 2021, 7:38:09 PM7/1/21
to munki-discuss
When managedsoftwareupdate 5.5.0.4360 runs as root (e.g., via the LaunchDaemon) and tries to authenticate to a server via client certificate stored in the System keychain, it returns the error: Download failed: (-1005, ’The network connection was lost.’)

I've tested this on a bare metal Intel Mac mini and on VMs, all running clean installs of macOS 11.4.

If I run managedsoftwareupdate under a user account with sudo, it asks for keychain credentials and then downloads fine. Also if I copy the cert into /Library/Managed Installs/certs it works fine as root.


Apple's support documentation https://developer.apple.com/library/archive/qa/qa1941/_index.html suggests a network issue or diagnosing with CFNetwork. Since it's obviously a root vs user issue, and I don't know how to use CFNetwork, I'm posting here for help. I've checked env variables and file permissions and can't find any smoking guns.

Steps to reproduce:
Install a client certificate via Configuration Profile.
Install Munki 5.5 from the GitHub releases.
Set the SoftwareRepoURL to a server with mTLS configured for the client certificate installed on the Mac.
Run managedsoftwareupdate as root (not just sudo)

Attaching relevant logs of sudo (working) and root (not working).

root.txt
sudo.txt

Gregory Neagle

unread,
Jul 1, 2021, 7:48:38 PM7/1/21
to 'Gregory Neagle' via munki-discuss
How does managedsoftwareupdate know _which_ client cert to use when it communicates with your Munki server?

When you provide a cert in /Library/Managed Installs/certs, Munki has all the information it needs to know which one to use. If you have MDM install one to the System keychain -- how does managedsoftwareupdate tell which cert to use?

You note "If I run managedsoftwareupdate under a user account with sudo, it asks for keychain credentials " -- so you are _telling_ it which one to use.

(I don't know the answer to this question either -- I don't use client certs myself, and have never tried to use a client cert installed via MDM)

-Greg

--
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/81c9c0be-1278-4f06-af38-4cc202a22df3n%40googlegroups.com.
<root.txt><sudo.txt>

Shawn Maddock

unread,
Jul 1, 2021, 8:50:57 PM7/1/21
to munki-...@googlegroups.com
As part of the SSL handshake, the server sends a list of client cert names it will accept. msu loops through all the ones in the keychain to see if there’s a match. The logs show this much is working, it finds the matching cert.

To be clear, I didn’t use MDM to install the cert, I just created a mobileconfig and double-clicked to install it in System Prefs. Any profile-installed cert gets put in the System keychain. When I run msu as sudo, I’m not selecting a specific cert or even keychain, msu just needs elevated privileges to access they keychains.

-shawn

You received this message because you are subscribed to a topic in the Google Groups "munki-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/munki-discuss/Hvlqfx-bWD0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to munki-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/29AAE5C8-2FB5-488E-B854-6B354E00A672%40mac.com.

Rob Renstrom

unread,
Jul 1, 2021, 9:08:00 PM7/1/21
to munki-...@googlegroups.com
Is it possible that you have a matching cert in another keychain (other than the System keychain) that’s in the keychain search path but locked?

-rob

On Jul 1, 2021, at 5:50 PM, Shawn Maddock <smad...@gocirrus.com> wrote:



Shawn Maddock

unread,
Jul 1, 2021, 9:14:40 PM7/1/21
to munki-discuss
No, the test device and VM were wiped with clean installs of macOS, it’s a private CA, and the server is only listing a single name to match against. The root certificate is installed as part of the chain of trust, but it’s also in the System keychain.

Happy to provide unredacted logs offline if anyone thinks it would help.

-shawn

Daniel E.

unread,
Jul 8, 2021, 1:20:38 PM7/8/21
to munki-discuss
Hi shawn,

we just setup our first munki infrastructure and have exactly the same problem. We created a CA on our MDM server and deployed certificates using SCEP.

Running sudo managedsoftwareupdate by the user brings up the python prompt for Keychain Access. The launch daemons or anything else will fail.

So I can confirm your problem but unfortunately, I don't have a solution except maybe switching to username/password basic auth.

Rob Renstrom

unread,
Jul 10, 2021, 10:46:08 PM7/10/21
to munki-discuss
The problem here is that Munki doesn't have access to the private key for the certificate, since it's protected by a keychain access control list. 

The new logic in Munki 5.5 finds the server requested cert in a keychain, but without access to the key, the authentication challenge will fail. 

This works for the keychain Munki creates from /Library/Managed Installs/certs/, since it imports the cert with Allow any application to access this key without warning.

For MDM deployed certificates, to allow Munki to access the private key, set AllowAllAppsAccess to true, in the certificate payload.



To test this with a manually added cert/key in the System keychain, use the Keychain Access app, select the private key, choose File > Get Info, and in the Access Control pane set "Allow all applications to access this item". 

Additional info in this old issue: 


-rob

Shawn Maddock

unread,
Jul 13, 2021, 12:02:15 PM7/13/21
to munki-discuss
Thank you so much. This resolved the issue we were experiencing.

There's also now a pull request to add that key to the third party GUI profile editors: https://github.com/ProfileCreator/ProfileManifests/pull/449

-shawn

Gregory Neagle

unread,
Jul 13, 2021, 12:05:22 PM7/13/21
to munki-...@googlegroups.com
It would be extraordinarily helpful for someone to write up documentation on this for the Munki wiki.

If you are looking for a way to contribute back to the Munki project, this would be a good way.

-Greg

Message has been deleted

subrat kheti

unread,
Nov 11, 2021, 9:06:56 AM11/11/21
to munki-discuss
We are also facing the same issue. As per mentioned above is this the right wrokaround? Can someone please confirm.

For MDM deployed certificates, to allow Munki to access the private key, set AllowAllAppsAccess to true, in the certificate payload.



Shawn Maddock

unread,
Nov 11, 2021, 9:16:11 AM11/11/21
to munki-...@googlegroups.com
Rob’s solution of setting AllowAllAppsAccess to true is the correct one.

-shawn

subrat kheti

unread,
Nov 11, 2021, 10:01:30 AM11/11/21
to munki-...@googlegroups.com
Thanks Shawn for the update.

--
Thanks & Regrads,
Subrat Kumar Kheti
Mob:09160111547
Reply all
Reply to author
Forward
0 new messages