Munki and Ventura - early notes

[Greg Neagle]

TL;DR:

It looks like Munki is going to need a signed binary and an MDM-delivered PPPC/TCC config profile to continue working properly on macOS Ventura.

Details

Last week was Apple’s Worldwide Developer Conference 2022. Among others things announced and introduced was macOS Ventura, or macOS 13.

As is not unusual for macOS releases, this release contains changes that may affect the functionality of Munki (and other similar tools).

Two changes in particular have the potential to affect Munki functionality (and there may be more, but these were discussed in WWDC sessions):

1) A new “Login Items” section in the System Settings (formerly System Preferences) application. This settings view allows users to easily disable Login Items, Launch Agents, and Launch Daemons. Users require admin rights to disable Launch Daemons. Since Munki makes heavy use of Launch Agents and Launch Daemons, a user could easily and trivially disable Munki from running. At present there appear to be no management options/tools to prevent this. File Feedback with Apple about how important managing would be to your organization.

2) A new privacy protection around “App Management”. This prevents software from changing or removing items from /Applications without user approval. Its affect on Munki can be subtle and easy to miss in simple testing.

On a fresh Ventura install, Munki can install software as it does under Monterey and earlier. But when updating or removing software, it may be blocked from making changes to apps in /Applications. Again, this can be easy to miss in testing. It’s common to have Terminal set for Full Disk Access, and also common for ssh to have Full Disk Access (via /usr/libexec/sshd-keychain-wrapper). Running managedsoftwareupdate via either of these methods will allow Munki to update and/or remove app bundles in /Applications. But Munki operations triggered via launchd jobs will fail to update or remove app bundles in /Applications.

Apple has not yet provided supported/documented management controls to affect or manage this behavior. It _appears_ at present that managing/allowing Full Disk Access will allow processes to update or remove app bundles in /Applications, but I’d really like to see official Apple documentation and recommendations on this topic.

It is possible to “pre-approve” binaries or apps for Full Disk Access via an MDM-delivered profile, but the binary or app must be signed. In the case of Munki, this will most likely have to take the form of a signed binary wrapper/launcher that calls munki-python and the needed Munki scripted tools.

But this surfaces another problem. I don’t want to sign this binary with my personal Apple Developer ID and then distribute that as part of the official Munki release. And (while I have not asked) I suspect my employer doesn’t want to publicly distribute a Munki binary signed by them. While orgs using Munki could sign the tool themselves (and some will), many orgs will find this hard to do. I’d like to find some way to have a “Munki Project” or “Mac Admins” identity for a publicly-distributed release of the Munki tools. If anyone has useful thoughts, ideas, or information on how that might happen, I’d greatly appreciate it.

-Greg

[Ben Toms]

One caveat with the signing, the PPPCP might need updating every time Python is updated for Munki.

We found this: 

https://support.datajar.co.uk/hc/en-us/articles/4408958448017-Auto-Update-Code-Signing-Details

• identifier:/usr/local/munki/Python.framework/Versions/3.9/bin/python3.9
• type: path

So might be something that needs investigation too.

But as things are, the above link is what folks have needed with the signed Munki we deploy.

--
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/BBA2DCBF-2D89-4D32-856F-A892A42C0A48%40mac.com.

--

Regards,

Ben

[Greg Neagle]

I’m actually not suggesting signing munki-python.

I’m suggesting that “managedsoftwareupdate” become a fairly simple signed C/Obj-C binary that essentially calls `munki-python msu.py [managedsoftwareupdate options]`.

But in either case, if the signing is done with a “stable” signing identifier, changes in the binary should not trigger the need for an updated PPPC/TCC profile.

-Greg

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/CADjniupUGfOd8HmEtCHT3WyONZ9iMRa4EDvPEXKdWd8PhhMRkA%40mail.gmail.com.

[Ben Toms]

Sure, it’s worth a try.

We did see where we are manually triggering Munki with Jamf Pro, that despite the Jamf Pro binaries having a PPPCP.. we needed a separate PPPCP for python.. obvs to do with the calling process somewhere and changes will change results.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/63BC7692-C3CD-4163-AD30-88B0E3752975%40mac.com.

--

Regards,

Ben

[Nixtar]

In the past ( very early versions of Mojave ), I had noted issues with TCC inheritance.

It would be inexplicitly unreliable and further down the chain the child processes got the more likely that the TCC profile would not have the desired affect.

I have not retested this since then however.

This is why we sign the munki python binaries in our org.

Our use case for this is that we manage ARD settings via the kickstart command in a nopkg package.

Kickstart requires full disk access to enable ARD.

A wrapper binary could be a good solution, assuming the inheritance isn't an issue anymore.

[Greg Neagle]

IMHO, signing Munki’s Python and then pre-approving Full Disk Access and friends for that python is an awfully broad approval: once you do that, _anything_ using munki-python has that approval — it would not be limited to Munki itself.

I’ve used a compiled wrapper tool to allow autopkg to access network volumes (by approving that wrapper), so I know the concept is sound. Time will tell about an actual implementation.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/b0a152ab-71dc-43f5-936e-fbbe9d7e3282n%40googlegroups.com.

[Nixtar]

I agree, however we had no choice as at the time our MDM had no ability to run commands/scripts and even now the scoping is too limited for our requirements.

The wrapper locks it down to just munki operations at least, if a bad actor really wanted to go deep they could just replace the munki python library's binaries. At this stage however, they have login access to the device and local admin/root so there isn't much stopping them.

Looking at https://github.com/apple/device-management/blob/seed_iOS-16_macOS-13/mdm/profiles/com.apple.TCC.configuration-profile-policy.yaml there does not appear to be any new entries for “App Management”. :(

I'll make sure to log feedback for that as well.

[Steve Major]

This is going to put us in a similar situation of Full Disk Access as well. We have certain State Cybersecurity requirements that some apps need to be kept up-to-date and our MDM doesn't do this. Probably a similar situation to Nixar and Munki has been our saving grace. While not optimal, the other choice is the beginning of the end of Macs on our campus. I've made our common concerns to our Apple SE to pass along as feedback, but unfortunately, the past years experience with Apple is that they won't change course unless it affects the bottomline. By the time they notice, we and probably others will have removed 1000's of machines, never to return.

[precu...@gmail.com]

I am testing on a Ventura beta client now. Is there anything you wanted me to try signing and testing or anything you wanted me to report on?

[Steve Major]

I was given this by our our Apple SE: "I’m told that this issue should be resolved by the new SystemPolicyAppBundles PPPC key which is included in the macOS 13 beta 3 release.  We encourage you to test and report feedback using the AppleSeed for IT program.

https://appleseeddownload.apple.com/appleseed_for_it/macOS/macOS13/beta3_22A5295h/macOS_13_Beta_3_Release_Notes.pdf?rid=sK7zBWl2ho0Q2RLOr2DgDA"

Looks like the key will allow apps to update and delete other apps.

[Greg Neagle]

This doesn’t _resolve_ anything. This is a mechanism admins might use to grant Munki the needed permissions.

Granting Munki Full Disk Access would work similarly, and you could put that in place _before_ upgrading machines to Ventura.

But using this (“SystemPolicyAppBundles”) or Full Disk Access still will require a signed binary.

-Greg

-- 
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
--- 
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/3955f797-4d45-4312-b59a-186f9e0d9637n%40googlegroups.com.

[Greg Neagle]

Update here.

I have a compiled binary wrapper for managedsoftwareupdate written in C that can be signed with a Developer ID or just left ad-hoc signed; once it’s “stable” I bet it will essentially never change.

But I cannot test it with Ventura b3 — because I cannot get Munki to fail to make changes to application bundles.

Here’s how I’ve been testing:

1) Manually install Google Chrome 102.x (a few versions out of date)
2) Launch Google Chome (apparently some of the protections don't kick in until an app has been launched at least once)
3) Wait for Munki to attempt to update Google Chrome (can't run managedsoftwareupdate manually as it would then inherit the permissions of Terminal.app or ssh-keychain-wrapper)

In my testing under Ventura b6, stock Munki (unsigned; no PPPC/TCC profile applied) has no issues updating Google Chrome.

If I can’t replicate what’s changed in Ventura, I can really test anything designed to adapt to these changes.

-Greg

> --
> Find related discussion groups here:
> https://github.com/munki/munki/wiki/Discussion-Group
> ---
> You received this message because you are subscribed to the Google Groups "munki-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/BBA2DCBF-2D89-4D32-856F-A892A42C0A48%40mac.com.

[Greg Neagle]

"In my testing under Ventura b6” should be "In my testing under Ventura b3”

-Greg

> You received this message because you are subscribed to the Google Groups "munki-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/0B0CA4FC-4555-4B60-809F-CA3D760AB0D2%40mac.com.

[ma...@umich.edu]

If you are looking for some applications that still have issues installing updates under Ventura b4, I have 4 Apps (three of which the installers are publicly available) that still won't update *if* the applications have been launched before updating:

BlackMagic Desktop Video and Davinci Resolve Studio (both available at  https://www.blackmagicdesign.com/support/)

and Resolume Arena (https://resolume.com/download/)

Those 3 apps, you can download old versions, launch them and the later versions will not update.   (The other app is JMP Pro 16, but I don't think this is publicly available.)     Arena is the most fun to deal with as their installer launches the application upon completion.

Just FYI if you are looking for something that's still been reproducible on my end.

[Nixtar]

Hey Greg,

Any plans on adding the compiled wrapper in any of the upcoming betas for v6?

We are interested in using this for the MDM controls around the new launch daemon/agent prompts in Venture as well as doing full disk access ( for enabling ARD/SSH ) without needing to sign the python binaries. (As we currently do now)
Happy to spend some time testing with a handful of devices.


Cheers,

Nick

[Gregory Neagle]

No plans currently — because in my testing, Munki just works as it did under Monterey in every Ventura beta minus the few couple. I’ve also had zero reports of issues from anyone else.

Currently I can't find a reproducible case where "managedsoftwareupdate" is prevented from updating or removing an application, so there is no way for me to test that my compiled wrapper actually helps solve this potential issue with Ventura.

It seems too risky to add something like that unless I can demonstrate it actually addresses the issue it was intended to address.

You don’t need any signed things to manage the Login Items stuff in Ventura.

(Hint:

<key>Rules</key>
    <array>
         <dict>
               <key>RuleType</key>
               <string>LabelPrefix</string>
               <key>RuleValue</key>
               <string>com.googlecode.munki</string>
               <key>Comment</key>
               <string>Label prefix for Munki's launchd items</string>
         </dict>
    </array>

)

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/92b41445-3c6a-49b8-a17a-efeaeb0287e6n%40googlegroups.com.

[pne...@gmail.com]

Hey all,

I can't replicate the problem described here that managedsoftwareupdate can't update apps.

I've tested Munki updating the following apps: Chrome, Premiere and Davinci Resolve 17. I created 2 scenarios: macOS Upgraded from Monterey and an Erase and Install from Ventura.

Updating apps while a user was logged in and tested it updating at the loginwindow.

If you read the release notes for Ventura for this subject there are a few options. I didn't need those. In our environment Munki is installed by Erik Gomez installapplications. The installapplications package is being installed by our mdm Jamf School.

Patrick

[Gregory Neagle]

I’m sorry, I somehow missed this post back in July (oh: I was on vacation!)

I just tested this with Ventura beta 9.

I had Munki install BlackMagic Desktop Video version 12.2.2. Install completed successfully.

I launched the various apps installed by the package, as I’ve been informed the app bundle update protection does not “kick in” until an app has been launched the first time.

I then had Munki upgrade BlackMagic Desktop Video to version 12.4. The upgrade also completed successfully, with no errors reported.

So again: I don’t have a reproducible case of Munki being prevented from updating app bundles.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/7ab56130-e57f-4ccc-9483-db36d9f16972n%40googlegroups.com.

[pne...@gmail.com]

I also forgot to mention that I also started the app when it was installed by Munki for the first time before the update process.

[b...@macmule.com]

I've been discussing/harassing folks across Slack about SystemPolicyAppBundles, and have a case which Munki can hit (with thanks for Patrick and others for assisting discover).

So, first of all the item you're looking to update must:

• Be signed, notarised and have hardened runtime enabled (in some manner, struggled to recreate this with our own apps so far which have all 3).
• The app needs to have been launched at least once.

To update, you need to:

• Have a Distribution pkg (not component as munki-pkg creates), and this can either be unsigned or signed with a different identifier than the app you're looking to update.

If the above is all true, and you try to update the item from Munki, it'll fail at %97.750000.

Additionally, using the same PKG within Installer.app will prompt to allow Installer.app access to modify applications.

So far, the latest Google Chrome can trigger this.

In attempting to resolve, I granted Munki Full Disk Access via a PPPCP (we sign Munki). But this did not work.

Instead, I had to grant /usr/sbin/installer Full Disk Access (as path, with the requirement: identifier "com.apple.installer" and anchor apple and with "Validate static content" checked). This might be due to how munki is laid out on disk, vs a .app.

Now the above isn't great, but isn't the great big gaping security hole it might seem. As invoking /usr/sbin/installer via Terminal (for example), requires/prompts for SystemPolicyAppBundles enablement.

The discovery was made on beta 11, but might well have been in place prior., This was tested by myself on an Intel and a Silicon device. As well as tested by Patrick separately.

It is bizarre that this is only triggered via Distribution pkg's and not Component (other pkg types might trigger this too).