Latest managedsoftwareupdate failing on i7 macOS 13.1; appears to forget URL of munki server, reverting to http://munki/repo

[Frank Seesink]

I've noticed something odd with one of my systems.  It's a 2018 iMac i7 running macOS 13.1 with Munki tools v6.0.1.4523 installed.

Specifically, when running the Managed Software Update client, I notice it coming back with no updates, when I know for sure that updates exist (since I just updated the Munki repo).  And when I run Terminal and check, here's what I see:

____

% sudo managedsoftwareupdate
Password:
Managed Software Update Tool
Copyright 2010-2022 The Munki Project
https://github.com/munki/munki

Starting...
WARNING: Client is configured to use the default repo, which is insecure. Client could be trivially compromised when off your organization's network. Consider using a non-default URL, and preferably an https:// URL.
    Looking for local Munki repo server...
    Auto-detected Munki repo at http://munki/repo
Checking for available updates...
ERROR: Could not retrieve managed install primary manifest.: (-1003, 'A server with the specified hostname could not be found.')
Finishing...
    Performing postflight tasks...
Done.
____

Mind you, I have my Munki site at an HTTPS URL, and if I check

/Library/Preferences/ManagedInstalls.plist

it shows properly there.

Now, if I reboot the iMac, it then seems to update once again.  But only for a short time.  Eventually this occurs again.

I never experienced this before on any system I manage.  So wondering if anyone else has experienced this or has advice on where to look/how to fix this.  It's been a real head scratcher for me.  And no, there is nothing obviously different about this particular Mac.  I tend to setup my systems pretty uniformly, as I move between different systems a bit.

Thanks in advance for any/all help with this.

P.S.  Originally posted this in the -dev group, so apologies for using the wrong group.  Also, next time I see this issue, instead of rebooting I will try disabling/re-enabling the /Library/LaunchDaemon processes to see if that sorts it.

[Gregory Neagle]

I’d start with `sudo managedsoftwareupdate —show-config` to get a better sense of how Munki is configured, since macOS preferences can be stored several places and not only in /Library/Preferences/ManagedInstalls.plist.

See https://github.com/munki/munki/wiki/Preferences

You also say: "if I reboot the iMac, it then seems to update once again”. I’d be looking for specific confirmation of that. 

-Greg

--
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/78ea5a50-0e03-42d0-88d3-502188705b10n%40googlegroups.com.

[Frank Seesink]

Sorry, I should have been more clear.  After a reboot, I run the same `sudo managedsoftwareupdate` command and it pulls down the manifests and properly updates.  Is that specific confirmation enough?

[Gregory Neagle]

I’d want to see the output of `sudo managedsoftwareupdate —show-config` in both states. If it’s different, you need to figure out what mechanism is changing the config.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/7cd01f97-c046-4af0-8907-d7881f52ed6bn%40googlegroups.com.

[Frank Seesink]

Hey Greg,

Ok, it happened again.  This time I was able to get more info.  (Man I wish Google Groups had better formatting options.  But I'll do my best here.)

____________________________________________________________

# OUTPUT WHEN UPDATE NOT WORKING

% sudo managedsoftwareupdate              

Managed Software Update Tool
Copyright 2010-2022 The Munki Project
https://github.com/munki/munki

Starting...
WARNING: Client is configured to use the default repo, which is insecure. Client could be trivially compromised when off your organization's network. Consider using a non-default URL, and preferably an https:// URL.
    Looking for local Munki repo server...
    Auto-detected Munki repo at http://munki/repo
Checking for available updates...
ERROR: Could not retrieve managed install primary manifest.: (-1003, 'A server with the specified hostname could not be found.')
Finishing...
    Performing postflight tasks...
Done.

% sudo managedsoftwareupdate --show-config
Current Munki configuration:
                   AdditionalHttpHeaders:  None [not set]
        AggressiveUpdateNotificationDays:    14 [/Library/Preferences/ManagedInstalls.plist]
                AppleSoftwareUpdatesOnly: False [/Library/Preferences/ManagedInstalls.plist]
                              CatalogURL:  None [not set]
                   ClientCertificatePath:  None [not set]
                        ClientIdentifier:    '' [/Library/Preferences/ManagedInstalls.plist]
                           ClientKeyPath:  None [not set]
                       ClientResourceURL:  None [not set]
                 ClientResourcesFilename:  None [not set]
                DaysBetweenNotifications:     1 [/Library/Preferences/ManagedInstalls.plist]
                   EmulateProfileSupport: False [/Library/Preferences/ManagedInstalls.plist]
                     FollowHTTPRedirects: 'none' [/Library/Preferences/ManagedInstalls.plist]
                                 HelpURL:  None [not set]
                                 IconURL:  None [not set]
                     IgnoreSystemProxies: False [/Library/Preferences/ManagedInstalls.plist]
             InstallAppleSoftwareUpdates: False [/Library/Preferences/ManagedInstalls.plist]
                   InstallRequiresLogout: False [/Library/Preferences/ManagedInstalls.plist]
                       LocalOnlyManifest:  None [not set]
                                 LogFile: '/Library/Managed Installs/Logs/ManagedSoftwareUpdate.log' [/Library/Preferences/ManagedInstalls.plist]
                             LogToSyslog: False [/Library/Preferences/ManagedInstalls.plist]
                            LoggingLevel:     1 [/Library/Preferences/ManagedInstalls.plist]
                       ManagedInstallDir: '/Library/Managed Installs' [/Library/Preferences/ManagedInstalls.plist]
                             ManifestURL:  None [not set]
                              PackageURL:  None [not set]
                 PackageVerificationMode: 'hash' [/Library/Preferences/ManagedInstalls.plist]
                     PerformAuthRestarts: False [/Library/Preferences/ManagedInstalls.plist]
                         RecoveryKeyFile:  None [not set]
 ShowOptionalInstallsForHigherOSVersions: False [/Library/Preferences/ManagedInstalls.plist]
               SoftwareRepoCACertificate:  None [not set]
                      SoftwareRepoCAPath:  None [not set]
                         SoftwareRepoURL: 'http://munki/repo' [/Library/Preferences/ManagedInstalls.plist]
                 SoftwareUpdateServerURL:  None [not set]
                     SuppressAutoInstall: False [/Library/Preferences/ManagedInstalls.plist]
              SuppressLoginwindowInstall: False [/Library/Preferences/ManagedInstalls.plist]
             SuppressStopButtonOnInstall: False [/Library/Preferences/ManagedInstalls.plist]
                SuppressUserNotification: False [/Library/Preferences/ManagedInstalls.plist]
                  UnattendedAppleUpdates: False [/Library/Preferences/ManagedInstalls.plist]
                    UseClientCertificate: False [/Library/Preferences/ManagedInstalls.plist]
UseClientCertificateCNAsClientIdentifier: False [/Library/Preferences/ManagedInstalls.plist]
               UseNotificationCenterDays:     3 [/Library/Preferences/ManagedInstalls.plist]

% ps -ef | grep managed
    0 35988     1   0  4:10PM ??         0:00.08 /usr/local/munki/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python /usr/local/munki/supervisor --delayrandom 3600 --timeout 43200 -- /usr/local/munki/managedsoftwareupdate --auto
  501 37038 37006   0  4:37PM ttys001    0:00.00 grep managed


% cd /Library/LaunchDaemons
% cat com.googlecode.munki.managedsoftwareupdate-check.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.googlecode.munki.managedsoftwareupdate-check</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/munki/supervisor</string>
        <string>--delayrandom</string>
        <string>3600</string>
        <string>--timeout</string>
        <string>43200</string>
        <string>--</string>
        <string>/usr/local/munki/managedsoftwareupdate</string>
        <string>--auto</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Minute</key>
        <integer>10</integer>
    </dict>
</dict>
</plist>

% cat com.googlecode.munki.managedsoftwareupdate-manualcheck.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>KeepAlive</key>
    <dict>
        <key>PathState</key>
        <dict>
            <key>/private/tmp/.com.googlecode.munki.updatecheck.launchd</key>
            <true/>
        </dict>
    </dict>
    <key>Label</key>
    <string>com.googlecode.munki.managedsoftwareupdate-manualcheck</string>
    <key>OnDemand</key>
    <true/>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/local/munki/supervisor</string>
        <string>--timeout</string>
        <string>43200</string>
        <string>--</string>
        <string>/usr/local/munki/managedsoftwareupdate</string>
        <string>--manualcheck</string>
    </array>
</dict>
</plist>

____________________________________________________________

After this, I tried various things.  I killed the one process above, thinking maybe it was in a zombie state.  But then I couldn't get it launched again.  Tried using sudo launchctl commands to stop/start, unload/load, but in the end had to reboot the Mac.  I then immediately did the following, which show the config right after reboot, as well as the fact it was working once again.  (NOTE:  Anywhere you see <... ...> I have sanitized the output but noted what was there):

____________________________________________________________

# OUTPUT AFTER REBOOT

% sudo managedsoftwareupdate --show-config
Password:
Current Munki configuration:
                   AdditionalHttpHeaders: (
    "Authorization: Basic <...hash...>"
) [/Library/Preferences/ManagedInstalls.plist]
        AggressiveUpdateNotificationDays:    14 [/Library/Preferences/ManagedInstalls.plist]
                AppleSoftwareUpdatesOnly: False [/Library/Preferences/ManagedInstalls.plist]
                              CatalogURL:  None [not set]
                   ClientCertificatePath:  None [not set]
                        ClientIdentifier: '<...actual name of Mac...>' [/Library/Preferences/ManagedInstalls.plist]
                           ClientKeyPath:  None [not set]
                       ClientResourceURL:  None [not set]
                 ClientResourcesFilename:  None [not set]
                DaysBetweenNotifications:     1 [/Library/Preferences/ManagedInstalls.plist]
                   EmulateProfileSupport: False [/Library/Preferences/ManagedInstalls.plist]
                     FollowHTTPRedirects: 'none' [/Library/Preferences/ManagedInstalls.plist]
                                 HelpURL:  None [not set]
                                 IconURL:  None [not set]
                     IgnoreSystemProxies: False [/Library/Preferences/ManagedInstalls.plist]
             InstallAppleSoftwareUpdates: False [/Library/Preferences/ManagedInstalls.plist]
                   InstallRequiresLogout: False [/Library/Preferences/ManagedInstalls.plist]
                       LocalOnlyManifest:  None [not set]
                                 LogFile: '/Library/Managed Installs/Logs/ManagedSoftwareUpdate.log' [/Library/Preferences/ManagedInstalls.plist]
                             LogToSyslog: False [/Library/Preferences/ManagedInstalls.plist]
                            LoggingLevel:     1 [/Library/Preferences/ManagedInstalls.plist]
                       ManagedInstallDir: '/Library/Managed Installs' [/Library/Preferences/ManagedInstalls.plist]
                             ManifestURL:  None [not set]
                              PackageURL:  None [not set]
                 PackageVerificationMode: 'hash' [/Library/Preferences/ManagedInstalls.plist]
                     PerformAuthRestarts: False [/Library/Preferences/ManagedInstalls.plist]
                         RecoveryKeyFile:  None [not set]
 ShowOptionalInstallsForHigherOSVersions: False [/Library/Preferences/ManagedInstalls.plist]
               SoftwareRepoCACertificate:  None [not set]
                      SoftwareRepoCAPath:  None [not set]
                         SoftwareRepoURL: 'https://<...actual URL to Munki server...>' [/Library/Preferences/ManagedInstalls.plist]
                 SoftwareUpdateServerURL:  None [not set]
                     SuppressAutoInstall: False [/Library/Preferences/ManagedInstalls.plist]
              SuppressLoginwindowInstall: False [/Library/Preferences/ManagedInstalls.plist]
             SuppressStopButtonOnInstall: False [/Library/Preferences/ManagedInstalls.plist]
                SuppressUserNotification: False [/Library/Preferences/ManagedInstalls.plist]
                  UnattendedAppleUpdates: False [/Library/Preferences/ManagedInstalls.plist]
                    UseClientCertificate: False [/Library/Preferences/ManagedInstalls.plist]
UseClientCertificateCNAsClientIdentifier: False [/Library/Preferences/ManagedInstalls.plist]
               UseNotificationCenterDays:     3 [/Library/Preferences/ManagedInstalls.plist]

% sudo managedsoftwareupdate              

Managed Software Update Tool
Copyright 2010-2022 The Munki Project
https://github.com/munki/munki

Starting...

Checking for available updates...
    Preventing idle sleep
    Retrieving catalog "testing"...
ERROR: Could not retrieve catalog testing from server: HTTP result 404: not found
    Retrieving catalog "production"...
    0..20..40..60..80..100
    Retrieving catalog "testing"...
ERROR: Could not retrieve catalog testing from server: HTTP result 404: not found
    Retrieving catalog "testing"...
ERROR: Could not retrieve catalog testing from server: HTTP result 404: not found
<...>
    Getting list of available icons...
    0..20..40..60..80..100
    Getting client resources...
    Getting client resources...
    Allowing idle sleep
   
    The following items will be installed or upgraded:
<...>

Run managedsoftwareupdate --installonly to install the downloaded updates.

Finishing...
    Performing postflight tasks...
Done.

____________________________________________________________

In the outputs above, I bolded 2 lines in the --show-config output to show how when things go brain dead, it forgets what the client identifier is, as well as the repo URL.  Note also how when things are dead, there's a lot less known in the initial output, whereas when it's working, you see the usual HTTP authorization password hash, etc.

Now as to "If it’s different, you need to figure out what mechanism is changing the config", that's just it.  The config (as in what's stored in the config files) ISN'T changing.  A simple reboot, without doing a thing, restores things.  It's as if the process spawned by /Library/LaunchDaemons/com.googlecode.munki.managedsoftwareupdate-check.plist  is forgetting the settings after a period of time.  But I couldn't begin to explain why.  The rest of the system is working fine.  Memory bleed?  Bug?  I was hoping you might have some insight on where to look.

Since Munki now uses its own Python interpreter, is it safe to assume that everything is properly sandboxed via virtualenv/etc. so as not to be influenced by any other Python interpreters installed on the host system?  I mean, I write Python code and have Python 3.11.1 installed currently (using the Python.org macOS installer) on all of my systems, meaning it's under /Library/Frameworks/Python.framework/Versions/Current/.  And as I said, I tend to keep them pretty uniform, so if I add something via pip to one, odds are I've added it to all of them.  So I'm scratching my head here why one box behaves differently than the others.

I mean, even if I were doing pip module installs to my instance of Python, and had different pip modules on different systems, I would think Munki wouldn't be affected, unless somehow it's not looking at its own interpreter/setup but rather there's some kind of leak/etc. where it's using the system's setup instead.  But where/how that might be happening is outside my scope and more in your wheelhouse.

The most unique things I'm running on the Mac having problems are Munki-related.  That is, this Mac is the one that runs AutoPkgr to pull down updates to begin with, after which I use MunkiAdmin to check/move packages from test to production.  And then I run a small AppleScript I created that uses rsync to copy any repo changes from that Mac up to the actual server that acts as the Munki repo server.  And this setup has worked well for years now.

Only recently (as in at some point in the last few months after v6.0) have I noticed this issue.  It began when my Macs were still on macOS 12.x "Monterey", and it continues now, even though I'm up to date with macOS 13.1 "Ventura" as I type this on all my systems.  Well at least those that can be updated.  I have one 2014 MBP that's stuck at macOS 11 "Big Sur".  But even that one works just fine doing updates.

Anyway, if there's anything else I can do to test this or try to nail this down, let me know.  But this feels like an issue with Munki.  I just can't figure out why.

[Frank Seesink]

Greg,

Ok, thought "ARE these 2 Macs--the laptop I'm writing this on which does NOT appear to have any issues--and the Mac on which this problem keeps happening truly the same?"  So began looking.

Digging into where the Munki version of Python gets installed, I ran the following command on each system:

/usr/local/munki/Python.framework/Versions/3.9/bin/pip3 list > filename.txt

where filename.txt was the name of each Mac.  I then ran a diff on them.  And here I found something interesting.  While most of the packages are exactly the same, there ARE a few differences.  On the problematic Mac I see

Package                                           Version
------------------------------------------------- -------

...

pip                                               22.3.1    (on working Mac it's 22.3)

...

wheel                                             0.38.4    (on working Mac it's 0.37.1)

That's it.  Now this made me wonder.  So I did the same command over on my laptop running "Big Sur" (which also works fine), and I compared its output to that of the other working Mac.  They showed the exact same module versions.  So what I show above is the only thing which is unique.

Now frankly, I couldn't tell you why those are different.  I don't use that instance.  In fact, until now I didn't really know where it was buried.  As I said, I use the Python.org packages, which install under /Library/Frameworks/Python.framework/Versions/.  And I have my PATH in ZSH set to point at /Library/Frameworks/Python.framework/Versions/Current/bin as the first entry, meaning it will use my setup for anything I do, which is how it should be.

[Greg Neagle]

With the evidence so far at hand, the most likely scenario here is that you have two scripts:

The first script sets the values of “AdditionalHttpHeaders”, “ClientIdentifier”, and “SoftwareRepoURL” in /Library/Preferences/ManagedInstalls.plist.

The second script either clears/deletes those values, or removes  /Library/Preferences/ManagedInstalls.plist entirely, causing the values to revert to their defaults.

The first script appears to run at boot. It might run at other times as well, but almost certainly runs at boot.  If the Munki configuration is always wrong after it breaks until the machine is rebooted, then it’s most likely the script only runs at boot. There are many mechanisms for running scripts at boot; one is Outset. (I’d link to that, but presumably if you are using Outset, you’d know, and if you aren’t, it’s not relevant.)

The second script runs later. It might run on a schedule, or might be triggered by something else. You should experiment to see if you can find a pattern: after reboot, is the first Munki run successful, and then it’s broken after that? Or are multiple Munki runs successful, but after 5/10/15 minutes, they break? Or they break after a certain minutes past the hour?  Any of these patterns would help narrow down what might be running/triggering this second script.

I can’t be of much more help finding these scripts — that’s going to be a task only you can do at this point.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/397e3ac8-bc53-412b-b69a-90457dd3a5e7n%40googlegroups.com.

[Frank Seesink]

Hey Greg,

It happened again, so that circled me back here. Just FYI:  The system is still macOS 13.1, but the version of managedsoftwareupdate/etc. is now `6.1.0.4536` (latest) with the last update to Munki.

If I run `sudo managedsoftwareupdate --show-config`, I see once again that the ClientIdentifer has been reset to `''` and `SoftwareRepoURL: 'http://munki/repo' [/Library/Preferences/ManagedInstalls.plist]`, which all same as before.

I'm not using Outset, so that's out.  (Had to Google to see what that was.)

And looking now, ` /Library/Preferences/ManagedInstalls.plist` still exists and contains the correct info in it such as the RepoURL. (I do notice that file's timestamp shows today at 5:52AM.  And AutoPkgr is set to run at 6:AM each day.  Correlation isn't causation, but figured to mention that just in case.)

As far as discernible patterns go, I haven't been able to figure one out yet.

Yes, to date after reboot it runs successfully.

No, it's not the case that it runs once and then is broken.

No, it's not the case that "multiple Munki runs successful, but after 5/10/15 minutes, they break".  In fact, this one I "stress tested" by intentionally running managedsoftwareupdate repeatedly on the machine.

And so far, no, I can't say whether "they break after a certain minutes past the hour", mostly because I'm not in front of this machine all day long.  So I only know when it's not working when I run the Managed Software Center after having specifically updated the Munki repo, only to find it claims no updates available, which I know not to be true.  So I can only speak to the times when I observe this, which is not the same as when it stops working.  But I suspect it's more on the order of "days" before it starts failing.

Not sure what this second script you speak of might be that makes managedsoftwareupdate suddenly go brain dead, because the output of `--show-config` indicates it's pulling the information from the right .plist file, when it clearly is not.  What it displays vs. what I see in the contents of that .plist file don't match, with the .plist file showing the right ClientIdentifier and Munki URL.

I'll keep looking, but so far, I'm not getting how a second script is making this managedsoftwareupdate software do this.  Keep in mind nothing has really changed on this system other than basic software updates to existing software and OS.  And it's not doing anything magical nor very different from my other Macs which do not exhibit this issue.  Even the one key piece of software that is different (notably that it's the Mac running AutokPgr to pull down updates) hasn't been updated since last May.  And this issue manifested much more recently than that.

I would say it's due to macOS 13.x "Ventura", but the MBP I'm typing this on is running the same version and has no such problem.  And both systems were upgraded about the same time.  Yet only the one is suffering this issue.  So I'm scratching my head trying to figure this out.

[Mike Solin]

Hi Frank -

Could I suggest checking the possible places a script could be running that would reset your Munki preferences?

Check these locations for launchd items that can run at boot or login:

/Library/LaunchAgents

/Library/LaunchDaemons

/Users/yourusername/LaunchAgents

You can also check cron:

crontab -l

sudo crontab -l

Finally, open System Settings (since you mentioned that you're running macOS Ventura) and look at General > Login Items to see if you have anything listed in the upper or lower sections that could be custom.

Did you inherit this Mac from someone else? If so, you might want to consider replacing the hardware and building it up from scratch.

Mike

--

https://mikesolin.com

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/2e4ab7c9-55e2-404c-a1a1-8d0cd989aa64n%40googlegroups.com.

[Gregory Neagle]

"I'll keep looking, but so far, I'm not getting how a second script is making this managedsoftwareupdate software do this.”

Quite simply.

The second script or package install or some process either clears/deletes the values of the values of “AdditionalHttpHeaders”, “ClientIdentifier”, and “SoftwareRepoURL” in /Library/Preferences/ManagedInstalls.plist, or removes /Library/Preferences/ManagedInstalls.plist entirely, causing the values to revert to their defaults.

"And so far, no, I can't say whether "they break after a certain minutes past the hour", mostly because I'm not in front of this machine all day long. “

This is where looking at the logs in /Library/Managed\ Installs/Logs would help. You’d want to look at the ManagedSoftwareUpdate.log* files. After a reboot, you should see a successful run. Keep looking at subsequent runs. Eventually one fails. Clue!

I am quite confident this is not a _Munki_ issue. This is an issue with some additional script or process or package install that is modifying /Library/Preferences/ManagedInstalls.plist either purposely, or accidentally.

You could, of course, side-step the issue entirely by managing Munki’s preferences via configuration profile. Whatever is modifying /Library/Preferences/ManagedInstalls.plist might continue to do that, but preferences defined in config profile would have higher precedence.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/2e4ab7c9-55e2-404c-a1a1-8d0cd989aa64n%40googlegroups.com.

[Frank Seesink]

Ok, it happened again.  And I guess I am not being very clear here.  With it being the case right now that doing

sudo managedsoftwareupdate --show-config

shows the values reset, to be perfectly clear, the file /Library/Preferences/ManagedInstalls.plist shows the CORRECT values.  That is, that file has NOT been reset.  It never has.  It is ONLY the software which appears to have gone brain dead and reset itself.  So not sure why you keep harping on this .plist file being modified/etc.  It's not.  The file is intact.  I don't touch it.  If rebooting restores things, and that file was actually modified, that wouldn't work.

Now, as for

/Library/LaunchAgents

/Library/LaunchDaemons

/Users/yourusername/LaunchAgents

there is nothing new, odd, or out of place there.  And how that could be impacting things when the .plist file is still correct eludes me.

As for

crontab -l

sudo crontab -l

both show no cron jobs.

As for "System Settings (since you mentioned that you're running macOS Ventura) and look at General > Login Items", the only oddity in my mind is that Munki's 'supervisor' process is listed 4x in the lower section.  Unfortunaly, Ventura's Settings don't give you any useful information about this other than showing you the actual file referenced if you click the "(i)" icon, which in this case for all 4 instances point to /usr/local/munki/supervisor.

And no, this is NOT an inherited Mac.  This is the same Mac I have used since 2019 which I built from the ground up and have used from then 'til now.  I manage my own boxes, so there's nothing on here I'm not responsible for.  No one else manages them nor uses them.  And AutoPkgr, MunkiAdmin, and Munki have been installed on this system all that time and working just fine.  As I said, this is the box that checks for updates of packages.  And it's been working fine for YEARS now.

Also, not sure how to drill this point home, but the config file is INTACT.  So HOW, exactly, is managedsoftwareupdate showing default settings?  And again, this only started happening in the past few months.

Finally, THANK YOU!  That bit about /Library/Managed\ Installs/Logs/ManagedSoftwareUpdate.log let me find the exact time when this started (or at least the first time when managedsoftwareupdate went from knowing the right values to going brain dead:

____

Jan 31 2023 07:00:29 -0500 ### Starting managedsoftwareupdate run: auto ###
Jan 31 2023 07:00:29 -0500 Starting...
Jan 31 2023 07:00:31 -0500 ### Beginning managed software check ###
Jan 31 2023 07:00:31 -0500 Checking for available updates...

...

<successfully connected>

...

Jan 31 2023 07:30:46 -0500 ### Starting managedsoftwareupdate run: auto ###
Jan 31 2023 07:30:46 -0500 Starting...
Jan 31 2023 07:30:46 -0500 WARNING: Client is configured to use the default repo, which is insecure. Client could be trivially compromised when off your organization's network. Consider using a non-default URL, and preferably an https:// URL.
Jan 31 2023 07:30:46 -0500     Looking for local Munki repo server...
Jan 31 2023 07:30:46 -0500     Auto-detected Munki repo at http://munki/repo
Jan 31 2023 07:30:47 -0500 ### Beginning managed software check ###
____

Now using that info, went checking the console logs for any indication of what might have happened.  Unfortunately, nothing jumps out.  There's only 3 lines in the Mac Analytics Data between the last successful attempt and first failed attempt, and they're all related to Apple's AddressBook and occur at 7:08-:09.  So unless you're saying Apple's own system is borking the managedsoftwareupdate somehow, I'm at a loss.

[Ben Toms]

Are you setting the URL via MDM?

Perhaps that profile is being removed at times?

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/36a3aaba-3eda-47c4-a747-4d334ec8916dn%40googlegroups.com.

--

Regards,

Ben

[Gregory Neagle]

You have shown evidence that the /Library/Preferences/ManagedInstalls.plist is indeed being changed:

We have to go with the evidence we have. The evidence we have is something is changing the contents of /Library/Preferences/ManagedInstalls.plist, and that evidence was provided by you.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/36a3aaba-3eda-47c4-a747-4d334ec8916dn%40googlegroups.com.

[Mike Solin]

With CFPreferences caching, the file on disk might not match what you receive via sudo managedsoftwareupdate --show-config. That's expected.

Mike

--

https://mikesolin.com

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/F8B7E669-B21B-42AE-9F36-9BF857FEE10D%40mac.com.

[Frank Seesink]

"Are you setting the URL via MDM?"

No.

"You have shown evidence that the /Library/Preferences/ManagedInstalls.plist is indeed being changed:

...

We have to go with the evidence we have. The evidence we have is something is changing the contents of /Library/Preferences/ManagedInstalls.plist, and that evidence was provided by you."

Umm, no.  Incorrect.  The evidence provided by me simply shows that managedsoftwareupdate is reporting in its output that it has default values set.  That is NOT the same thing.

And what I have repeatedly said is that the .plist file is INTACT, showing the correct values, even when managedsoftwareupdate is reporting default values.  Now one would have to know or go digging into the source of that program to see how it comes to present those default values.  But the .plist file CORRECTLY shows things like the client ID and repo URL, while the program seems to think otherwise.  And THAT, in my mind, points at the program.

"With CFPreferences caching, the file on disk might not match what you receive via sudo managedsoftwareupdate --show-config. That's expected."

Now THAT--the idea of caching somehow being involved and getting mucked up in memory--I COULD see.  But how to fix that if it's the case?  And WHY does it appear in sync after a reboot but at some point later changes?

[Mike Solin]

Since 10.9 Mavericks, if you do a "defaults write something something", that's cached in memory by cfprefsd. Some time later, it's written to disk. You can typically force it to be written to disk by restarting the cfprefsd process. I'd guess rebooting would have the same effect.

It sounds like you have something doing "defaults delete" for some/all of the keys in ManagedInstalls.plist, sometime after your Mac is booted. That's why I suggested digging into what could be running in the background, if the process that's doing it isn't an item in your Munki manifest.

As Greg suggested, you can sidestep this entirely by managing Munki's settings via a configuration profile, deployed by your MDM. I'd still suggest getting to the bottom of whatever is going on, but a profile is a great solution here.

Mike

--

https://mikesolin.com

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/34d0f519-5538-44cb-9f38-33a36f9e168bn%40googlegroups.com.

[Greg Neagle]

"Now one would have to know or go digging into the source of that program to see how it comes to present those default values. “

Hmm. Who here might know that? :-) And of course, the code is completely available for anyone to read.

https://github.com/munki/munki/blob/main/code/client/munkilib/prefs.py#L230-L308

I am open to alternate explanations and theories, but with the evidence we’ve seen so far, I can’t come up with a better theory.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/34d0f519-5538-44cb-9f38-33a36f9e168bn%40googlegroups.com.

[Frank Seesink]

"Since 10.9 Mavericks, if you do a "defaults write something something", that's cached in memory by cfprefsd. Some time later, it's written to disk. You can typically force it to be written to disk by restarting the cfprefsd process. I'd guess rebooting would have the same effect.

It sounds like you have something doing "defaults delete" for some/all of the keys in ManagedInstalls.plist, sometime after your Mac is booted. That's why I suggested digging into what could be running in the background, if the process that's doing it isn't an item in your Munki manifest."

Again, thank you, Mike.  That makes a lot of sense. But I have to ask what may well be a dumb question.  For `something doing "defaults delete" for some/all of the keys in ManagedInstalls.plist`, would that not require something specifically targeting the keys in that one particular file (or its cached copy)?  And what, if anything, have you seen doing this in the past?  That seems... well frankly, odd to me.  Or am I missing something?

And Greg, I am sincerely looking for help here, though the tone of your writing makes me think you feel otherwise.  While I know you're the key developer, to be blunt (but not intending to be harsh), your writing style comes across a bit arrogant.  I'm trying to understand why something with this software isn't working as it has for years now.  I love using Munki, and I really do appreciate all it has done for me, and for all the work you and others have put into it.  If I didn't care, I wouldn't be here.  I figured maybe I'd stumbled on a weird bug.  But so far Mike's explanations have been far more enlightening/helpful in where to focus.  I get you live and breath all this.  It's your baby.  But don't assume everyone who uses this software does (live and breath it).

To me it's odd that a program which reads a config file shows one thing while the config file itself shows another.  And the idea that something is coming along and clearing out specific keys/values from a cached/in-memory copy of that config isn't immediately obvious.  However, with that bit of info on cfprefsd, I went looking and found that something I will try next time this happens is to run

defaults read /Library/Preferences/ManagedInstalls

and see whether THAT appears differently than the actual contents of the file itself.  If they do not match, and specifically if the defaults read reveals that the keys in question have been reset, then the real question is "What could possibly be targeting these settings?"  It seems... well odd, at least to me.

[Greg Neagle]

"To me it's odd that a program which reads a config file shows one thing while the config file itself shows another."

While that might be the case, you haven’t shown that to us. Again, we can only work with the evidence we actually _have_.

Until we have additional evidence, I’m sticking by my current theory.

Remember: you are looking for help, but we only have access to the information you give us. And it might be incomplete or inaccurate - we don’t know!

And we all are doing this investigation and thinking and theorizing for you, a person we don’t actually know, for free. We’re doing the best we can, and what we are doing is worth every single penny you are paying for it! :-)

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/5071e220-490a-4035-bc33-cf405eb04d03n%40googlegroups.com.

[Greg Neagle]

When you set up a new machine: how are you installing Munki?

How are you setting Munki’s configuration?

Has that configuration ever changed? If so, how did you change it?

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/5071e220-490a-4035-bc33-cf405eb04d03n%40googlegroups.com.

[Greg Neagle]

"To me it's odd that a program which reads a config file shows one thing while the config file itself shows another.”

To be precise, Munki does not directly read /Library/Preferences/ManagedInstalls.plist.

It uses Apple’s CFPreferences API to ask for the values for certain keys in the “ManagedInstalls” preferences domain.

Apple’s CFPreferences/NSUserDefaults/defaults system can read those values from many possible places in the filesystem, and also from MCX records and installed configuration profiles.

See https://github.com/munki/munki/wiki/Preferences for a taste of the possibilities here.

managedsoftwareupdate —show-config attempts to help the poor admin here by showing the values that managedsoftwareupdate is working with, and where it appears to be getting those values from. It’s possible there are poorly-understood edge cases here that might cause the code to mis-identify exactly where the effective value is coming from, but I’ve seen no reports that would support that.

In any case, the values are definitely in the preferences system, and something is changing them over time.

-Greg

[Frank Seesink]

"When you set up a new machine: how are you installing Munki?"

By simply running the MunkiTools .pkg on each system.

"How are you setting Munki’s configuration?"

I then simply run some commands I've had documented for years now (so had to go look them up, as I haven't setup a new machine in a bit) which (sanitized, of course) include

sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL "https://<urltoserver>"
sudo defaults write /Library/Preferences/ManagedInstalls AdditionalHttpHeaders -array "Authorization: <***>"
sudo defaults write /Library/Preferences/ManagedInstalls ClientIdentifier "<name of client>”
sudo /usr/local/munki/managedsoftwareupdate

Mind you, this is a pretty small setup.  I use it to manage about 1/2 dozen Macs of my own, along with those of family members spread across various states and my mother-in-law in South America.

"Has that configuration ever changed? If so, how did you change it?"

When I first built Munki back ~2015-2016 (I want to say about a year after your presentation at PSU, a video I found online back then), I had gone all in creating client certs for each device. But that got to be a bit of a headache, so eventually went with a simpler setup involved just HTTPS and a single basic auth for all clients.  That's been in place since ~2018-2019 I think.

Anyway, haven't really mucked with this in years.  Last time I tweaked anything was when my mother-in-law got a new Mac Mini, so added Munki to it to quickly deploy apps.  But that was like a year and a half ago I think.  But it's just this one Mac, which happens to be the one that also runs AutoPkgr to pull down updates, that' exhibiting this behavior.

__________

Ok, thought maybe things would be fixed by updating to macOS 13.2.  But no such luck.  A few days on and it happened again.  So checked, and here's what I found:

__________

 % sudo managedsoftwareupdate --show-config
Password:
Current Munki configuration:

# Checking what's in the cache, I see

% defaults read /Library/Preferences/ManagedInstalls
2023-02-04 13:01:04.553 defaults[42448:1320534]
Domain /Library/Preferences/ManagedInstalls does not exist

# However, if I actually check the file itself, I find

% ls -al /Library/Preferences/Managed*
-rw-r--r--  1 root  wheel  1183 Feb  4 05:31 /Library/Preferences/ManagedInstalls.plist

 % file /Library/Preferences/ManagedInstalls.plist
/Library/Preferences/ManagedInstalls.plist: Apple binary property list

__________

As you can see, it looks like the cached copy is gone.  And the preference file's timestamp is updated to this morning, but it's NOT empty.  And if I use Finder to drill down into /Library/Preferences/, select the file and hit the spacebar to preview, I can see that the client identifier the repo URL, etc., are all still intact.  So the config file is intact.

So if I understand this cfprefsd bit, it appears that something is either wiping out the entire /Library/Preferences/ManagedInstalls domain (as opposed to just changing the values of a few keys within in), or at least the OS thinks so.  And to be clear, if I do a simple defaults read by itself, I can see that all the other config data is still in there (e.g., Zoom config, etc., for apps on that system).

But what would target that one domain in the cache?  I mean, have you all ever seen anything like this before?  Mind you, nothing else on that iMac is having issues.  Everything is running just fine.  And I don't have this happening on any of the other Macs.  And again, the MBP I'm sitting on is setup nearly identical to the iMac having trouble.  I upgraded the MBP as well to macOS 13.2 and so far, it behaves as it always has, never having this issue.  Just that one iMac is doing this.  It just seems odd that something would be this oddly specific.

I can't find anything that's running at/near the timeframe which might somehow explain this.  In fact, around the timestamp update of the preference file (~5:30AM), the only thing that happens is that AutoPkgr is set to run daily @6:AM.  But that's half an hour AFTER this timestamp.  Looking at my MBP (where things have always been working fine), I see that this preference file timestamp matches when I just ran Managed Software Center.  So I'm guessing that this pref file's timestamp is updated every time managedsoftwareupdate runs, correct?  (Or maybe I should say SUCCESSFULLY runs?)  However, when I run Managed Software Update or do sudo managedsoftwareupdate on the affected Mac, that file's timestamp never changes.  It stays right at 5:31AM.

Anyway, just trying to provide whatever data points I can here.

[Alan]

But what would target that one domain in the cache?  I mean, have you all ever seen anything like this before?  Mind you, nothing else on that iMac is having issues.  Everything is running just fine.  And I don't have this happening on any of the other Macs.

I haven't seen this before, but the fact that it's happening on only this one Mac (and not the other Macs you're managing with Munki) and the fact that nobody else seems to be experiencing this problem indicates that this isn't Munki having an issue. There's something on that particular Mac that is deleting the Munki preferences (and thus resetting SoftwareRepoURL back to its default setting).

You'll have to do your own audit of launch daemons on your system or anything else automated that may be accidentally scripting something that deletes that pref.

--
You received this message because you are subscribed to the Google Groups "munki-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/a80d3404-1c4b-420e-9111-a7448cef4aacn%40googlegroups.com.

[Frank Seesink]

The only thing I can find that is unique to this one particular iMac is that it has AutoPkg/AutoPkgr installed.  And digging into that, I see that AutoPkg did have an update back in early December, this after noting

% ls -al /Library/LaunchDaemons
total 192
drwxr-xr-x  28 root  wheel   896 Feb  1 15:04 .
drwxr-xr-x  72 root  wheel  2304 Feb  1 14:04 ..

...

-rw-r--r--   1 root  wheel   615 Dec  7 11:45 com.github.autopkg.autopkginstalld.plist

-rw-r--r--   1 root  wheel   599 Dec  7 11:45 com.github.autopkg.autopkgserver.plist

...

and then checking here:  https://github.com/autopkg/autopkg/releases

That does also seem to fit the timeframe for when this began.  (I can't pinpoint it as I only started realizing this issue after it happened multiple times, with reboots involved, etc.  But it "feels" right.)

So maybe I'm focusing on the wrong end of things here.  And noting that both projects are related, not to mention have the same core developer, that at least would explain the specificity going on here.  Not that this is the right place to ask, but does AutoPkg in any way make reference to this same domain/keys?

As a side note, is it all that odd for folks to run AutoPkg and the Munki client on the same box?  I wouldn't think so, but what do I know.  I mean I have AutoPkgr running on this iMac because it's always on.  But I also run the Munki client to update the apps on this iMac, as it's just like any other Mac I use.

[Greg Neagle]

# Checking what's in the cache, I see

% defaults read /Library/Preferences/ManagedInstalls
2023-02-04 13:01:04.553 defaults[42448:1320534] 
Domain /Library/Preferences/ManagedInstalls does not exist

# However, if I actually check the file itself, I find

% ls -al /Library/Preferences/Managed*
-rw-r--r--  1 root  wheel  1183 Feb  4 05:31 /Library/Preferences/ManagedInstalls.plist

 % file /Library/Preferences/ManagedInstalls.plist 
/Library/Preferences/ManagedInstalls.plist: Apple binary property list

I find that an amazing combination of things. But `defaults read /Library/Preferences/ManagedInstalls` returning "Domain /Library/Preferences/ManagedInstalls does not exist” is yet another sign that _something_ is modifying or interfering with the preferences. 

"is it all that odd for folks to run AutoPkg and the Munki client on the same box?”

I don’t know if it’s odd or not. I’ve always done so, though, and without issue. I can’t really think of any potential issues, either.

"but does AutoPkg in any way make reference to this same domain/keys?”

AutoPkg itself doesn’t read or write Munki’s preferences, but it’s possible a specific recipe or processor could.

(A search through the AutoPkgs code repository on GitHub has zero results for “ManagedInstalls”; there are also no results for “ManagedInstalls” in the entire autopkg organization, which would cover the vast majority of publicly available recipes and custom processors.)

"I mean I have AutoPkgr running on this iMac because it's always on.”

I don’t use AutoPkgr (this is a separate project/product from autopkg), but I do not believe it reads or write Munki’s preferences (and a search through its code repository on GitHub has zero results for “ManagedInstalls”), so it is unlikely to be the source of this issue.

If/when you discover what is causing this it will be quite interesting, I’m sure. I’m hoping this doesn’t turn out to be something like “the startup disk is getting full periodically”.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/12d32ad8-7e0c-4d96-a73f-ce1b13afa316n%40googlegroups.com.

[Frank Seesink]

"If/when you discover what is causing this it will be quite interesting, I’m sure."

Tell me about it.  It's got me scratching my head for sure.  I mean, other than the usual OS and app updates, this box hasn't done anything new/unusual in eons.

"I’m hoping this doesn’t turn out to be something like “the startup disk is getting full periodically”."

Nah, checked that early on.  This particular Mac currently is sitting with nearly 200GB free.  I'm pretty sure I'm safe on that front. :-)

Regarding

"AutoPkg itself doesn’t read or write Munki’s preferences, but it’s possible a specific recipe or processor could."

I started wondering about whether somehow it might be a recipe somehow.  But what's truly odd is that if that were the case (e.g., some rogue recipe tweak is deleting that entire domain), wouldn't that happen with each run of autopkg?  The strangest part to this is that it'll work for days, and then at some point it'll stop.

I mean here's my overall workflow, which hasn't changed in years.  This iMac runs AutoPkgr once a day @6:AM as I mentioned.  If it finds any updates, it downloads them, puts them in the 'test' catalog, and it sends me a Slack message.  (I have some of my own boxes set to pull from the 'test' catalog, so I'm one of the guinea pigs for the updates.)  Then, usually once a day (sometimes it might be a few days if I'm busy or there's no important updates), I'll remote in and run MunkiAdmin, where I'll move things from 'test' to 'production'.  This typically involves unchecking the new version of some app from 'test', then unchecking an older version of that app (e.g., Firefox) and checking the new one.  I save the config, at which point MunkiAdmin throws the old ones in the trash.  I empty the trash, and then I run a small AppleScript I have configured which rsyncs the setup to my actual Munki server, which is a Linux box running in a cloud server.  Again, all the same bits for years now.  Finally, I run Managed Software Center on that same box to pull whatever updates I've just pushed out to the server.  It's my sanity check.

But whenever this issue happens and I then reboot this Mac in question, everything works again.  I can manually fire off Managed Software Center or the CLI command repeatedly, and I can watch that everything works as expected.  And then for days it seems everything works just fine.  But at some point, usually after I've done the above routine and do my sanity check, that's when I realize things are broken again, because I'll have updated some app I know is on that box, so when I run the Managed Software Center, I expect to see it pull that update down and it doesn't.

So it's really got me wondering what the heck is going on.  But I'll keep plugging away.  It's frustrating to be sure.  But there has to be some explanation for this. I just haven't been able to find it. And it just seems unlikely that it's something outside of the whole AutoPkg/Munki ecosystem that's doing this, as that seems a bit too targeted.  For sure if I can nail this down, I'll report back.  But if you all think of anything else that might be at play here, I am all ears.

[Frank Seesink]

Ok, happened again this morning.  And one more data point.

As before,

1. doing sudo managedsoftwareupdate --show-config shows settings reset to default
2. looking at /Library/Preferences/ManagedInstalls.plist shows settings still correct
   

Timestamp on .plist file is 5:43AM.

HOWEVER, looking in /Library/Managed Installs/Logs/ManagedSoftwareUpdate.log I see a successful run done at 6:29:06 (so nearly an hour after the .plist file timestamp) followed by the next attempt at 7:55:46 which failed.  This latter timestamp is also that of the .log file itself, so I caught this just after the first bad attempt.

But still at a loss what could be causing this.  I see nothing out of the ordinary.  Anyway, grabbed the log bits and will keep digging.

[Alan]

I think I said this in a previous message, but you may want to look beyond the Munki log, and start looking at your launch daemons to see if there's some other script that may be regularly messing with your Munki settings.

Also, I'm assuming you don't have any item in the Munki repo itself that is messing with the Munki settings? You may want to double-check that as well.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/e3f75301-63d3-4da5-8652-7fbcb6fd1472n%40googlegroups.com.

[Daniel Moore]

I remembered something today that might help. This command will show preference keys as they are written:

log stream --debug --predicate 'process == "cfprefsd" && eventMessage CONTAINS "wrote the key"'

(Command from https://boberito.medium.com/config-profile-and-manage-all-the-things-just-about-cafea8627d4b)

Perhaps that can help identify what is making changes?

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/CAE%2BBptxKzphaGT_%2B8vMHzjUWkPUsgt0gvi33Mc4pku6GtSG0Xg%40mail.gmail.com.

[Greg Neagle]

Perhaps. It’s also very possible that the entire prefs domain is being deleted.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/CACOmFV20T6Pn1d3M4rWV_ZqsgLMdb4rsPhAOb3yVYLRSH5gXfg%40mail.gmail.com.

[Daniel Moore]

A few variations that might also be helpful:

to watch for defaults delete

log stream --debug --predicate 'process == "cfprefsd" && eventMessage CONTAINS "flush"'

to watch everything cfprefsd logs

log stream --debug --predicate 'process == "cfprefsd"'

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/6C8D7F8C-11B7-4FE7-8855-2DDE5F4DD482%40mac.com.

[Gregory Neagle]

Thank you. Though it could also be as basic as `rm /Library/Preferences/ManagedInstalls.plist`

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/CACOmFV2V%2B4SNwgcWF%3Dcnv%3DYQ-LY%2BvPuy0hTx%3D4LR_0jR3ng-bA%40mail.gmail.com.

[Frank Seesink]

Daniel, thanks.  Those commands may well come in handy.

"Thank you. Though it could also be as basic as `rm /Library/Preferences/ManagedInstalls.plist`"

Um, again, no.  That would flat out delete the preference file from the file system, something which, as I have stated multiple times now, is and always has been INTACT, complete with the correct Munki server URL and client ID.  Whatever is doing this appears to be deleting the cfprefsd cached copy of this domain.  Unless you somehow think something is deleting the .plist file, only to regenerate it WITH all the proper settings in place, but then somehow removing the cached copy that this cfprefsd holds.

[Frank Seesink]

Back on Thursday, I had opened a Terminal session and did
__________

% date

Thu Feb 9 21:11:37 EST 2023

% log stream --debug --predicate 'process == "cfprefsd" && eventMessage CONTAINS "flush"'

__________

and then left the Terminal up and locked the screen.

On Friday (10 Feb.) I found the iMac in this situation again.  Thanks to the above, I had a pile of logs.  Unfortunately, I'm not sure what I'm looking for.  There isn't a single reference to "ManagedInstalls" in the logs anywhere.  And just a small sample

__________

2023-02-09 22:08:05.874125-0500 0x10d3bc Debug 0x169090 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-09 23:06:10.012500-0500 0x111910 Debug 0x173ec0 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-09 23:14:38.224998-0500 0x11265a Debug 0x175c40 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 01:05:49.713642-0500 0x119920 Debug 0x17f2b0 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 01:15:14.734991-0500 0x11a72b Debug 0x180a60 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 02:45:16.192050-0500 0x120737 Debug 0x185600 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 03:52:10.042500-0500 0x124e6a Debug 0x189560 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 04:45:12.508369-0500 0x128665 Debug 0x18c4c0 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 05:48:04.123624-0500 0x12cc64 Debug 0x18fdf0 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 06:40:55.677795-0500 0x135935 Debug 0x1a1610 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 07:29:31.140794-0500 0x139f0b Debug 0x1a9500 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 08:46:28.387956-0500 0x13f2bf Debug 0x1ad520 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

2023-02-10 08:53:31.875754-0500 0x13fc96 Debug 0x126d2c 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] Process 667 (iCloudNotificationAgent) requested flush of managed sources

2023-02-10 08:53:31.875825-0500 0x13fc96 Debug 0x126d2c 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.mediaplaybackcore, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Managed Preferences/com.apple.mediaplaybackcore.plist, managed: 1 } because client invalidated domain

2023-02-10 08:53:31.875906-0500 0x13fc96 Debug 0x126d2c 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.CallHistory, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Managed Preferences/com.apple.CallHistory.plist, managed: 1 } because client invalidated domain

2023-02-10 08:53:31.875981-0500 0x13fc96 Debug 0x126d2c 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.hjuutilainen.MunkiAdmin, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Managed Preferences/com.hjuutilainen.MunkiAdmin.plist, managed: 1 } because client invalidated domain

...

2023-02-10 08:53:32.559270-0500 0x13fd89 Debug 0x1ae280 489 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.Spotlight, frank, kCFPreferencesCurrentHost, /Library/Managed Preferences/frank/com.apple.Spotlight.plist, managed: 1 } because client invalidated domain

2023-02-10 08:53:32.559454-0500 0x13fd89 Debug 0x1ae280 489 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.TextInputMenu, frank, kCFPreferencesCurrentHost, /Library/Managed Preferences/frank/com.apple.TextInputMenu.plist, managed: 1 } because client invalidated domain

2023-02-10 08:53:32.559695-0500 0x13fd89 Debug 0x1ae280 489 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.proactive.PersonalizationPortrait, frank, kCFPreferencesCurrentHost, /Library/Managed Preferences/frank/com.apple.proactive.PersonalizationPortrait.plist, managed: 1 } because client invalidated domain

=== Messages dropped during live streaming (use `log show` to see what they were)

2023-02-10 09:59:42.260656-0500 0x144798 Debug 0x1b38f0 160 0 cfprefsd: (CoreFoundation) [com.apple.defaults:cfprefsd] flushing cache for { com.apple.alf, kCFPreferencesAnyUser, kCFPreferencesCurrentHost, /Library/Preferences/com.apple.alf.plist, managed: 0 } because we're avoiding the cache

__________

There are over 600 of these "...flushing cache for... because client invalidated domain" lines all around 08:53AM, followed solely by the one line just before I stopped the logging ~10:AM.

Anyway, don't see any entries to indicate the ManagedInstalls domain or anything within it was flushed.  All the lines appear to be specific to a given app flushing cache for  its own .plist.

I thought to try the other variant of the logging command given:

log stream --debug --predicate 'process == "cfprefsd"'

but that one looks to generate a LOT of data.

[Kevin M. Cox]

Frank, it won't identify what is causing this problem, but if you set your Munki options with a configuration profile I think it will at least fix the problem. The config profile should take precedence over anything happening with defaults commands or the file on disk.

~ Kevin

[Frank Seesink]

Hey Kevin,

Your suggestion sent me down a rabbit hole.  Do you, by any chance, have any recommended way of creating config files (by which I assume you mean .mobileconfig)?

It seems in the past a lot of folks used this tool for the task:  https://github.com/timsutton/mcxToProfile

Sadly, I can't get that to work on my systems.  I'm up on macOS 13.2 at this point, and even though I have Python3 installed, along with the Command Line Tools for XCode 14.2, seems this script is looking to "import Foundation..." which isn't part of a stock install I'm guessing (or not anymore).

I used my Google-fu to look for alternatives, but other than one paid-for app that was like $60 (a bit hard to justify in this setup), best I've been able to do is try to find sample .mobileconfig files (since it looks like XML, just in a different layout than text .plist files).  But not sure if there's a better/faster/easier way, or at least a simple guideline on what the minimum is you can put into such a profile, as I only really need a base few settings.

I downloaded the Apple Configurator 2, but that thing looks like it has a lot of stock OS level settings, but not sure it lets you put in custom ones (which I'm guessing is what I need here).  So I can't use it to generate a sample for me that I can hack on.

Anyway, any insight would be appreciated.  At this point, I agree.  If I can use profiles and put this to bed, that's fine.

[Vaughn Miller]

ProfileCreator is one popular tool : https://github.com/ProfileCreator/ProfileCreator

Vaughn

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/2398610e-9da8-4b49-8b7b-7c52ff34f371n%40googlegroups.com.

[Gregory Neagle]

I’d try running the mcxToProfile tool using munki-python.

/usr/local/munki/munki-python /path/to/mcxToProfile.py

Note you don’t want to just use /Library/Preferences/ManagedInstalls.plist as input to that tool and then use the output without editing it.

Manage only the specific keys you must.

-Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/2398610e-9da8-4b49-8b7b-7c52ff34f371n%40googlegroups.com.

[Kevin M. Cox]

The two most common free options are:

https://github.com/ProfileCreator/ProfileCreator

https://imazing.com/profile-editor

I've also got an example here you could just download and edit for your needs:

https://github.com/kevinmcox/Configuration-Profiles/blob/main/Munki.mobileconfig

~ Kevin

[Frank Seesink]

Hey Vaugh, Gregory, and Kevin,... and for that matter, everyone else who's responded in here,

Thanks everyone.  Seriously.  Really appreciate all the help with this.

Yeah, I'd found those apps earlier tonight (except for Kevin's example .mobileconfig) after Kevin's post had me go off digging.

iMazing was the paid one I mentioned.  (Just hard to justify for a personal Munki setup of just a dozen machines, though. :-/ )

And wasn't until I downloaded the `.zip` of the ProfileCreator and gave that a whirl that I realized what a really nice little program that is!  Just wow.  I set just the settings I wanted (they even had Munki specifically as one of the apps... super handy).  I did a quick preview of the `/Library/Preferences/ManagedInstalls.plist` and copy/pasted the XML content (since the actual file is in binary form) over into VSCode, just so I could look at ProfileCreator's settings while comparing to the XML stanzas set in that file.

So in the end I made a .mobileconfig file for that iMac which I double-clicked on and installed there.  And sure enough (I hadn't yet rebooted), I found now `sudo managedsoftwareupdate --show-config` showed the right settings again.  So yeah!  No more reboots needed.

No, it doesn't tell me what the root cause is, and why this changed in recent months on this one box when it worked fine for years.  I suspect it's somewhere in the whole Munki ecosystem, like one of the recipes that AutoPkgr is running.  I can't think what else would be doing this.  But at this point, I'm gonna live with that.

I DO like the idea of creating these profiles for each client device.  It's a much nicer/cleaner setup than hand-crafting the `default write` commands for each client.  This way if a family member adds a new device, I can just

1. Have them install MunkiTools

2. Send them their .mobileconfig file that they double-click on and install

3. ...

4.  Profit!

:-)

Seriously, though, much appreciated for all the help, everyone.  I learned some interesting bits along the way (and I LOVE when I learn something new).  My setup got one step better.  And I have a new tool to add to my arsenal.  Though I don't have as much time to tinker as I'd like, that's one for the "tuck in the back of my head" situations.  I can see leveraging those config profiles more in the future.

[Kevin M. Cox]

Glad it seems to be working Frank.

Just for clarity, iMazing Profile Editor is a completely free tool that is NOT part of the paid iMazing application: https://imazing.com/profile-editor

~ Kevin

[Frank Seesink]

Hey Kevin,

Oh, thanks!  I did not realize that.  What I found was a commercial only app.  Appreciate that link.

[Frank Seesink]

Now for the bad news.  The issue is still there.

I wish I were kidding.  But this morning was doing the usual check on the iMac for app updates found by AutoPkgr using MunkiAdmin, and after doing my usual of moving updated apps from testing to production and rsyncing to the server, when I ran Managed Software Center, no updates.  I checked from Terminal with usual "sudo managedsoftwareupdate --show-config", and once again the ClientID and Repo URL were reset to defaults!!  WTH?

Now mind you, I went to check in my System Settings, and sure enough, that profile I created/installed was still there!  So now what?

Next, just to see what would happen, I removed that profile, then added it back in, hoping it would do like I saw last time, showing me those 2 attributes marked as "[Managed]".  But nope.  Still shows those attributes set to defaults.  So if managedsoftwareupdate takes profile data over the .plist file, that still doesn't get around this issue apparently.  Guessing it still relies on this cached copy?

Finally, I rebooted.  Sure enough, once rebooted, ClientID and Repo URL were showing what they should, and it worked as expected.

Anyway, will keep watching this.  But please note that using .mobileconfig profiles does NOT resolve this issue apparently.

[Alan]

There's something wrong with your macOS installation. That isn't normal behavior.

I created a Munki profile using iMazing and installed it manually.

sudo managedsoftwareupdate --show-config showed the settings as managed:

ClientIdentifier: 'dontmesswiththis' [MANAGED]

SoftwareUpdateServerURL: 'https://www.munki.com' [MANAGED]

You cannot script uninstalling profiles (I think since Big Sur). When I tried to do so with a command, I got an error (that error is normal behavior):

sudo profiles -r -p com.github.aysiu.munkisettings -u 33C17EB7-D046-4BB3-A5FA-73333F2E7FA7
Unable to remove the profile because it is locked. (-204).

I did notice that iMazing has a weird part which is The date on which the profile will be automatically removed, but even with that date present, I didn't see any related key to that in my .mobileconfig file.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/9cc2d8d8-d2cc-4fab-b6be-295acf5a8098n%40googlegroups.com.

[Frank Seesink]

So this issue is still ongoing.  Been too busy to circle back to write anything 'til now.  But basically nothing improved by using .mobileconfig profiles.  After several days, that iMac seems to forget its Munki settings.  I typically reboot that machine on Wednesdays when I'm sitting in front of it, and this usually manifests before the following Wed. (like this morning, after everything was still working fine yesterday).  And as before, a reboot fixes things.

So coupla questions here.

1. When first deployed, as mentioned, I was using `/Library/Preferences/ManagedInstalls.plist` on all my client machines including this one.  I have now added a .mobileconfig profile, and it does show with the `[MANAGED]`.  But I never removed the .plist file.  Does this matter?  That is, if both are on a client machine vs. just having a profile, does that impact how things work?
2. If someone only uses .mobileconfig profiles on client machines, does that mean managedsoftwareupdate isn't still relying on the cache to pull these settings?  Because it seems to me whatever is going on is somehow removing these settings from the cache, as nothing else on the Mac seems affected.  But just wanting to verify that the s/w itself, regardless of whether you use profiles or the older .plist approach, makes any real difference in how the s/w runs.
   

Just trying to figure out what the heck is causing this.  And still drawing a blank.  I mean, if by simply having the .plist file on the client machine, that somehow alters the manner in which the s/w is getting its config (e.g., if the .plist exists it's somehow relying on the cache, but if it does not, it pulls directly from the profile somehow), then I'll remove the .plist, reboot, and go from there.  But figured let me ask, as it feels like no matter where you're putting your settings, whether a .plist or a .mobileconfig, while the latter might take priority over the former, in the end all the settings are stored in the cache and accessed from the cache by the s/w, then it really makes no difference whether someone uses .plist or .mobileconfig files.

[Alan]

You shouldn't be deploying a ManagedInstalls.plist file to /Library/Preferences as a payload. If you write to that file, it should be using defaults write commands.

That said, a .mobileconfig profile should always override what's written to the preferences file. If sudo managedsoftwareupdate --show-config shows a preference as [MANAGED], that means it doesn't matter what's written to the prefs using defaults write.

If you're describing the behavior accurately, that macOS installation is just broken. On a working Mac, .mobileconfig preferences always override locally set preferences, and .mobileconfig profiles don't just randomly go away (unless you're deploying via MDM and then using the MDM to pull the previously deployed profile), and rebooting doesn't randomly bring .mobileconfig profiles back.

So either you're lying (I don't think you are), you're not describing things properly (you're probably not), or your macOS installation is broken (most likely scenario).

I don't think anyone here can replicate on macOS 13.1 (or any macOS version Big Sur or higher) what you're describing.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-discuss/84717bdc-7dd9-452e-9d6f-fb60d94311d6n%40googlegroups.com.