Discussion topic: precaching of optional items

[Gregory Neagle]

In this Munki3dev commit: https://github.com/munki/munki/commit/3f3d3ea0324d2db735cc1410fb76a632c065db2c I've merged in some code to support the precaching of certain optional installs.

You can mark an item for precache by adding 

<key>precache</key>

<true/> to the item's pkginfo.

If such an item is in the optional_installs for a given machine and is available for install/update, additional information about the item will be recorded in InstallInfo.plist, and the installer item, if it exists in the Managed Installs Cache folder, will not be deleted even if it's not currently scheduled to be installed.

Mark one or more (currently uninstalled) optional items with the precache flag, then run the managedsoftwareupdate from this branch.

Examine /Library/Managed Installs/InstallInfo.plist for those items to see the additional detail.

There is also code to actually download items marked for precache, but it is not currently "wired up" to anything. You can manually run it like so:

% python

Python 2.7.10 (default, Feb  7 2017, 00:08:15) 

[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> from munkilib.updatecheck import precaching

>>> precaching.cache()

You should see it download the optional_installs items that have the precache flag in /Library/Managed Installs/InstallInfo.plist.

I haven't wired this up anywhere yet because I can't decide how to implement the actual precaching. Two strategies occur to me, each with their plusses and minuses:

1) Do the precaching as part of a `managedsoftwareupdate --auto` run; that is, during the "normal" periodic background runs on `managedsoftwareupdate`. This would be fairly easy to implement. But if a poor soul happened to open Managed Software Center while a large item was precaching, they might be frustrated that they could not install some optional item for some possibly very long time until after the precaching completed. (This type of issue can happen today if Munki happens to be doing a lengthy managed_install download or install when Managed Software Center is launched, so maybe this is not a huge deal).

2) Have a secondary process do the precaching independently of managedsoftwareupdate runs. This avoids some of the issues in #1, but presents a few issues of its own. For example, while the secondary process was precaching Foo, a user could launch MSC.app and choose Foo for immediate install. managedsoftwareupdate would start downloading the item -- it would have no idea that the secondary process was already downloading the item. This could lead to less than excellent results.

There are ways to work around the issues with either approach, but I think we should take the time to think through the options, the possible issues, the possible workarounds, and the implementation difficulties.

-Greg

[Gregory Neagle]

Oh -- I forgot to give Alan Siu credit for starting the ball rolling on the precaching code.

-Greg

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.
To post to this group, send email to munk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eric Holtam]

A possible workaround for method #2 would be to not show an optional item in MSC that has precache set until it exists in the Managed Installs Cache folder or it's already been installed by munki (so it can be listed to be removed). If an item is configured to be precached that is a deliberate action by the admin so the desired state for said item is to be precached. Not listing the item means there would be no attempt to install an item that is precaching and it would avoid the issue in method #1 since the precache process is independent of the normal --auto runs. 

One issue with this approach is that it would take two munki runs to get an precached optional item installed. One for the precache process to download then another run to list in MSC and install it.  However if speed of installation/availability is of the essence then precaching doesn't make sense.

I don't know what complexity that entails in the implementation side of things but that's my initial thought on this.

-Eric 

[Mr. Alan Siu]

I'd be inclined to go with #1. It's up to the admin when to put the precache key in for a pkginfo (not that every organization has a "convenient" time for --auto downloads to happen), and, as Greg mentioned, it would be the same inconvenience as a large managed_install. You're basically saying this is an optional item that is a managed download.

Alan Siu

Client Systems Analyst

St. Ignatius College Preparatory

--

[Nick McSpadden]

Since I'm more likely to adopt Strategy #2, here's a proposal for handling external tools copy large files into the cache when they are marked for preache:

As you pointed out, if we start copying, say, Xcode9.dmg into the cache folder, it's entirely possible Munki runs while the copy is still going. Munki would then start downloading Xcode9 because the existing file doesn't match the hash in the pkginfo.

If we named the precache temp download/copy differently (for a bad example, "Xcode9.dmg.precache"), Munki would run and then handle it in the following situations:

Xcode9 is in the optional installs, but not in a managed install in SelfServeManifest. 

1) Munki will see that there is a precache download in the cache folder, but it doesn't match the hash in the catalog. Therefore Munki ignores it completely, and the file is not deleted if Munki finishes without performing any operations (i.e. the cache is not cleaned up for this file). 

1a) If a user clicks Install in MSC, Munki begins downloading Xcode9.dmg into the cache as Xcode9.dmg.download as it normally does, and the precache is not used (because it was incomplete at time of user interaction).

2) Xcode9.dmg.precache *does* match the hash in the catalog. Since Munki knows it's there and matches, if a user clicks Install in MSC, Munki uses it and does not try to download, because it's already there.

This makes the admin responsible for cleaning up cache files that are incomplete (such as an interrupted download, rsync, copy, or whatever), but I think that's acceptable. If the admin is going to take on the responsibility of populating precached items, then that also includes detecting cleanup/error conditions.

It also means that Munki gets to operate as normal if the external tool populating the precache isn't complete by the time a user interacts with MSC / Munki. Almost no other Munki behavior should have to change to accommodate it, since the precache has a different naming scheme and can be treated mostly independently.

To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+unsubscribe@googlegroups.com.

To post to this group, send email to munk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+unsubscribe@googlegroups.com.

To post to this group, send email to munk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
Nick McSpadden
nmcsp...@gmail.com

[Mr. Alan Siu]

I like the idea of a second location, because it means Munki could do the precaching or an outside process could handle it.

I would see it as Munki first looking in /Library/Managed Installs/Cache to see if the item with the correct hash is already precached there, then look in whatever that secondary location is to see if the item with the correct hash is there instead. If it's not downloaded properly in either case, Munki would download it to /Library/Managed Installs/Cache

Alan Siu

Client Systems Analyst

St. Ignatius College Preparatory

To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

To post to this group, send email to munk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

To post to this group, send email to munk...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

--
Nick McSpadden
nmcsp...@gmail.com

--

Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

[Erik Gomez]

Should there be some code for also a threshold of space?

Perhaps a key for cachethreshold and do a set amount of space. Munki could then not cache if it's under that threshold and purge any cached items to free up space for the user.

Thanks,

Erik Gomez

---

From: munk...@googlegroups.com <munk...@googlegroups.com> on behalf of Mr. Alan Siu <as...@siprep.org>
Sent: Friday, May 18, 2018 1:28:21 PM
To: munk...@googlegroups.com
Subject: Re: [munki-dev] Discussion topic: precaching of optional items

 

[Gregory Neagle]

I'm not that interested in spending a lot of time/effort working on things that would support some outside/unrelated to Munki process that does precaching.

If people want that, they don't need _any_ changes to Munki itself to implement it, and they can go off and build it (and optionally make it available to others).

You could have a process you write/create precache items somewhere/anywhere. Let's say "/Library/Managed Installs/Precache".

A Munki preflight script could then create hard links (or maybe even soft links) to items in /Library/Managed Installs/Precache in /Library/Managed Installs/Cache

Munki itself would still delete items in /Library/Managed Installs/Cache it didn't think were needed for install, but it would be deleting _links_; the original items would still exist in /Library/Managed Installs/Precache, where they'd be relinked on the next Munki run.

There'd need to be some additional logic to clean up the Precache if/when a preached item is installed.

The thing _I_ want to work on is _integrating_ precaching with Munki.

-Greg

[Mr. Alan Siu]

I think the approach that makes the most sense, then, is to have it download as part of an --auto run.

The precache pkginfo key will optional. And if you found some other way to get the download into /Library/Managed Installs/Cache (with the correct name and hash), Munki wouldn't redownload it.