Parallel downloading of Munki packages

[pranav shenoy]

Hi Greg,

Recently, our team participated in an internal open source hackathon where we showcased downloading munki packages in parallel. We were able to reduce the overall installation time by upto 40% with minimal changes.

We would like to know any possible limitations that will prevent this feature from going into the production.  Refactoring might be needed to our changes as it was done in.a short span of 24 hrs, but would like to know your inputs before proceeding with it.

Changes  : https://github.com/pranavshenoy/munki/commit/810878a677f6ad260549f60e6940ffe975b67b6a

Please provide your inputs.

Thanks and Regards, 

Pranav Shenoy

[Gregory Neagle]

Hey Pranav:

This sounds interesting.

Can I ask what problem you are trying to solve?

Yes, I understand that this allows Munki to download multiple items faster --  but there is a cost. (There's always a cost). Munki is going to consume more CPU and a higher percentage of the available bandwidth. I assume that Macs  you manage are being used for something other than downloading and installing software. If I'm an artist submitting a render job, I want as much network bandwidth and CPU available for _my_ use, and don't want management tools using it all. Munki downloads generally happen in the background so I (the artist) don't really care how long they take.

(This is similar to the issue of Antivirus/Security software that has excessive CPU usage, or slows down network transfer while it sniffs them)

If you can better describe your scenario where it is important to reduce the time it takes to download multiple items, perhaps we can tune it to not consume too much bandwidth at other times.

I'll look at the technical details of the implementation later.

-Greg

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/7e0827ea-abdf-4498-8185-a1cbb93d67e5n%40googlegroups.com.

[Alan]

If this does get implemented, I'm hoping it would be a certain preference in ManagedInstalls and not just the default behavior. As Greg mentioned, a lot of us have users who would prefer Munki not take up too many resources, and the vast majority of the time, Munki does downloads in the background.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/F5435707-A1CB-491F-8A97-9E1E6E5EA0A9%40mac.com.

[Joaquin Cabrerizo Egea]

I’m with Alan,

I would prefer to not have this as the default behavior but having this for example to speed up the onboarding of a new user’s laptop or set up a lab for first time would be nice to have it as an option.

Enviado desde mi iPhone

El 1 abr 2021, a las 19:20, Alan <alan...@gmail.com> escribió:



To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/CAE%2BBptxHyjmdvwi8Zn%3DYCWVtQRa-1TvaJsXShEWhxABzr2yEOQ%40mail.gmail.com.

[pranav shenoy]

Thanks for the response. 

The issue that we are facing is the delay the user faces to set up his/her workstation. Admins push dozens of applications that are needed for users to start their onboarding process and we were looking at ways to reduce the downloading/installation time for faster onboarding.

Like you mentioned, there would be a cost, with high CPU and bandwidth consumption, but with the concerns that users are raising, I feel that they are ready to compromise the higher CPU and bandwidth consumption over reduced time (at least during the onboarding process when the applications are installed for the first time). 

I totally agree with the issues of Antivirus software's impact on excessive usage of CPU and bandwidth but one thing that I feel is different in the case of Munki is that users could be waiting for few apps (if not all) to be installed as soon as possible so that they could use it. 

Even If we could have a parallel downloading mechanism only for the onboarding process and sequential download for the rest of the time, I think it could be a great improvement.

Please share your thoughts.

Thanks.

[Ben Toms]

or.. you lessen the amount of items installed when provisioning and offer folks items via MSC

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/b3bf8671-4047-40fe-922a-0a0e18525230n%40googlegroups.com.

--

Regards,

Ben

[Mike Pullen]

Could this be enabled for login-screen installs but not for when users are signed in?

I can imagine that downloading in parallel for login-screen installs could be very beneficial, but would not be so beneficial when users are signed in...

Mike

--

[Mike Solin]

I'd love to use this for onboarding, as well as installs at the login window (like in our computer labs and classrooms). I agree with the sentiment that it'd probably consume too many resources during an 'auto' run with a user logged in.

We offer most items as optional installs in MSC, so our onboarding takes about 15 minutes, but any opportunity to further reduce that would be fantastic.

Mike

--

mi...@mikesolin.com / @flammable

https://mikesolin.com

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/CAHvh0Vt9G%3DawGddXfpP2D7P2Y-wLx94-XMVdktwia8bRxTvZng%40mail.gmail.com.

[Nick McSpadden]

I'd actually be interested in seeing this to be configurable in lots of different ways. Some ways I'd be interested in:

* As others have expressed, choose to run parallel-mode at loginwindow installations

* Manual check runs via MSC can be parallel by default;

* Specify exemption per-package (i.e. Xcode cannot/should not be downloaded parallel to other updates)

It's an interesting idea, if we're brainstorming.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/CAN1%2Bh_ae8aeFLg4JXm9J%2BqgyOrWnP7RGZdkKwhtWsNqzULM88Q%40mail.gmail.com.

--

--
Nick McSpadden
nmcsp...@gmail.com

[Allister Banks]

What’s one more knob! <stares at airplane cockpit full of knobs> 😅
One consideration is that we still have installer as a single-threaded process, although many things are moved into place via ditto from DMGs in recent times/many folks SOE’s.
Instead of login window, activating this functionality during DEPNotify type bootstrapping runs would be more applicable for my environment, nobody ever sees the login window almost ever for the lifetime of our devices (including bootstrap).
Allister

[Jim Zajkowski]

The downside to doing it in DEP runs is that - at least for me - a lot of our at-home customers have pathetic bandwidth and mediocre WiFi; trying to download a bunch of things at the same time makes them all crawl or time out.

For classroom builds, we’ve often saturated our package servers with 60+ systems reloading at the same time. Moving to multithreaded downloads is unlikely to make any of them faster.

—Jim

[Erik Gomez]

I would prefer a preference/flag so any command could be triggered with something like this. 

We use custom Munki commands for our provisioning process so gating it behind --auto/dep workflows would not help us. 

---

From: 'Jim Zajkowski' via munki-dev <munk...@googlegroups.com>
Sent: Friday, April 2, 2021 9:17:47 AM
To: 'Gregory Neagle' via munki-dev <munk...@googlegroups.com>
Subject: Re: [munki-dev] Parallel downloading of Munki packages

 

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/F8F4459C-A383-41F6-84A5-62B2B02C8CA0%40icloud.com.

[Eric Holtam]

I agree that the option to use parallel downloads should be configurable. Preferences/options are necessary to make any tool compatible with varying environments.

Settings can be changed via preflight/scripts/configprofiles, etc. for admins that have certain scenarios where parallel downloads should or should not be used.  As an option, the onus is on the admin to use it at the appropriate time.

A quick glance at the proposed code changes seems like the number of concurrent downloads could be a variable as well.

Possible Preference keys:

  ParallelDownloads -bool [Default: False] #Overall setting that auto-sets the next two

  ParallelDownloadsManualCheck -bool [Default: False] #Control if manual MSC update checks should trigger parallel downloads

  ParallelDownloadsLoginCheck -bool [Default: False] #Control if checks run when no user is logged in should use parallel downloads

  ParallelDownloadsMaxThreads -int [Default: 5] #Parallel may be handy but not at the same threshold as other sites, make the number of threads a setting.

-Eric

[Gregory Neagle]

This is getting to sound hopelessly complex to implement, test, and maintain. 

Sent from my iPhone

On Apr 2, 2021, at 1:25 PM, Eric Holtam <eho...@gmail.com> wrote:

I agree that the option to use parallel downloads should be configurable. Preferences/options are necessary to make any tool compatible with varying environments.

To view this discussion on the web visit https://groups.google.com/d/msgid/munki-dev/c636f16a-1424-4946-adf8-a7da46d93f8fn%40googlegroups.com.

[micha...@gmail.com]

I would love to use this feature during the bootstrap. My use case would be something likes this:

1. InstallApplications tool installs Munki package and loads its daemons and agents

2. Parallel download Munki settings are set

3. First Munki run is initiated. Downloads run in parallel.

4. Parallel download Munki settings are unset.