Proposal: Add retries for repo downloads

[Per Olofsson]

When munki needs to fetch a resource from the repo any network interruption causes the current run to fail. This is normally not an issue as munki will try again an hour later, or the user can check again in MSC. It is however a missed opportunity and it causes a fair amount of noise in the logs. We're seeing a steady stream of errors in Grafana and MunkiReport — not a huge issue, but it raises the noise floor.

I've experimented a bit with implementing retries in network/middleware/fetch.swift:getHTTPfileIfChangedAtomically(). It adds two new pref keys:

DownloadRetries: int between 0 and 10. Defaults to 0 which is no retries and thus keeps the old behavior.

RetrySleepSeconds: int between 1 and 30. Defaults to 10 seconds.

The code is here: https://github.com/munki/munki/compare/Munki7dev...MagerValp:munki:retry-download

Are you also seeing these issues and is this something you would find useful?

--
Per Olofsson, IT-service, University of Gothenburg

[Allister Banks]

This sounds up our alley. I can get more scientific/comprehensively gathered metrics data like percentages, but roughly I do notice the statistically significant failures are with downloading manifests, and our bootstrap can get hung up and miss parts of the relatively minimal set we configure and iterate on when testing new versions of our bootstrap (which starts with --download-only). For the bootstrap we’ve recently built in retries and approximations of a trivial backoff to increase chances of success, but considering we’re employing AWS to CDN’ify our pkgs via Cloudfront and manifests have been in elasticache (reducing the likelihood it’s source/server-side) we’d be appreciative if client/agent-side could gain this functionality.
Allister

[pne...@gmail.com]

I also see the same errors, but I have always ignored this because the retries set in the launchd item.

Would be great if it can "tweak" it a little bit with the suggested keys.

Patrick van Nerum

Hogeschool Rotterdam

[Gregory Neagle]

A test build containing support for this is here: https://github.com/macadmins/munki-builds/releases/download/v7.1.0.5477/munkitools-7.1.0.5477.pkg

This incorporates https://github.com/munki/munki/pull/1307

The retry support adds two ManagedInstalls preferences:

DownloadRetries: int between 0 and 10. Defaults to 0 which is no retries and thus keeps the old behavior.

RetrySleepSeconds: int between 1 and 30. Defaults to 10 seconds.

In addition, the test build also contains the PR from Jordan Calhoun adding support for Managed Software Center quitting and relaunching blocking applications when updates are selected for installation.

https://github.com/munki/munki/pull/1305

-Greg

--
Find related discussion groups here:
https://github.com/munki/munki/wiki/Discussion-Group
---
You received this message because you are subscribed to the Google Groups "munki-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to munki-dev+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/munki-dev/012CE28F-CBCA-438D-A10B-CB865F957172%40gu.se.