[PATCH 1/1] improve apt fetching from rate-limited snapshot mirrors

[Felix Moessbauer]

The snapshot mirrors apply strict rate limiting, making it impossible to
use them with a default apt configuration. On the other hand, apt does
not obey the retry-after http response header, but uses a exponential
backoff mechanism (retry after 1,2,4,...) seconds. By that, the
currently used 3 retries are basically useless as apt aborts way before
the rate-limit expires.

We change that to retry at most 10 times with a futher upper limit of 10
minutes (both only when using ISAR_USE_APT_SNAPSHOT). This is a
compromise between not stalling failed builds but also making them more
reliable.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
doc/user_manual.md | 4 +++-
meta/classes/rootfs.bbclass | 5 ++++-
meta/conf/bitbake.conf | 2 ++
3 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 284847c8..4b07f903 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -433,7 +433,9 @@ Some other variables include:
- `HOST_DISTRO_BOOTSTRAP_KEYS` - Analogously to DISTRO_BOOTSTRAP_KEYS: List of gpg key URIs used to verify apt bootstrap repo for the host.
- `DISTRO_APT_PREMIRRORS` - The preferred mirror (append it to the default URI in the format `ftp.debian.org my.preferred.mirror`. This variable is optional. PREMIRRORS will be used only for the build. The final images will have the sources list as mentioned in DISTRO_APT_SOURCES.
- `ISAR_USE_APT_SNAPSHOT` - Use a frozen apt snapshot instead of the live mirror. Optional.
- `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
+ - `ISAR_APT_DL_LIMIT` - Rate limit the apt fetching to n kB / s. Optional.
+ - `ISAR_APT_RETRIES` - Number of apt fetching retries before giving up. Optional
+ - `ISAR_APT_DELAY_MAX` - Maximum time in seconds apt performs retries. Optional
- `DISTRO_APT_SNAPSHOT_PREMIRROR` - Similar to `DISTRO_APT_PREMIRRORS` but for a snapshot, pre-defined for supported distros.
- `ISAR_APT_SNAPSHOT_TIMESTAMP` - Unix timestamp of the apt snapshot. Automatically derived from `SOURCE_DATE_EPOCH` if not overwritten. (Consider `ISAR_APT_SNAPSHOT_DATE` for a more user friendly format)
- `ISAR_APT_SNAPSHOT_DATE` - Timestamp in upstream format (e.g. `20240702T082400Z`) of the apt snapshot. Overrides `ISAR_APT_SNAPSHOT_TIMESTAMP` if set. Otherwise, will be automatically derived from `ISAR_APT_SNAPSHOT_TIMESTAMP`
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index c7011508..03e80018 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -148,7 +148,10 @@ rootfs_configure_apt() {

mkdir -p '${ROOTFSDIR}/etc/apt/apt.conf.d'
{
- echo 'Acquire::Retries "3";'
+ echo 'Acquire::Retries "${ISAR_APT_RETRIES}";'
+ if [ -n "${ISAR_APT_DELAY_MAX}" ]; then
+ echo 'Acquire::Retries::Delay::Maximum "${ISAR_APT_DELAY_MAX}";'
+ fi
if [ -n "${ISAR_APT_DL_LIMIT}" ]; then
echo 'Acquire::http::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
echo 'Acquire::https::Dl-Limit "${ISAR_APT_DL_LIMIT}";'
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 13966a62..5fe50787 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -148,6 +148,8 @@ export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
SOURCE_DATE_EPOCH_FALLBACK ??= "1709565251"
# Debian snapshots
ISAR_USE_APT_SNAPSHOT ??= "0"
+ISAR_APT_RETRIES ??= "${@'10' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else '3'}"
+ISAR_APT_DELAY_MAX ??= "${@'600' if bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT')) else ''}"
ISAR_APT_SNAPSHOT_TIMESTAMP ??= "${SOURCE_DATE_EPOCH}"

# Default parallelism and resource usage for xz
--
2.39.5

[Jan Kiszka]

Seems to help, just unbroke the riscv build of isar-cip-core with that
which works against snapshot.d.o so far.

jan

--
Siemens AG, Technology
Linux Expert Center

[MOESSBAUER, Felix]

On Wed, 2024-11-06 at 13:41 +0100, Felix Moessbauer wrote:
> The snapshot mirrors apply strict rate limiting, making it impossible
> to
> use them with a default apt configuration. 

Hi, can we please merge this soon-ish, if there are no objections? We
already have multiple downstream layers that carry this patch to be
able to build against snapshots again.

At the same time I'm working on improve the performance of snapshot.d.o
, together with the upstream maintainers.

Best regards,
Felix

[Uladzimir Bely]

On Wed, 2024-11-06 at 13:41 +0100, 'Felix Moessbauer' via isar-users
wrote:

Applied to next, thanks.

--
Best regards,
Uladzimir.