Reliability and build time improvements with apt-cacher-ng

[MOESSBAUER, Felix]

Dear ISAR users,

I just played around with apt-cacher-ng to circumvent the throttling
issues on snapshots.d.o, and got some impressive results:

First of all, it is easy to configure ISAR to use the apt-cacher-ng, by
just using the DISTRO_APT_PREMIRRORS:

DISTRO_APT_SNAPSHOT_PREMIRROR = "deb.debian.org/(.*)
localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"

Then, an apt-cacher-ng instance needs to be started on the host.
When running inside the kas container, just use the external host IP,
or forward port 3142 into the container. While the build is running,
you can monitor the caching statistics under
http://localhost:3142/acng-report.html.

Second, the cache directory can - in theory - be cached in a CI as
well, so subsequent jobs can use it. Then, even less load is applied
onto snapshots.d.o. Even when using standard debian mirrors, this helps
as usually all ISAR build jobs run in parallel, hence do not share
already downloaded packages via ISARs internal cache.

Currently I'm playing around with adding a built-in apt-cacher-ng into
kas and the kas-container. By that, all users can instantly profit from
the speedup. For CI systems, an ambient caching service (e.g. via
gitlabs "services") would probably make more sense, but is also more
complicated to configure.

I'm happy to hear your thoughts.

Best regards,
Felix Moessbauer

--
Siemens AG, Technology
Linux Expert Center

[Alexander Heinisch]

Hi Felix!

> I just played around with apt-cacher-ng to circumvent the throttling
> issues on snapshots.d.o, and got some impressive results:

We just migrated our gitlab-runners (for a downstream project based on isar-cip-core) to a new build server.
The project builds 4 images (all for amd64 arch) whereas 2 of them are done in a multiconfig setup using the isar-installer.
In total the complete build fetches 423MB of apt packages from snapshot-cloudflare.debian.org

From an initial build time of ~12min (~35min on the old build server :-)) we could reduce build times (building all 4 images) down to ~7min using apt-cacher-ng.

> First of all, it is easy to configure ISAR to use the apt-cacher-ng, by
> just using the DISTRO_APT_PREMIRRORS:

Yes, we just passed `DISTRO_APT_PREMIRRORS` to our kas yaml like:

```
env:

  DISTRO_APT_PREMIRRORS: ""
```

> DISTRO_APT_SNAPSHOT_PREMIRROR = "deb.debian.org/(.*)
> localhost:3142/snapshot.debian.org/archive/\1/${APT_SNAPSHOT_DATE}/\n"

Although, we are using snapshot versions of the packages, we don't use `DISTRO_APT_SNAPSHOT_PREMIRROR`  as of now, but will switch soon!

 
> Then, an apt-cacher-ng instance needs to be started on the host.
> When running inside the kas container, just use the external host IP,
> or forward port 3142 into the container. While the build is running,
> you can monitor the caching statistics under
> http://localhost:3142/acng-report.html.
 

So for local builds we installed `apt install apt-cacher-ng` and
```
export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian/20240311/\n \
                              snapshot-cloudflare.debian.org/archive/debian-security/(.*) localhost:3142/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
```
does the job!

> Second, the cache directory can - in theory - be cached in a CI as
> well, so subsequent jobs can use it. Then, even less load is applied
> onto snapshots.d.o. Even when using standard debian mirrors, this helps
> as usually all ISAR build jobs run in parallel, hence do not share
> already downloaded packages via ISARs internal cache.

For the build server we added a dummy interface to bind on and used something like:

```
image: ghcr.io/siemens/kas/kas-isar:4.0
variables:
  APT_CACHE: 192.168.42.1:3142
script:
  - export DISTRO_APT_PREMIRRORS="snapshot-cloudflare.debian.org/archive/debian/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian/20240311/\n snapshot-cloudflare.debian.org/archive/debian-security/(.*) ${APT_CACHE}/snapshot-cloudflare.debian.org/archive/debian-security/20240311/\n"
  - kas build ${TARGET}
```
in our CI pipeline.

 
> Currently I'm playing around with adding a built-in apt-cacher-ng into
> kas and the kas-container. By that, all users can instantly profit from
> the speedup. For CI systems, an ambient caching service (e.g. via
> gitlabs "services") would probably make more sense, but is also more
> complicated to configure.

Usage with kas-container script (from cip-core) did not work for me.
Since the handling of `DISTRO_APT_PREMIRRORS` as well as `--runtime-args`
leads to `docker: invalid reference format.` once spaces are contained,
or, if escaped, the parser used in `isar-bootstrap.inc` could not split it correctly.
(We didn't dig too deep into that, since most of us use kas natively.)
For completeness: We are using:
isar-cip-core @ 95e952569b59602dd05f4138de9602a5b0398f0a
isar @ 3d863aad5e12c7bef382cb6fb3f1f5e18b236156

> I'm happy to hear your thoughts.

Hope that helps :-)

BR Alexander

[MOESSBAUER, Felix]

On Thu, 2024-08-22 at 04:10 -0700, 'Alexander Heinisch' via isar-users
wrote:

> Hi Felix!
>
> > I just played around with apt-cacher-ng to circumvent the
> > throttling
> > issues on snapshots.d.o, and got some impressive results:
>
> We just migrated our gitlab-runners (for a downstream project based
> on isar-cip-core) to a new build server.
> The project builds 4 images (all for amd64 arch) whereas 2 of them
> are done in a multiconfig setup using the isar-installer.
> In total the complete build fetches 423MB of apt packages from
> snapshot-cloudflare.debian.org

Please note, that snapshot-cloudflare stopped mirroring in June. The
only official debian snapshot mirror is snapshot.debian.org, and there
we have massive problems with rate limiting.

For more details, please also see "[PATCH 1/1] switch debian snapshot
mirror to snapshot.debian.org".

Best regards,
Felix

[Heinisch, Alexander]

> > Hi Felix!
> >
> > > I just played around with apt-cacher-ng to circumvent the throttling
> > > issues on snapshots.d.o, and got some impressive results:
> >
> > We just migrated our gitlab-runners (for a downstream project based on
> > isar-cip-core) to a new build server.
> > The project builds 4 images (all for amd64 arch) whereas 2 of them are
> > done in a multiconfig setup using the isar-installer.
> > In total the complete build fetches 423MB of apt packages from
> > snapshot-cloudflare.debian.org
>
> Please note, that snapshot-cloudflare stopped mirroring in June. The only official debian snapshot mirror is snapshot.debian.org, and there we have massive problems with rate limiting.
>
> For more details, please also see "[PATCH 1/1] switch debian snapshot mirror to snapshot.debian.org".

Thanks for the hint!
The rate limiting was the reason to switch to cloudflare mirror in the first place.

Currently, we are on a snapshot preceding the current cloudflare limitation, but
we will upgrade in the next weeks when preparing for the next release.

I never experienced the rate limiting outside our corporate network.
That way, maybe, rate limiting won't hit us that hard, when using the cache in our ci/cd.

What do you think of hosting snapshot mirrors for isar based projects (or others) on our own?

BR Alexander