base-apt: Inconsistent priorities between apt-cache and .deb data break offline builds

[MOESSBAUER, Felix]

Dear isar-devs,

we noticed that cross-architecture offline builds are currently broken
and single-architecture offline builds differ from the online
variant. One failure pattern is the following, whereby the offline
isar-mmdebstrap-host fails with an error message like this:

The following packages have unmet dependencies:
libtext-wrapi18n-perl : Depends: libtext-charwidth-perl but it is not
installable
E: Unable to correct problems, you have held broken packages.
E: The following information from --solver 3.0 may provide additional
context:
Unable to satisfy dependencies. Reached two conflicting decisions:
1. libtext-wrapi18n-perl:amd64=0.06-10 is selected for install
2. libtext-wrapi18n-perl:amd64 Depends libtext-charwidth-perl
but none of the choices are installable:
[no choices]

The root cause for this is that we ignore the upstream's repo priority
overrides when creating the base-apt: reprepro includedeb records the
priority from the .deb's own control field, and that becomes the
priority published in base-apt's Packages index.

When these priorities differ, the set of packages installed in offline
mode are different from the ones in online mode. This per-se is plain
wrong, but only surfaces if a now-required arch=all package depends on
its architecture specific variant which never has been fetched (because
the online bootstrap had access to the prio overrides). For non-cross
builds the result is also different, which can be checked by comparing
the SBOMs.

There are HACKY ways to re-create the upstream priorities, but IMHO our
approach for offline builds is problematic and should be changed:
Instead of explicitly recreating the base-apt repo, we better just
preserve the upstream apt lists and inject our downloads/deb dir into
the apt cache that is used by the build. This would allow us to
completely drop the base-apt.

Reproducer:

1. kas-container menu, select qemuarm64 machine, build
2. mv build build-online && mkdir build && cp -rf build-
online/downloads build/
3. kas-container build .config.yaml:offline.yml

offline.yml:
header:
version: 14
local_conf_header:
offline: |
ISAR_USE_CACHED_BASE_REPO = "1"
BB_NO_NETWORK = "1"

The initial report of this error is by Herbert.

Best regards,
Felix

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[Anton Mikanovich]

Issues like that supposed to be fixed by "Improving base-apt usage"
patchset.
It should address storing only needed packages, while DL_DIR is just a
storage
for everything without storing any real packages state in it.
The idea is to have base-apt as an input, while isar-apt is used as an
output.

We will look into the issue, starting from reproducing it.

>
> Reproducer:
>
> 1. kas-container menu, select qemuarm64 machine, build
> 2. mv build build-online && mkdir build && cp -rf build-
> online/downloads build/
> 3. kas-container build .config.yaml:offline.yml
>
> offline.yml:
> header:
> version: 14
> local_conf_header:
> offline: |
> ISAR_USE_CACHED_BASE_REPO = "1"
> BB_NO_NETWORK = "1"

Do we need such an option in kas menu?

[MOESSBAUER, Felix]

This rings a bell. Looks like the series could need a revival.

> It should address storing only needed packages, while DL_DIR is just a
> storage
> for everything without storing any real packages state in it.
> The idea is to have base-apt as an input, while isar-apt is used as an
> output.

I just checked the series, but this issue is not solved by it, as the
apt lists are not cached but only the packages. This is exactly what
breaks here: The packages have a different priority field than the apt
list.

I'm wondering if a simpler solution is a pull-through cache, like many
people use for cached building of containers (like here [1]). That
let's apt take care of the caching instead of the surrounding build
environment.

[1]
https://github.com/siemens/kas/blob/81bab5294be29de4b64adedce962d36184bfc66f/Dockerfile#L37

>
> We will look into the issue, starting from reproducing it.

It's easy to reproduce with these steps. You can even further reduce
the set of packages to the bare minimum to reproduce faster.

> >
> > Reproducer:
> >
> > 1. kas-container menu, select qemuarm64 machine, build
> > 2. mv build build-online && mkdir build && cp -rf build-
> > online/downloads build/
> > 3. kas-container build .config.yaml:offline.yml
> >
> > offline.yml:
> > header:
> >     version: 14
> > local_conf_header:
> >     offline: |
> >       ISAR_USE_CACHED_BASE_REPO = "1"
> >       BB_NO_NETWORK = "1"
> Do we need such an option in kas menu?

Probably? :) At least I did not see this option, but the variables are
documented.

Felix

[Anton Mikanovich]

Turns out not so easy: tried twice on clean setup with no luck so far.
Are there any other options or changes?

Here are debs in downloads after the build (on next b90b722f, default
debian-trixie):
libtext-wrapi18n-perl_0.06-10_all.deb
libtext-charwidth-perl_0.04-11+b4_amd64.deb
libtext-charwidth-perl_0.04-11+b4_arm64.deb

[MOESSBAUER, Felix]

Hi, you must start with an empty downloads folder. The libtext-
charwidth-perl_0.04-11+b4_amd64.deb indicates that the isar-image-base
was built for amd64.

For debugging, I recommend to just build the bootstrap target for amd64
and check that the libtext-charwidth-perl_0.04-11+b4_arm64.deb is NOT
in the downloads dir. Once that is confirmed, please retry the steps
from below.

Felix

>
> Here are debs in downloads after the build (on next b90b722f, default
> debian-trixie):
> libtext-wrapi18n-perl_0.06-10_all.deb
> libtext-charwidth-perl_0.04-11+b4_amd64.deb
> libtext-charwidth-perl_0.04-11+b4_arm64.deb
> >
> > > > Reproducer:
> > > >
> > > > 1. kas-container menu, select qemuarm64 machine, build
> > > > 2. mv build build-online && mkdir build && cp -rf build-
> > > > online/downloads build/
> > > > 3. kas-container build .config.yaml:offline.yml
> > > >
> > > > offline.yml:
> > > > header:
> > > >       version: 14
> > > > local_conf_header:
> > > >       offline: |
> > > >         ISAR_USE_CACHED_BASE_REPO = "1"
> > > >         BB_NO_NETWORK = "1"
> > > Do we need such an option in kas menu?
> > Probably? :) At least I did not see this option, but the variables are
> > documented.
> >
> > Felix
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/6de97274-3f2c-4b62-a7e9-82f6e1f2c338%40ilbers.de.

[Anton Mikanovich]

No, it was fetched during cross-building of any of hello/hello-isar/cowsay
packages, which actually mask this issue. So disabling default packages is
needed not just to speedup the build, but to unmask the issue.
Now I can reproduce it.