Offline build broken with sbuild?

[Jan Kiszka]

Hi,

seems like at least sbuild-chroot-host is not properly cached /wrt the
additional Debian packages is pulls. Therefore, offline build fails.
I've just confirmed this bug report:

https://github.com/siemens/meta-iot2050/issues/382

What could be missing?

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Uladzimir Bely]

In the email from Monday, 21 November 2022 09:49:00 +03 user Jan Kiszka wrote:
> Hi,
>
> seems like at least sbuild-chroot-host is not properly cached /wrt the
> additional Debian packages is pulls. Therefore, offline build fails.
> I've just confirmed this bug report:
>
> https://github.com/siemens/meta-iot2050/issues/382
>
> What could be missing?
>
> Jan

Just performed a quick test with the latest Isar 'next':

1. Run first build
```
bitbake mc:qemuarm64-bullseye:sbuild-chroot-host
```
2. Cleaned "tmp" directory

3. Added ISAR_USE_CACHED_BASE_REPO = "1" and BB_NO_NETWORK = "1"

4. To be sure, stopped network on the machine and run build again:
```
bitbake mc:qemuarm64-bullseye:sbuild-chroot-host
```
Repo "base-apt" was created and the build was OK.

I'll try to check more things (e.g. Isar revision mentioned in the bug;
complete image build, etc), but at the first glance sbuild-chroot-host looks
to be cached OK, at least with the latest Isar.

--
Uladzimir Bely

[Uladzimir Bely]

In the email from Monday, 21 November 2022 10:36:43 +03 user Uladzimir Bely
wrote:

>
> I'll try to check more things (e.g. Isar revision mentioned in the bug;
> complete image build, etc), but at the first glance sbuild-chroot-host looks
> to be cached OK, at least with the latest Isar.
>

So, I've got the following result for now:

- In Isar itself cached cross-build works well (while both HOST_DISTRO and
DISTRO are equal). The only exception: raspberry targets, while DISTRO and
HOST_DISTRO are different by nature.

- In meta-iot2050, cached cross-build doesn't work for both "sbuild" and "pre-
sbuild" cases. Evend "isar-bootstrap-host" task doesn't work for cached build.

I've applied the following patch for now and it at least fixed sbuild-chroot-
host task for cached build. Waiting for full build finished and will update
you with the result.

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-
devtools/base-apt/base-apt.bb
index 506a28f..316960d 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -13,7 +13,9 @@ KEYFILES ?= ""
BASE_REPO_FEATURES ?= ""

populate_base_apt() {
- find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
+ distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+ find "${DEBDIR}"/"${distro}" -name '*\.deb' | while read package; do
# NOTE: due to packages stored by reprepro are not modified, we can
# use search by filename to check if package is already in repo. In
# addition, md5sums are compared to ensure that the package is the
@@ -38,7 +40,7 @@ populate_base_apt() {
"${package}"
done

- find "${DEBSRCDIR}"/"${DISTRO}" -name '*\.dsc' | while read package; do
+ find "${DEBSRCDIR}"/"${distro}" -name '*\.dsc' | while read package; do
repo_add_srcpackage "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
"${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}" \
"${BASE_DISTRO_CODENAME}" \


--
Uladzimir Bely

[Jan Kiszka]

Ah, another DISTRO-override issue! Hope we don't have more of that type
in the core. Maybe worth to scan again and check.

Thanks for analyzing and (likely) fixing!

[Jan Kiszka]

sbuild-chroot-host is indeed working now, but there are likely more
cases with (custom) DISTRO vs. BASE_DISTRO:

ERROR: isar-bootstrap-target-1.0-r0 do_bootstrap: ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None)
ERROR: Logfile of failure stored in: /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/temp/log.do_bootstrap.21764
Log data follows:
| DEBUG: Executing python function sstate_task_prefunc
| DEBUG: Python function sstate_task_prefunc finished
| DEBUG: Executing shell function do_bootstrap
| W: qemu-debootstrap is deprecated. Please use regular debootstrap directly
| I: Running command: debootstrap --verbose --variant=minbase --include=locales --no-check-gpg --arch=arm64 --components=main,contrib,non-free bullseye /build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/rootfs file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian
| I: Retrieving InRelease
| I: Retrieving Release
| I: Retrieving Packages
| I: Validating Packages
| I: Resolving dependencies of required packages...
| I: Resolving dependencies of base packages...
| I: Checking component main on file:///build/tmp/deploy/base-apt/iot2050-debian/apt/debian...
| E: Couldn't find these debs: apt
| WARNING: exit code 1 from a shell command.
| ERROR: ExecutionError('/build/tmp/work/iot2050-debian-arm64/isar-bootstrap-target/1.0-r0/temp/run.do_bootstrap.21764', 1, None, None)
ERROR: Task (/build/../work/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) failed with exit code '1'

[Jan Kiszka]

I think we have two problems here:

1. base-apt is set up in sources-list using the BASE_DISTRO, but we are
creating it using DISTRO, not only in bast-apt, also dpkg-base

2. the whole base-apt caching does not yet consider
BASE_DISTRO(DISTRO) != BASE_DISTRO(HOST_DISTRO), and that maybe even
before all the sbuild changes

The latter should break the Raspian case.

[Uladzimir Bely]

In the email from Wednesday, 23 November 2022 19:17:31 +03 user Jan Kiszka
wrote:

I've just send patchset that fixes case when DISTRO is changed from default
value (but still in fact remains "debian").

I set up the following configuration in local.conf:

> require conf/distro/debian-bullseye.conf
> DISTRO_NAME = "My Debian System"
> DISTRO = "my-debian"

And got the same problem with sbuild-chroot-host installation at 2nd (cached)
build. The patches solved the issue.

The case with Raspberry (target) / Debian (host) is more complex and still
doesn't work. To make cached build work in this case we at least need to
rework `base-apt` - split it onto 2 repos that are populated only from their
own original repos.

What concerns `meta-iot2050`, it still fails to do cached build, but due to
some local problems. For example, `python-absl` recipe from `meta-coral` layer
conflicts with BB_NO_NETWORK = "1" option...

--
Uladzimir Bely

[Jan Kiszka]

Left a comment

>
> What concerns `meta-iot2050`, it still fails to do cached build, but due to
> some local problems. For example, `python-absl` recipe from `meta-coral` layer
> conflicts with BB_NO_NETWORK = "1" option...
>

Yes, this is understood already, and a fix for meta-coral is pending.

Thanks,