[PATCH v2 0/4] Only build arch=all package from -native variant

[Felix Moessbauer]

Architecture-all packages often cannot be built in cross mode, as their
dependencies cannot be resolved for the host architecture. Therefore, we
need to dispatch the build of arch=all packages from the target recipe
to the -native variant. This introduces some side effects:

- A mixed arch=(all|any) package needs to be built twice: once in
cross mode for the arch=any parts, and once in -native mode for the
arch=all parts (p2).
- Dependencies on arch=all packages from mixed source packages need to
be modeled so that consumers of a package do not need to be aware of
this distinction (p2).
- The arch=all packages must be available in the apt repository used
during target installation (p3).

Since we now ensure that arch=all packages are only built once, we can
safely revert the repo-split patch. Moreover, we must do so to make the
arch=all packages available during target installation.

PS: As our multiarch code history contains numerous fix tags,
I kindly ask everyone to test this series carefully.

Changes since v1:

- model dependencies to arch=all packages via DEPENDS / PROVIDES instead
of an explicit dependency to -native.
- add revert of repo-split patch
- add testcase

Best regards,
Felix Moessbauer

Felix Moessbauer (4):
mark cowsay package as architecture all
sbuild: do not build arch all packages on cross
Revert "Split up isar-apt into distro-arch specific instances"
add test for mixed arch all arch any packages

RECIPE-API-CHANGELOG.md | 16 ++++++++++++++++
meta-isar/recipes-app/cowsay/cowsay_git.bb | 2 ++
meta-test/recipes-app/test-all-any/files/control | 16 ++++++++++++++++
.../recipes-app/test-all-any/test-all-any.bb | 13 +++++++++++++
meta/classes-recipe/dpkg.bbclass | 3 ++-
meta/classes-recipe/multiarch.bbclass | 4 ++++
meta/conf/bitbake.conf | 4 ++--
testsuite/citest.py | 3 ++-
8 files changed, 57 insertions(+), 4 deletions(-)
create mode 100644 meta-test/recipes-app/test-all-any/files/control
create mode 100644 meta-test/recipes-app/test-all-any/test-all-any.bb

--
2.53.0

[Felix Moessbauer]

By that we do not cross-compile the cowsay package. It only consists
of arch=all packages.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
meta-isar/recipes-app/cowsay/cowsay_git.bb | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-isar/recipes-app/cowsay/cowsay_git.bb b/meta-isar/recipes-app/cowsay/cowsay_git.bb
index b0b8d3eb..0fa1e67c 100644
--- a/meta-isar/recipes-app/cowsay/cowsay_git.bb
+++ b/meta-isar/recipes-app/cowsay/cowsay_git.bb
@@ -5,6 +5,8 @@

inherit dpkg-gbp

+DPKG_ARCH = "all"
+
SRC_URI = "git://salsa.debian.org/debian/cowsay.git;protocol=https;branch=master"
SRC_URI += "file://isar.patch"
SRCREV = "756f0c41fbf582093c0c1dff9ff77734716cb26f"
--
2.53.0

[Felix Moessbauer]

Architecture all packages often cannot be built in cross mode, as the
dependencies cannot be resolved in the host architecture. This
especially applies to all packages which split their build dependencies
into Build-Depends-Arch and Build-Depends-Indep. We already have logic
in isar to built DPKG_ARCH=all packages non-cross, but this does not
work for packages that generate both arch specific and arch all binary
packages.

While sbuild automatically disables builds of arch-all packages on
cross, we previously explicitly overrode this setting. We now change
this by explicitly setting --no-arch-all flag on cross and --arch-all on
non-cross.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 16 ++++++++++++++++

meta/classes-recipe/dpkg.bbclass | 3 ++-
meta/classes-recipe/multiarch.bbclass | 4 ++++

3 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 856da5de..d4b093f4 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1001,3 +1001,19 @@ do_copy_boot_files: The recipe isar-image-base is trying to install
files into a shared area when those files already exists. It happens
when some files have the same names (e.g., dtb files) for different
distros.
+
+### No building of arch=all packages when cross building
+
+Architecture all packages often cannot be built in cross mode, as the
+dependencies cannot be resolved in the foreign architecture. This especially
+applies to all packages which split their build dependencies into
+`Build-Depends-Arch` and `Build-Depends-Indep`. We already have logic
+in isar to built `DPKG_ARCH = "all"` packages non-cross, but this does not
+work for packages that generate both arch specific and arch all binary packages.
+
+We now explicitly disable building `arch=all` binary packages during cross
+compilation (this only affects packages that produce both architecture-specific
+and `arch=all` binary packages). Recipes for such mixed packages should append
+`-all` to the `arch=all` binary packages listed in `PROVIDES` (e.g.,
+`PROVIDES="foo-doc-all"` for an `arch=all` binary package `foo-doc`). Consumers
+can then simply reference the package by its original name (e.g., `foo-doc`).
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index c1c38184..500aaefe 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -109,8 +109,9 @@ dpkg_runbuild() {

DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)

- sbuild -A -n -c ${SBUILD_CHROOT} --chroot-mode=schroot \
+ sbuild -n -c ${SBUILD_CHROOT} --chroot-mode=schroot \
--host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
+ ${@'--no-arch-all' if 'cross' in isar_deb_build_profiles(d).split() else '--arch-all'} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
--bd-uninstallable-explainer=apt \
--no-apt-update --apt-distupgrade \
diff --git a/meta/classes-recipe/multiarch.bbclass b/meta/classes-recipe/multiarch.bbclass
index cdd35f12..6c0fa809 100644
--- a/meta/classes-recipe/multiarch.bbclass
+++ b/meta/classes-recipe/multiarch.bbclass
@@ -83,6 +83,10 @@ python multiarch_virtclass_handler() {
for v in val.split():
if v.endswith('-compat') or v.endswith('-native'):
multiarch_var.append(v)
+ # dispatch -all (arch=all) to native variant
+ if v.endswith('-all'):
+ if suffix == '-native':
+ multiarch_var.append(v[:-len('-all')])
else:
multiarch_var.append(v + suffix)
d.setVar(var, ' '.join(multiarch_var))
--
2.53.0

[Felix Moessbauer]

This reverts commit dfa31ce4c8274b18c09c24fee804be6961c2ffd6.

The split was needed, because both the native and target builds
generated the arch=all packages, leading to conflicts. As we now only
build arch=all packages from the -native variant, each arch=all package
is just build once. By that, we also must use the same repo for both
host and target packages, as otherwise the arch=all packages are not
visible to the target install step.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/conf/bitbake.conf | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 5c71078d..08c525d9 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -93,8 +93,8 @@ TARGET_OS = "isar"
PACKAGE_ARCH ?= "${DISTRO_ARCH}"

# Isar apt repository paths
-REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt"
-REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/db"
+REPO_ISAR_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}/apt"
+REPO_ISAR_DB_DIR = "${DEPLOY_DIR}/isar-apt/${DISTRO}/db"

# Base apt repository paths
REPO_BASE_DIR = "${DEPLOY_DIR}/base-apt/${DISTRO}/apt"
--
2.53.0

[Felix Moessbauer]

This test checks the fixup of the PROVIDES and DEPENDS logic
on mixed architecture all and architecture any packages.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-app/test-all-any/files/control | 16 ++++++++++++++++
.../recipes-app/test-all-any/test-all-any.bb | 13 +++++++++++++

testsuite/citest.py | 3 ++-
3 files changed, 31 insertions(+), 1 deletion(-)

create mode 100644 meta-test/recipes-app/test-all-any/files/control
create mode 100644 meta-test/recipes-app/test-all-any/test-all-any.bb

diff --git a/meta-test/recipes-app/test-all-any/files/control b/meta-test/recipes-app/test-all-any/files/control
new file mode 100644
index 00000000..fc2bae7c
--- /dev/null
+++ b/meta-test/recipes-app/test-all-any/files/control
@@ -0,0 +1,16 @@
+Source: test-all-any
+Section: misc
+Priority: optional
+Maintainer: isar-users <isar-...@googlegroups.com>
+Build-Depends: debhelper-compat (= 10),
+Standards-Version: 4.1.3
+
+Package: test-all-any
+Architecture: any
+Description: Architecture specific test package
+ This is an architecture specific test package.
+
+Package: test-all-any-doc
+Architecture: all
+Description: Architecture independent test package
+ This is an architecture independent test package.
diff --git a/meta-test/recipes-app/test-all-any/test-all-any.bb b/meta-test/recipes-app/test-all-any/test-all-any.bb
new file mode 100644
index 00000000..2ff38557
--- /dev/null
+++ b/meta-test/recipes-app/test-all-any/test-all-any.bb
@@ -0,0 +1,13 @@
+# test package generating both arch=all and arch=any binary packages
+DPKG_ARCH = "any"
+SRC_URI = "file://control"
+
+MAINTAINER = "isar-users <isar-...@googlegroups.com>"
+
+inherit dpkg-raw
+
+PROVIDES += "test-all-any-doc-all"
+
+do_prepare_build:append() {
+ cp ${WORKDIR}/control ${S}/debian/
+}
diff --git a/testsuite/citest.py b/testsuite/citest.py
index fc6ec24c..78b3b772 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -413,8 +413,9 @@ class CrossTest(CIBaseTest):
lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'"]

self.init()
+ image_install = 'test-all-deponlycross test-all-any-doc'
self.perform_build_test(targets, lines=lines,
- image_install='test-all-deponlycross')
+ image_install=image_install)

class PrebuiltTest(CIBaseTest):
"""
--
2.53.0

[Jan Kiszka]

All this sounded very nice - but then I counted packages in trixie which
end on -all: 60 right now. I suppose we need a different suffix, maybe
"-archall" or "-arch-all".

BTW, we have the same problem with -compat and 25 name clashes there.
Just -native is fine so far.

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[MOESSBAUER, Felix]

Even there we have a clash:

apt search '\-native$'
mlton-runtime-native/unstable 20241230+git20251029+dfsg-5 amd64
Optimizing compiler for Standard ML - native runtime libraries

How about -archall? This hopefully also makes it more clear, that the -
archall is not part of the Debian package name.

Felix

[Felix Moessbauer]

By that we do not cross-compile the cowsay package. It only consists
of arch=all packages.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

Architecture-all packages often cannot be built in cross mode, as their
dependencies cannot be resolved for the host architecture. Therefore, we
need to dispatch the build of arch=all packages from the target recipe
to the -native variant. This introduces some side effects:

- A mixed arch=(all|any) package needs to be built twice: once in
cross mode for the arch=any parts, and once in -native mode for the
arch=all parts (p2).
- Dependencies on arch=all packages from mixed source packages need to
be modeled so that consumers of a package do not need to be aware of
this distinction (p2).
- The arch=all packages must be available in the apt repository used
during target installation (p3).

Since we now ensure that arch=all packages are only built once, we can
safely revert the repo-split patch. Moreover, we must do so to make the
arch=all packages available during target installation.

PS: As our multiarch code history contains numerous fix tags,
I kindly ask everyone to test this series carefully.

Changes since v2:

- use -archall suffix instead of -all
- improve commit message of p2 to better reflect the changes

Changes since v1:

- model dependencies to arch=all packages via DEPENDS / PROVIDES instead
of an explicit dependency to -native.
- add revert of repo-split patch
- add testcase

Best regards,
Felix Moessbauer

Felix Moessbauer (4):
mark cowsay package as architecture all
sbuild: do not build arch all packages on cross
Revert "Split up isar-apt into distro-arch specific instances"
add test for mixed arch all arch any packages

RECIPE-API-CHANGELOG.md | 17 +++++++++++++++++
meta-isar/recipes-app/cowsay/cowsay_git.bb | 2 ++
.../recipes-app/test-all-any/files/control | 16 ++++++++++++++++
.../recipes-app/test-all-any/test-all-any.bb | 13 +++++++++++++

meta/classes-recipe/dpkg.bbclass | 3 ++-
meta/classes-recipe/multiarch.bbclass | 4 ++++

meta/conf/bitbake.conf | 4 ++--
testsuite/citest.py | 3 ++-

8 files changed, 58 insertions(+), 4 deletions(-)

create mode 100644 meta-test/recipes-app/test-all-any/files/control
create mode 100644 meta-test/recipes-app/test-all-any/test-all-any.bb

--
2.53.0

[Felix Moessbauer]

Architecture all packages often cannot be built in cross mode, as the
dependencies cannot be resolved in the host architecture. This
especially applies to all packages which split their build dependencies
into Build-Depends-Arch and Build-Depends-Indep. We already have logic
in isar to built DPKG_ARCH=all packages non-cross, but this does not
work for packages that generate both arch specific and arch all binary
packages.

While sbuild automatically disables builds of arch-all packages on
cross, we previously explicitly overrode this setting. We now change
this by explicitly setting --no-arch-all flag on cross and --arch-all on
non-cross.

To bitbake depend on arch=all packages, we introduce an interface
similar to the -native: For mixed-architecture packages, the user needs
to extend the PROVIDES by adding -archall to the arch=all packages. This
internally is then dispatched to the -native target, not requiring any
changes on the consumer side (i.e. DEPENDS does not need to add the
-archall).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 17 +++++++++++++++++

meta/classes-recipe/dpkg.bbclass | 3 ++-
meta/classes-recipe/multiarch.bbclass | 4 ++++

3 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 659a03b0..fe026580 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1067,3 +1067,20 @@ Changes in next
The DEPLOY_DIR_IMAGE path and artifacts naming changes introduced in
13cb77dd (included in Isar v1.0) have been reverted. Deployment paths
and artifact names are the same as pre-v1.0.

+
+### No building of arch=all packages when cross building
+
+Architecture all packages often cannot be built in cross mode, as the
+dependencies cannot be resolved in the foreign architecture. This especially
+applies to all packages which split their build dependencies into
+`Build-Depends-Arch` and `Build-Depends-Indep`. We already have logic
+in isar to built `DPKG_ARCH = "all"` packages non-cross, but this does not
+work for packages that generate both arch specific and arch all binary packages.
+
+We now explicitly disable building `arch=all` binary packages during cross
+compilation (this only affects packages that produce both architecture-specific
+and `arch=all` binary packages). Recipes for such mixed packages should append
+`-all` to the `arch=all` binary packages listed in `PROVIDES` (e.g.,

+`PROVIDES="foo-doc-archall"` for an `arch=all` binary package `foo-doc`).
+Consumers can then simply reference the package by its original name (e.g.,
+`foo-doc`).

diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index c1c38184..500aaefe 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -109,8 +109,9 @@ dpkg_runbuild() {

DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)

- sbuild -A -n -c ${SBUILD_CHROOT} --chroot-mode=schroot \
+ sbuild -n -c ${SBUILD_CHROOT} --chroot-mode=schroot \
--host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
+ ${@'--no-arch-all' if 'cross' in isar_deb_build_profiles(d).split() else '--arch-all'} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
--bd-uninstallable-explainer=apt \
--no-apt-update --apt-distupgrade \
diff --git a/meta/classes-recipe/multiarch.bbclass b/meta/classes-recipe/multiarch.bbclass

index cdd35f12..5f84fac2 100644

--- a/meta/classes-recipe/multiarch.bbclass
+++ b/meta/classes-recipe/multiarch.bbclass
@@ -83,6 +83,10 @@ python multiarch_virtclass_handler() {
for v in val.split():
if v.endswith('-compat') or v.endswith('-native'):
multiarch_var.append(v)

+ # dispatch -archall (arch=all) to native variant
+ if v.endswith('-archall'):

+ if suffix == '-native':

+ multiarch_var.append(v[:-len('-archall')])

[Felix Moessbauer]

This reverts commit dfa31ce4c8274b18c09c24fee804be6961c2ffd6.

The split was needed, because both the native and target builds
generated the arch=all packages, leading to conflicts. As we now only
build arch=all packages from the -native variant, each arch=all package
is just build once. By that, we also must use the same repo for both
host and target packages, as otherwise the arch=all packages are not
visible to the target install step.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

This test checks the fixup of the PROVIDES and DEPENDS logic
on mixed architecture all and architecture any packages.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-app/test-all-any/files/control | 16 ++++++++++++++++
.../recipes-app/test-all-any/test-all-any.bb | 13 +++++++++++++

testsuite/citest.py | 3 ++-
3 files changed, 31 insertions(+), 1 deletion(-)

create mode 100644 meta-test/recipes-app/test-all-any/files/control
create mode 100644 meta-test/recipes-app/test-all-any/test-all-any.bb

diff --git a/meta-test/recipes-app/test-all-any/files/control b/meta-test/recipes-app/test-all-any/files/control
new file mode 100644
index 00000000..fc2bae7c
--- /dev/null
+++ b/meta-test/recipes-app/test-all-any/files/control
@@ -0,0 +1,16 @@
+Source: test-all-any
+Section: misc
+Priority: optional
+Maintainer: isar-users <isar-...@googlegroups.com>
+Build-Depends: debhelper-compat (= 10),
+Standards-Version: 4.1.3
+
+Package: test-all-any
+Architecture: any
+Description: Architecture specific test package
+ This is an architecture specific test package.
+
+Package: test-all-any-doc
+Architecture: all
+Description: Architecture independent test package
+ This is an architecture independent test package.
diff --git a/meta-test/recipes-app/test-all-any/test-all-any.bb b/meta-test/recipes-app/test-all-any/test-all-any.bb
new file mode 100644

index 00000000..0f1b0dbd

--- /dev/null
+++ b/meta-test/recipes-app/test-all-any/test-all-any.bb
@@ -0,0 +1,13 @@
+# test package generating both arch=all and arch=any binary packages
+DPKG_ARCH = "any"
+SRC_URI = "file://control"
+
+MAINTAINER = "isar-users <isar-...@googlegroups.com>"
+
+inherit dpkg-raw
+

+PROVIDES += "test-all-any-doc-archall"

+
+do_prepare_build:append() {
+ cp ${WORKDIR}/control ${S}/debian/
+}
diff --git a/testsuite/citest.py b/testsuite/citest.py

index b3866675..51bd7f49 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -412,8 +412,9 @@ class CrossTest(CIBaseTest):

lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'"]

self.init()
+ image_install = 'test-all-deponlycross test-all-any-doc'
self.perform_build_test(targets, lines=lines,
- image_install='test-all-deponlycross')
+ image_install=image_install)

def test_cross_mira_trixie(self):
targets = [
--
2.53.0