[PATCH v2 0/3] test and fix transitive multiarch dependencies

[Andreas Naumann]

This is v2 of the previous "Prevent adding unneeded package variants to
recursive dependency chain" set.

Unfortunately even with [1] applied, we still see packages in transitive
dependency chains being built for the wrong architecture (in addition to
the correct one).

The original commits which introduce the undesired behaviour have been
identified to be [2] and [3].

[1] b03093bf (propagate distro-specific dependencies of arch all packages)
[2] 5a7c2f70 (handle DPKG_ARCH=all case for transitive deps)
[3] 2ca3a7e5 (dpkg-source: Build source package only once)

Changes v2:
- change title and rebase
- consolidate testcases
- slightly rephrase commit message in rootfs fix
- add fix for second issue where native package variants are built
unnecessarily

Andreas Naumann (3):
test: Add test to check correct multiarch dependency propagation
rootfs: Do not recursively build unneeded packages
multiarch: Replace divertion of deploy task for "all" packages

.../recipes-app/test-all-depnocross/files/rules | 11 +++++++++++
.../test-all-depnocross/test-all-depnocross.bb | 11 +++++++++++
.../recipes-app/test-all-deponlycross/files/rules | 11 +++++++++++
.../test-all-deponlycross.bb | 11 +++++++++++
.../recipes-app/test-any-nocross/files/rules | 11 +++++++++++
.../test-any-nocross/test-any-nocross.bb | 11 +++++++++++
.../recipes-app/test-any-onlycross/files/rules | 11 +++++++++++
.../test-any-onlycross/test-any-onlycross.bb | 13 +++++++++++++
meta/classes/multiarch.bbclass | 15 ++-------------
meta/classes/rootfs.bbclass | 3 ++-
testsuite/citest.py | 11 +++++++++++
11 files changed, 105 insertions(+), 14 deletions(-)
create mode 100644 meta-test/recipes-app/test-all-depnocross/files/rules
create mode 100644 meta-test/recipes-app/test-all-depnocross/test-all-depnocross.bb
create mode 100644 meta-test/recipes-app/test-all-deponlycross/files/rules
create mode 100644 meta-test/recipes-app/test-all-deponlycross/test-all-deponlycross.bb
create mode 100644 meta-test/recipes-app/test-any-nocross/files/rules
create mode 100644 meta-test/recipes-app/test-any-nocross/test-any-nocross.bb
create mode 100644 meta-test/recipes-app/test-any-onlycross/files/rules
create mode 100644 meta-test/recipes-app/test-any-onlycross/test-any-onlycross.bb

--
2.43.0

[Andreas Naumann]

Create two transitive dependency chains with simple dpkg-raw packages as
follows:
image -> all package -> any package
imager -> all package as -native -> any package as -native

Assert that
- the all package is always build for the host architecture
- the dependent any package in the cross chain is built for target arch only
- the dependent any package in the native chain is built for host arch only
by detecting any undesired host/target architecture combination in the
override_dh_auto_configure function and exiting with an error.

Signed-off-by: Andreas Naumann <anau...@emlix.com>
---

.../recipes-app/test-all-depnocross/files/rules | 11 +++++++++++
.../test-all-depnocross/test-all-depnocross.bb | 11 +++++++++++
.../recipes-app/test-all-deponlycross/files/rules | 11 +++++++++++

.../test-all-deponlycross/test-all-deponlycross.bb | 11 +++++++++++
meta-test/recipes-app/test-any-nocross/files/rules | 11 +++++++++++

.../test-any-nocross/test-any-nocross.bb | 11 +++++++++++
.../recipes-app/test-any-onlycross/files/rules | 11 +++++++++++
.../test-any-onlycross/test-any-onlycross.bb | 13 +++++++++++++

testsuite/citest.py | 11 +++++++++++
9 files changed, 101 insertions(+)

create mode 100644 meta-test/recipes-app/test-all-depnocross/files/rules
create mode 100644 meta-test/recipes-app/test-all-depnocross/test-all-depnocross.bb
create mode 100644 meta-test/recipes-app/test-all-deponlycross/files/rules
create mode 100644 meta-test/recipes-app/test-all-deponlycross/test-all-deponlycross.bb
create mode 100644 meta-test/recipes-app/test-any-nocross/files/rules
create mode 100644 meta-test/recipes-app/test-any-nocross/test-any-nocross.bb
create mode 100644 meta-test/recipes-app/test-any-onlycross/files/rules
create mode 100644 meta-test/recipes-app/test-any-onlycross/test-any-onlycross.bb

diff --git a/meta-test/recipes-app/test-all-depnocross/files/rules b/meta-test/recipes-app/test-all-depnocross/files/rules
new file mode 100644
index 00000000..6640cf23
--- /dev/null
+++ b/meta-test/recipes-app/test-all-depnocross/files/rules
@@ -0,0 +1,11 @@
+#!/usr/bin/make -f
+%:
+ dh \$@
+
+# Detect cross-compilation and fail if so
+override_dh_auto_configure:
+ @if [ "$(DEB_BUILD_ARCH)" != "$(DEB_HOST_ARCH)" ]; then \
+ echo "Cross-compilation detected! This is an \"all\" package."; \
+ exit 1; \
+ fi
+ dh_auto_configure
diff --git a/meta-test/recipes-app/test-all-depnocross/test-all-depnocross.bb b/meta-test/recipes-app/test-all-depnocross/test-all-depnocross.bb
new file mode 100644
index 00000000..86ae3847
--- /dev/null
+++ b/meta-test/recipes-app/test-all-depnocross/test-all-depnocross.bb
@@ -0,0 +1,11 @@
+# Test package using dpkg-raw
+
+SRC_URI = "file://rules"
+
+inherit dpkg-raw
+
+DEPENDS = "test-any-nocross"
+
+do_install() {
+ bbnote "Test \"all\" package which depends on an any package."
+}
diff --git a/meta-test/recipes-app/test-all-deponlycross/files/rules b/meta-test/recipes-app/test-all-deponlycross/files/rules
new file mode 100644
index 00000000..6640cf23
--- /dev/null
+++ b/meta-test/recipes-app/test-all-deponlycross/files/rules
@@ -0,0 +1,11 @@
+#!/usr/bin/make -f
+%:
+ dh \$@
+
+# Detect cross-compilation and fail if so
+override_dh_auto_configure:
+ @if [ "$(DEB_BUILD_ARCH)" != "$(DEB_HOST_ARCH)" ]; then \
+ echo "Cross-compilation detected! This is an \"all\" package."; \
+ exit 1; \
+ fi
+ dh_auto_configure
diff --git a/meta-test/recipes-app/test-all-deponlycross/test-all-deponlycross.bb b/meta-test/recipes-app/test-all-deponlycross/test-all-deponlycross.bb
new file mode 100644
index 00000000..6af762b9
--- /dev/null
+++ b/meta-test/recipes-app/test-all-deponlycross/test-all-deponlycross.bb
@@ -0,0 +1,11 @@
+# Test all package using dpkg-raw
+
+SRC_URI = "file://rules"
+
+inherit dpkg-raw
+
+DEPENDS = "test-any-onlycross"
+
+do_install() {
+ bbnote "Test \"all\" package which depends on an any package."
+}
diff --git a/meta-test/recipes-app/test-any-nocross/files/rules b/meta-test/recipes-app/test-any-nocross/files/rules
new file mode 100644
index 00000000..74d905b6
--- /dev/null
+++ b/meta-test/recipes-app/test-any-nocross/files/rules
@@ -0,0 +1,11 @@
+#!/usr/bin/make -f
+%:
+ dh \$@
+
+# Detect cross-compilation and fail if so
+override_dh_auto_configure:
+ @if [ "$(DEB_BUILD_ARCH)" != "$(DEB_HOST_ARCH)" ]; then \
+ echo "Cross-compilation detected!"; \
+ exit 1; \
+ fi
+ dh_auto_configure
diff --git a/meta-test/recipes-app/test-any-nocross/test-any-nocross.bb b/meta-test/recipes-app/test-any-nocross/test-any-nocross.bb
new file mode 100644
index 00000000..6d0321ae
--- /dev/null
+++ b/meta-test/recipes-app/test-any-nocross/test-any-nocross.bb
@@ -0,0 +1,11 @@
+# Test package using dpkg-raw, which breaks when trying to cross
+# compile
+
+SRC_URI = "file://rules"
+
+inherit dpkg-raw
+DPKG_ARCH = "any"
+
+do_install() {
+ bbnote "Test \"any\" package which fails crosscompile."
+}
diff --git a/meta-test/recipes-app/test-any-onlycross/files/rules b/meta-test/recipes-app/test-any-onlycross/files/rules
new file mode 100644
index 00000000..02031f1a
--- /dev/null
+++ b/meta-test/recipes-app/test-any-onlycross/files/rules
@@ -0,0 +1,11 @@
+#!/usr/bin/make -f
+%:
+ dh \$@
+
+# Detect native compilation and fail if so
+override_dh_auto_configure:
+ if [ "$(DEB_BUILD_ARCH)" = "$(DEB_HOST_ARCH)" ]; then \
+ echo "Native compilation detected!"; \
+ exit 1; \
+ fi
+ dh_auto_configure
diff --git a/meta-test/recipes-app/test-any-onlycross/test-any-onlycross.bb b/meta-test/recipes-app/test-any-onlycross/test-any-onlycross.bb
new file mode 100644
index 00000000..5b2eace6
--- /dev/null
+++ b/meta-test/recipes-app/test-any-onlycross/test-any-onlycross.bb
@@ -0,0 +1,13 @@
+# Test package using dpkg-raw, which breaks when trying to cross
+# compile
+
+#MAINTAINER = "Your name here <y...@domain.com>"
+
+SRC_URI = "file://rules"
+
+inherit dpkg-raw
+DPKG_ARCH = "any"
+
+do_install() {
+ bbnote "Test \"any\" package which fails native compile."
+}
diff --git a/testsuite/citest.py b/testsuite/citest.py
index f944ee4a..6f8e03cf 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -255,6 +255,17 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets)

+ def test_cross_dependencies(self):
+ targets = [
+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ ]
+
+ lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'",
+ f"IMAGE_INSTALL:append = ' test-all-deponlycross'",
+ ]
+
+ self.init()
+ self.perform_build_test(targets, lines=lines)

class KernelTests(CIBaseTest):
"""
--
2.43.0

[Andreas Naumann]

Using recrdeptask has the side effect that every package of which a task
appears in the dependency chain, will be built and deployed, even if just a
subset of tasks was needed to fulfil an inter-task dependency.

For coupled packages like the native/compat ones, which can share part of
their tasks, see 2ca3a7e dpkg-source: Build source package only once,
this leads to a full build of the base package, while e.g. only the native
part needed to be produced.

Refrain from doing so by going back to using deptask which only adds the deploy
task of the direct dependencies and rely on correct cache and inter-task
settings of the used classes/recipes.
Add rdeptask handling to allow for runtime dependency settings.

This is a partial revert of 7c7628e rootfs: recursively depend on packages.

This fixes build failures with custom "any" packages which dont support
crosscompiling (-native only), improves performance as no unneeded compiling is
being done.

Signed-off-by: Andreas Naumann <anau...@emlix.com>
---

meta/classes/rootfs.bbclass | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index e1179e61..e12e1b8a 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -387,7 +387,8 @@ do_rootfs_install[root_cleandirs] = "${ROOTFSDIR}"
do_rootfs_install[vardeps] += "${ROOTFS_CONFIGURE_COMMAND} ${ROOTFS_INSTALL_COMMAND}"
do_rootfs_install[vardepsexclude] += "IMAGE_ROOTFS"
do_rootfs_install[depends] = "bootstrap-${@'target' if d.getVar('ROOTFS_ARCH') == d.getVar('DISTRO_ARCH') else 'host'}:do_build"
-do_rootfs_install[recrdeptask] = "do_deploy_deb"
+do_rootfs_install[deptask] = "do_deploy_deb"
+do_rootfs_install[rdeptask] = "do_deploy_deb"
do_rootfs_install[network] = "${TASK_USE_SUDO}"
python do_rootfs_install() {
configure_cmds = (d.getVar("ROOTFS_CONFIGURE_COMMAND") or "").split()
--
2.43.0

[Andreas Naumann]

Diverting the do_deploy_deb function of the base package to the native variant
had the sideeffect that the dependencies of the native variant were added to
the dependency chain even if the package was originally depended on in its non-
native form.

Fix this by not manipulating the dependencies at all, but just build the base
package for the host architecture. That way sub-dependencies are preserved for
all uses cases.

Fixes: 5a7c2f7004 ("handle DPKG_ARCH=all case for transitive deps")

Signed-off-by: Andreas Naumann <anau...@emlix.com>
---

meta/classes/multiarch.bbclass | 15 ++-------------
1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/meta/classes/multiarch.bbclass b/meta/classes/multiarch.bbclass
index 74b8f5b8..851d1aa1 100644
--- a/meta/classes/multiarch.bbclass
+++ b/meta/classes/multiarch.bbclass
@@ -31,8 +31,6 @@ python() {
# build native separately only when it differs from the target variant
# We must not short-circuit for DPKG_ARCH=all packages, as they might
# have transitive dependencies which need to be built for -native.
- # This special handling for DPKG_ARCH=all packages is left to the
- # multiarch_virtclass_handler
if archDiffers:
d.appendVar('BBCLASSEXTEND', ' native')
else:
@@ -103,17 +101,8 @@ python multiarch_virtclass_handler() {
fixup_pn_in_vars(e.data)
fixup_depends('-native', e.data)
elif archIsAll and archDiffers:
- # Arch=all packages might build depend on other arch=all packages,
- # hence we need to correctly model the dependency chain.
- # We implement this by dispatching the non-native variant to the -native
- # variant by adding a dependency. We further empty the non-native
- # do_deploy_dep task and clear the internal dependency chain, but keep
- # other attached variables like RDEPENDS to preserve the dependency chain.
- e.data.setVar('do_deploy_deb', '')
- # clear internal dependencies (e.g. to do_dpkg_build)
- e.data.setVarFlag('do_deploy_deb', 'deps', [])
- # dispatch to native variant
- e.data.setVarFlag('do_deploy_deb', 'depends', f'{pn}-native:do_deploy_deb')
+ # Speed up Arch=all package build
+ e.data.setVar('PACKAGE_ARCH', d.getVar('HOST_ARCH'))
}
addhandler multiarch_virtclass_handler
multiarch_virtclass_handler[eventmask] = "bb.event.RecipePreFinalise"
--
2.43.0

[Zhihang Wei]

Applied to next, thanks for adding a new test case.

Best regards,
Zhihang