[PATCH v3 0/3] Deploy DTBs with separate recipe

[Anton Mikanovich]

Image task "do_copy_boot_files" is used to deploy kernel, initrd and
devicetree binaries to deploy directory.

When different images for the same target are built (e.g., "-base" and
"-debug") in parallel, this causes issues with DTB deployment since
they have no any image_name-specific stuff in the path, unlike kernel
and initrd.

Here we make a separate recipe responsible for the DTB deployment.
DTBs are extracted from the kernel package (provided by distro or
self-built).

The recipe 'dtb-files' is MACHINE-dependent, so different DTB_FILES
may be deployed for different machines. At the same time, all these
'dtb-files-${MACHINE}' targets depend on the same kernel package, so
it is built only once.

The case when different distros (e.g., bullseye/bookworm) are using the
same name for DTB binaries is covered by the patch 1. It also covers
the case when image uses different kernel suffix (e.g., "realtime"
option for the kernel in some downstreams).

Related topics on maillist:

https://groups.google.com/g/isar-users/c/ht3YmKZcJlM
https://groups.google.com/g/isar-users/c/4BRDM02xC40
https://groups.google.com/g/isar-users/c/qAnjahjjfsw
https://groups.google.com/g/isar-users/c/ZMD4XY4dKWQ
https://groups.google.com/g/isar-users/c/PSGU_AcdPZ8
https://groups.google.com/g/isar-users/c/Va0Ue-ISYeA
https://groups.google.com/g/isar-users/c/ixzytHYnTmI

Changes since v2:
- Patch 1: Consider also kernel suffix.
- Patch 2: Use separate testcase for possible dtb clashes.
- Minor cosmetic fixes.

Changes since v1:
- Separate `dtb-files` recipe used instead of linux one
- Included older "[PATCH] Fix do_copy_boot_files error" into the
series ( https://groups.google.com/g/isar-users/c/Va0Ue-ISYeA )

Anton Mikanovich (1):
testsuite: Add test to check dtb deployment

Ilia Skochilov (1):
meta: Fix do_copy_boot_files error for different distros of same
machine

Uladzimir Bely (1):
meta: Move DTB deployment to a separate dtb-files recipe

RECIPE-API-CHANGELOG.md | 15 ++++++++
meta/classes/image.bbclass | 15 +++-----
meta/conf/bitbake.conf | 2 +-
meta/recipes-kernel/dtb-files/dtb-files.bb | 41 ++++++++++++++++++++++
testsuite/citest.py | 23 ++++++++++++
5 files changed, 85 insertions(+), 11 deletions(-)
create mode 100644 meta/recipes-kernel/dtb-files/dtb-files.bb

--
2.34.1

[Anton Mikanovich]

Build several images in parallel to catch all possible conflicts
between deployed DTB files:
- Same distro/machine, different images.
- Same machine, different distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
testsuite/citest.py | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index c99b2fb3..1f1de225 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -277,6 +277,29 @@ class WicTest(CIBaseTest):
)


+class DtbDeployTest(CIBaseTest):
+
+ """
+ Build some targets potentially causing DTB deploy issues.
+ Cover cases:
+ - Same machine, different distros
+ - Same machine/distro, different images
+
+ :avocado: tags=dtbdeploy,full
+ """
+
+ def test_dtb_deploy(self):
+ targets = [
+ 'mc:phyboard-mira-bullseye:isar-image-base',
+ 'mc:phyboard-mira-bullseye:isar-image-ci',
+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ 'mc:phyboard-mira-bookworm:isar-image-ci',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)
+
+
class NoCrossTest(CIBaseTest):

"""
--
2.34.1

[Anton Mikanovich]

From: Ilia Skochilov <iskoc...@ilbers.de>

When building different distros with the same machine (e.g.,
phyboard-mira-bullseye and phyboard-mira-bookworm) it happens that
some files with the same name (e.g, DTB files) are deployed
to the same location and this causes build error.

Use DISTRO- and KERNEL- dependent deploy directory.

Signed-off-by: Ilia Skochilov <iskoc...@ilbers.de>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

RECIPE-API-CHANGELOG.md | 15 +++++++++++++++
meta/conf/bitbake.conf | 2 +-
2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 2930ade9..0a66955b 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -761,3 +761,18 @@ the following variables in your image recipe. For example, to use German, add:
LOCALE_GEN = "de_DE.UTF-8 UTF-8\n"
LOCALE_DEFAULT = "de_DE.UTF-8"
```
+
+### Change DEPLOY_DIR_IMAGE path
+
+Change DEPLOY_DIR_IMAGE from ${DEPLOY_DIR}/images/${MACHINE} to
+${DEPLOY_DIR}/images/${MACHINE}-${DISTRO}-${KERNEL_NAME}. When building
+different distros with the same machine the following error occurs:
+
+do_copy_boot_files: The recipe isar-image-base is trying to install files
+into a shared area when those files already exists. It happens when some
+files have the same names (e.g., dtb files) for different distros.
+
+To prevent this new path is separated also by distro and kernel values.
+
+This change will influence on build artifacts location and should be taken
+into account by downstreams.
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 5ab8ced7..b3ceec44 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -57,7 +57,7 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}"
DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
-DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
+DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}-${DISTRO}-${KERNEL_NAME}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
--
2.34.1

[Anton Mikanovich]

From: Uladzimir Bely <ub...@ilbers.de>

Task do_copy_boot_files was deploying DTB files into the same location
for different images (e.g. -base or -debug). This causes build issue.

Introduce `dtb-files` recipe responsible for the extraction of DTBs
from linux-image package (distro or self-built) and their deployment.

Different images for the same machine now depend on this recipe and
shouldn't cause parallel execution issues.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/image.bbclass | 15 +++-----
meta/recipes-kernel/dtb-files/dtb-files.bb | 41 ++++++++++++++++++++++
2 files changed, 46 insertions(+), 10 deletions(-)
create mode 100644 meta/recipes-kernel/dtb-files/dtb-files.bb

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index bd1b8552..7cb9b9c9 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -335,6 +335,11 @@ INITRD_IMG = "${PP_DEPLOY}/${INITRD_DEPLOY_FILE}"
# only one dtb file supported, pick the first
DTB_IMG = "${PP_DEPLOY}/${@(d.getVar('DTB_FILES').split() or [''])[0]}"

+python() {
+ if d.getVar('DTB_FILES'):
+ d.appendVarFlag("do_copy_boot_files", "depends", "dtb-files-${MACHINE}:do_deploy")
+}
+
do_copy_boot_files[cleandirs] += "${DEPLOYDIR}"
do_copy_boot_files[sstate-inputdirs] = "${DEPLOYDIR}"
do_copy_boot_files[sstate-outputdirs] = "${DEPLOY_DIR_IMAGE}"
@@ -348,16 +353,6 @@ do_copy_boot_files() {
sudo cat "$kernel" > "${DEPLOYDIR}/${KERNEL_IMAGE}"
fi

- for file in ${DTB_FILES}; do
- dtb="$(find '${IMAGE_ROOTFS}/usr/lib' -type f \
- -iwholename '*linux-image-*/'${file} | head -1)"
-
- if [ -z "$dtb" -o ! -e "$dtb" ]; then
- die "${file} not found"
- fi
-
- cp -f "$dtb" "${DEPLOYDIR}/"
- done
}
addtask copy_boot_files before do_rootfs_postprocess after do_rootfs_install

diff --git a/meta/recipes-kernel/dtb-files/dtb-files.bb b/meta/recipes-kernel/dtb-files/dtb-files.bb
new file mode 100644
index 00000000..f64b4845
--- /dev/null
+++ b/meta/recipes-kernel/dtb-files/dtb-files.bb
@@ -0,0 +1,41 @@
+# This software is a part of Isar.
+# Copyright (C) ilbers GmbH, 2025
+#
+# SPDX-License-Identifier: MIT
+
+inherit dpkg-raw
+
+PN:append = "-${MACHINE}"
+
+KERNEL_IMAGE_PKG ??= "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}"
+
+DEPENDS = "${KERNEL_IMAGE_PKG}"
+DEBIAN_BUILD_DEPENDS = "${KERNEL_IMAGE_PKG}"
+
+DPKG_ARCH = "${PACKAGE_ARCH}"
+
+do_prepare_build:append() {
+ for dtb in ${DTB_FILES}; do
+ dir=debian/${PN}/usr/lib/${PN}/$(dirname ${dtb})
+ cat <<EOF >> ${S}/debian/rules
+ mkdir -p ${dir}
+ find /usr/lib/linux-image* -path "*${dtb}" -print -exec cp {} ${dir} \;
+EOF
+ done
+}
+
+DTB_PACKAGE ??= "${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb"
+
+do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
+do_deploy[cleandirs] = "${WORKDIR}/deploy"
+do_deploy() {
+ dpkg --fsys-tarfile ${WORKDIR}/${DTB_PACKAGE} | \
+ tar --wildcards --extract --directory ${WORKDIR}/deploy ./usr/lib/${PN}
+ for dtb in ${DTB_FILES}; do
+ mkdir -p ${DEPLOY_DIR_IMAGE}/$(dirname ${dtb})
+ find ${WORKDIR}/deploy/usr/lib/${PN} -path "*${dtb}" -print \
+ -exec cp {} ${DEPLOY_DIR_IMAGE}/${dtb} \;
+ done
+}
+
+addtask deploy before do_deploy_deb after do_dpkg_build
--
2.34.1