Deploy DTBs and other image artifacts into subdirs to avoid conflicts

[Zhihang Wei]

Image task "do_copy_boot_files" deploys kernel, initrd, and device tree
binaries to the deploy directory.

When different images for the same target (e.g., "-base" and "-debug")
are built in parallel, DTB deployment may fail because DTBs do not
contain image-specific components in their filenames, unlike the kernel
and initrd. As a result, artifacts from different builds may be written
to the same location.

Here we deploy the DTBs and other image artifacts to subdirectories,
with names of distro image encoded to the path.

Note that this results in separate copies of the same artifacts being
deployed for each image build.

Related topics on the mailing list:
https://lists.isar-build.org/isar-users/20250403170701.80774...@siemens.com/T/#u
https://lists.isar-build.org/isar-users/20240705071856.904-...@siemens.com/T/#u
https://lists.isar-build.org/isar-users/c426ad86-5cc5-42e6...@googlegroups.com/T/#u
https://lists.isar-build.org/isar-users/20240416100746.568...@mentor.com/T/#u
https://lists.isar-build.org/isar-users/DB3PR10MB690820CF3D...@DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM/T/#u
https://lists.isar-build.org/isar-users/DB3PR10MB690861E70E...@DB3PR10MB6908.EURPRD10.PROD.OUTLOOK.COM/T/#u
https://lists.isar-build.org/isar-users/20240131144459.23...@ilbers.de/T/#u

Changes since v6:
- Deploy DTBs and other image artifacts into distro- and
image-specific subdirs.
- Shorten image artifact filenames.

Changes since v5:
- Fix isoimage-isohybrid-isar WIC plugin.
- Split testcases.
- Update wording.

Changes since v4:
- Rebase on next.
- Squash latest patch to first one.
- Fix deploy paths in initrd tests.

Changes since v3:
- Rebase on next.
- Fix target deploy dir path for installer.

Changes since v2:
- Patch 1: Consider also kernel suffix.
- Patch 2: Use separate testcase for possible dtb clashes.
- Minor cosmetic fixes.

Changes since v1:
- Separate `dtb-files` recipe used instead of linux one.
- Included older "[PATCH] Fix do_copy_boot_files error" into the
series.

Anton Mikanovich (2):
wic: Obtain real machine name in isoimage source plugin
testsuite: Add testcases to check dtb deployment

Ilia Skochilov (1):
meta: Deploy image build artifacts into distro- and image-specific
subdirs

RECIPE-API-CHANGELOG.md | 63 +++++++++++++++++++
.../installer-add-rootfs.bbclass | 9 +--
meta/classes-recipe/image.bbclass | 11 ++--
.../imagetypes_container.bbclass | 2 +-
meta/classes-recipe/imagetypes_wic.bbclass | 2 +-
meta/conf/bitbake.conf | 3 +-
.../plugins/source/isoimage-isohybrid-isar.py | 6 +-
testsuite/cibase.py | 2 +-
testsuite/citest.py | 40 +++++++++++-
testsuite/start_vm.py | 2 +-
10 files changed, 121 insertions(+), 19 deletions(-)

--
2.39.5

[Zhihang Wei]

From: Anton Mikanovich <ami...@ilbers.de>

The current implementation tries to obtain machine name from the deploy
directory name, which relies on hardcoded path format. Add MACHINE to
the WIC environment and read it in isoimage-isohybrid-isar.py to fix
that.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
meta/classes-recipe/imagetypes_wic.bbclass | 2 +-
.../scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py | 4 +++-
2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/imagetypes_wic.bbclass b/meta/classes-recipe/imagetypes_wic.bbclass
index 6b82add3..d50022ce 100644
--- a/meta/classes-recipe/imagetypes_wic.bbclass
+++ b/meta/classes-recipe/imagetypes_wic.bbclass
@@ -107,7 +107,7 @@ WICVARS += "\
ROOTFS_SIZE STAGING_DATADIR STAGING_DIR STAGING_LIBDIR TARGET_SYS TRANSLATED_TARGET_ARCH"

# Isar specific vars used in our plugins
-WICVARS += "DISTRO DISTRO_ARCH KERNEL_FILE"
+WICVARS += "DISTRO DISTRO_ARCH KERNEL_FILE MACHINE"

python do_rootfs_wicenv () {
wicvars = d.getVar('WICVARS')
diff --git a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
index fb6c6b39..0ed61fff 100644
--- a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
@@ -191,7 +191,9 @@ class IsoImagePlugin(SourcePlugin):
if not image_type:
raise WicError("Couldn't find INITRAMFS_FSTYPES, exiting.")

- machine = os.path.basename(initrd_dir)
+ machine = get_bitbake_var("MACHINE")
+ if not machine:
+ raise WicError("Couldn't find MACHINE, exiting.")

pattern = '%s/%s*%s.%s' % (initrd_dir, image_name, machine, image_type)
pattern = '%s/%s-%s-initrd.%s' % (initrd_dir, image_name, machine, image_type)
--
2.39.5

[Zhihang Wei]

From: Ilia Skochilov <iskoc...@ilbers.de>

When building different distros or images for the same machine (e.g.
phyboard-mira-bullseye and phyboard-mira-bookworm, or isar-image-ci
and isar-image-base), artifacts with identical names (e.g. DTB files)
may be deployed to the same location, resulting file conflicts and
build failures.

Deploy image build artifacts into a subdirectory named
${DISTRO}-${IMAGE_PN} to avoid such collisions.

Since the directory now encodes the distro and image name, remove the
${DISTRO} and ${IMAGE_PN} prefixes from artifact filenames to
keep the names short and clear.

Fixes test_dtb_deploy_distros testcase:

ERROR: mc:phyboard-mira-bookworm:isar-image-base-1.0-r0 do_copy_boot_files: The recipe isar-image-base is trying to install files into a shared area when those files already exist. Those files and their manifest location are:
NOTE: recipe isar-image-base-1.0-r0: task do_copy_boot_files: Failed
build/tmp/deploy/images/phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
(not matched to any task)
Please verify which recipe should provide the above files.

Signed-off-by: Ilia Skochilov <iskoc...@ilbers.de>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
Signed-off-by: Badrikesh Prusty <badrikes...@siemens.com>
Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
Signed-off-by: Zhihang Wei <w...@ilbers.de>
---

RECIPE-API-CHANGELOG.md | 63 +++++++++++++++++++
.../installer-add-rootfs.bbclass | 9 +--
meta/classes-recipe/image.bbclass | 11 ++--
.../imagetypes_container.bbclass | 2 +-

meta/conf/bitbake.conf | 3 +-
.../plugins/source/isoimage-isohybrid-isar.py | 2 +-
testsuite/cibase.py | 2 +-
testsuite/citest.py | 7 ++-
testsuite/start_vm.py | 2 +-
9 files changed, 84 insertions(+), 17 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 0bad8a44..bc7abb37 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -962,3 +962,66 @@ INSTALLER_UNATTENDED_ABORT_ENABLE = "1"
# Optional: set countdown timeout in seconds (default 5)
INSTALLER_UNATTENDED_ABORT_TIMEOUT = "5"
```
+
+### Change DEPLOY_DIR_IMAGE path and artifacts naming
+
+Change DEPLOY_DIR_IMAGE from ${DEPLOY_DIR}/images/${MACHINE} to
+${DEPLOY_DIR}/images/${MACHINE}/${DISTRO}-${IMAGE_PN}.
+
+When building different distros with the same machine the following
+error occurs:
+
+do_copy_boot_files: The recipe isar-image-base is trying to install
+files into a shared area when those files already exists. It happens
+when some files have the same names (e.g., dtb files) for different
+distros.
+
+To prevent such collisions, image artifacts are now deployed into a
+distro- and image-specific subdirectory.
+
+Additionally, artifact filenames have been shortened by removing the
+${DISTRO} and ${IMAGE_PN} prefix, since this information is now
+encoded in the directory path.
+
+Example 1: Build isar-image-base (phyboard-mira, debian-bookworm)
+Under "build/tmp/deploy/images/":
+Before:
+phyboard-mira/imx6q-phytec-mira-rdk-nand.dtb
+phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira-initrd.img
+phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira-vmlinuz
+phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira.dpkg_status
+phyboard-mira/isar-image-base-debian-bookworm-phyboard-mira.ubi
+
+After:
+phyboard-mira/debian-bookworm-isar-image-base/imx6q-phytec-mira-rdk-nand.dtb
+phyboard-mira/debian-bookworm-isar-image-base/initrd.img
+phyboard-mira/debian-bookworm-isar-image-base/vmlinuz
+phyboard-mira/debian-bookworm-isar-image-base/phyboard-mira.dpkg_status
+phyboard-mira/debian-bookworm-isar-image-base/phyboard-mira.ubi
+
+Example 2: Build isar-image-ci (qemuamd64, debian-bookworm)
+Under "build/tmp/deploy/images/":
+Before:
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64-initrd.img
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64-vmlinuz
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.dpkg_status
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.manifest
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic.bmap
+qemuamd64/isar-image-ci-debian-bookworm-qemuamd64.wic.manifest
+
+After:
+qemuamd64/debian-bookworm-isar-image-ci/initrd.img
+qemuamd64/debian-bookworm-isar-image-ci/vmlinuz
+qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.dpkg_status
+qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.manifest
+qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic
+qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic.bmap
+qemuamd64/debian-bookworm-isar-image-ci/qemuamd64.wic.manifest
+
+Artifacts that do not belong to a full image (e.g. isar-image-base,
+isar-image-ci) remain unchanged. For example, a customized initramfs
+built independently is not affected.
+
+This change affects the location and naming of build artifacts and must
+be taken into account by downstream users.
diff --git a/meta-isar/classes-recipe/installer-add-rootfs.bbclass b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
index 2fa551bd..edf93016 100644
--- a/meta-isar/classes-recipe/installer-add-rootfs.bbclass
+++ b/meta-isar/classes-recipe/installer-add-rootfs.bbclass
@@ -13,9 +13,10 @@ INSTALLER_TARGET_IMAGES ??= "${INSTALLER_TARGET_IMAGE}"
INSTALLER_TARGET_MC ??= "installer-target"
INSTALLER_TARGET_DISTRO ??= "${DISTRO}"
INSTALLER_TARGET_MACHINE ??= "${MACHINE}"
-INSTALLER_TARGET_DEPLOY_DIR_IMAGE ??= "${DEPLOY_DIR}/images/${INSTALLER_TARGET_MACHINE}"
+INSTALLER_TARGET_IMAGE ??= "${IMAGE_PN}"
+INSTALLER_TARGET_DEPLOY_DIR_IMAGE ??= "${DEPLOY_DIR}/images/${INSTALLER_TARGET_MACHINE}/${INSTALLER_TARGET_DISTRO}-${INSTALLER_TARGET_IMAGE}"

-IMAGE_DATA_FILE ??= "${INSTALLER_TARGET_IMAGE}-${INSTALLER_TARGET_DISTRO}-${INSTALLER_TARGET_MACHINE}"
+IMAGE_DATA_FILE ??= "${INSTALLER_TARGET_MACHINE}"
IMAGE_DATA_POSTFIX ??= "wic.zst"
IMAGE_DATA_POSTFIX:buster ??= "wic.xz"
IMAGE_DATA_POSTFIX:bullseye ??= "wic.xz"
@@ -29,7 +30,7 @@ def get_installer_sources(d, suffix):
target_machine = d.getVar('INSTALLER_TARGET_MACHINE')
sources = []
for image in installer_target_images:
- image_data = f"{image}-{target_distro}-{target_machine}"
+ image_data = f"{target_machine}"
sources.append(f"{target_deploy_dir}/{image_data}.{suffix}")
return sources

@@ -41,7 +42,7 @@ def get_installer_destinations(d, suffix):
target_machine = d.getVar('INSTALLER_TARGET_MACHINE')
dests = []
for image in installer_target_images:
- image_data = f"{image}-{target_distro}-{target_machine}"
+ image_data = f"{target_machine}"
dests.append(f"/install/{image_data}.{suffix}")
return dests

diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index afe6906e..d4f9e2c0 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -18,8 +18,9 @@ IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"
KERNEL_IMAGE_PKG ??= "${@ ("linux-image-" + d.getVar("KERNEL_NAME")) if d.getVar("KERNEL_NAME") else ""}"
IMAGE_INSTALL += "${KERNEL_IMAGE_PKG}"

-# Name of the image including distro&machine names
-IMAGE_FULLNAME = "${PN}-${DISTRO}-${MACHINE}"
+# Name the image as the machine name only, since the path includes distro name now
+IMAGE_FULLNAME = "${MACHINE}"
+IMAGE_PN = "${PN}"

# Deprecated; this would be set to e.g. "${INITRAMFS_RECIPE}-${DISTRO}-${MACHINE}-initrd.img"
INITRD_IMAGE ?= ""
@@ -29,7 +30,7 @@ INITRD_IMAGE ?= ""
IMAGE_INITRD ?= ""

# Name of the deployed initrd image
-INITRD_DEPLOY_FILE = "${@ d.getVar('IMAGE_INITRD') or '${PN}' }-${DISTRO}-${MACHINE}-initrd.img"
+INITRD_DEPLOY_FILE = "initrd.img"

# Make sure dependent initramfs recipe is built
do_image[depends] += "${@ '${IMAGE_INITRD}:do_build' if '${IMAGE_INITRD}' else '' }"
@@ -52,7 +53,7 @@ python() {
ROOTFS_FEATURES += "${@ 'generate-initrd' if (d.getVar('INITRD_IMAGE') == '' and d.getVar('IMAGE_INITRD') == '') else ''}"

# This variable is used by wic and start_vm
-KERNEL_IMAGE ?= "${IMAGE_FULLNAME}-${KERNEL_FILE}"
+KERNEL_IMAGE ?= "${KERNEL_FILE}"

# This defines the deployed dtbs for reuse by imagers
DTB_FILES ?= ""
@@ -108,7 +109,7 @@ ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTAL
ROOTFS_VARDEPS += "IMAGE_INSTALL"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
ROOTFS_DPKGSTATUS_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"
-ROOTFS_PACKAGE_SUFFIX ?= "${PN}-${DISTRO}-${MACHINE}"
+ROOTFS_PACKAGE_SUFFIX ?= "${MACHINE}"

CACHE_DEB_SRC = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', '1', '0', d)}"
python () {
diff --git a/meta/classes-recipe/imagetypes_container.bbclass b/meta/classes-recipe/imagetypes_container.bbclass
index ea15decf..1c5386a8 100644
--- a/meta/classes-recipe/imagetypes_container.bbclass
+++ b/meta/classes-recipe/imagetypes_container.bbclass
@@ -9,7 +9,7 @@
CONTAINER_TYPES = "oci-archive docker-archive docker-daemon containers-storage"
USING_CONTAINER = "${@bb.utils.contains_any('IMAGE_BASETYPES', d.getVar('CONTAINER_TYPES').split(), '1', '0', d)}"

-CONTAINER_IMAGE_NAME ?= "${PN}-${DISTRO}-${DISTRO_ARCH}"
+CONTAINER_IMAGE_NAME ?= "container-${DISTRO_ARCH}"
CONTAINER_IMAGE_TAG ?= "${PV}-${PR}"
CONTAINER_IMAGE_CMD ?= "/bin/dash"
CONTAINER_IMAGE_ENTRYPOINT ?= ""
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0f84e715..7c544e5d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -57,7 +57,8 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}"
DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
-DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
+IMAGE_PN ?= ""
+DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}${@('/%s-%s' % (d.getVar('DISTRO'), d.getVar('IMAGE_PN'))) if d.getVar('IMAGE_PN') != '' else ''}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
diff --git a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
index 0ed61fff..eaef9c79 100644
--- a/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
+++ b/meta/scripts/lib/wic/plugins/source/isoimage-isohybrid-isar.py
@@ -196,7 +196,7 @@ class IsoImagePlugin(SourcePlugin):

raise WicError("Couldn't find MACHINE, exiting.")

pattern = '%s/%s*%s.%s' % (initrd_dir, image_name, machine, image_type)

- pattern = '%s/%s-%s-initrd.%s' % (initrd_dir, image_name, machine, image_type)
+ pattern = '%s/initrd.%s' % (initrd_dir, image_type)
files = glob.glob(pattern)
if files:
initrd = files[0]
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index fd6a3df9..fac2ab75 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -44,7 +44,7 @@ class CIBaseTest(CIBuilder):
self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs)
self.bitbake(targets, **kwargs)

- wic_path = f"{self.build_dir}/tmp/deploy/images/*/*.wic.p1"
+ wic_path = f"{self.build_dir}/tmp/deploy/images/*/*/*.wic.p1"
partition_files = set(glob.glob(wic_path))
if wic_deploy_parts and len(partition_files) == 0:
self.fail("Found raw wic partitions in DEPLOY_DIR")
diff --git a/testsuite/citest.py b/testsuite/citest.py
index ca701422..f36da786 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -496,8 +496,9 @@ class InitRdBaseTest(CIBaseTest):
super().init()
self.deploy_dir = os.path.join(self.build_dir, 'tmp', 'deploy')

- def deploy_dir_image(self, machine):
- return os.path.join(self.deploy_dir, 'images', machine)
+ def deploy_dir_image(self, mc, image):
+ multiconfig = f"{mc}:{image}"
+ return CIUtils.getVars('DEPLOY_DIR_IMAGE', target=multiconfig)

def dracut_in_image(self, targets):
machine = 'qemuamd64'
@@ -526,7 +527,7 @@ class InitRdBaseTest(CIBaseTest):
bb_should_fail=False):
mc = f'mc:{machine}-{distro.removeprefix("debian-")}'
initrd_image = f'{initrd}-{distro}-{machine}-initrd.img'
- initrd_path = os.path.join(self.deploy_dir_image(machine), initrd_image)
+ initrd_path = os.path.join(self.deploy_dir_image(mc, initrd), initrd_image)

# cleansstate if the initrd image was already built/deployed to verify
# that a new build does result in the image being deployed
diff --git a/testsuite/start_vm.py b/testsuite/start_vm.py
index 57d169df..a37ffe82 100755
--- a/testsuite/start_vm.py
+++ b/testsuite/start_vm.py
@@ -50,7 +50,7 @@ def format_qemu_cmdline(
image_type = image_fstypes.split()[0]
base = 'ubuntu' if distro in ['jammy', 'focal', 'noble'] else 'debian'

- rootfs_image = f"{image}-{base}-{distro}-qemu{arch}.{image_type}"
+ rootfs_image = f"qemu{arch}.{image_type}"

if image_type == 'ext4':
kernel_image = deploy_dir_image + '/' + kernel_image
--
2.39.5