[PATCH v3 0/1] use xz and gzip on host (outside chroot)

[Felix Moessbauer]

Changes since v1/v2:

instead of switching the compression logic,
this patch adds support to run the whole imager
in the buildchroot-host.

Felix Moessbauer (1):
run imager in buildchroot-host on cross

doc/user_manual.md | 11 +++++++++++
meta/classes/image-tools-extension.bbclass | 3 ---
2 files changed, 11 insertions(+), 3 deletions(-)

--
2.30.2

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

doc/user_manual.md | 11 +++++++++++
meta/classes/image-tools-extension.bbclass | 3 ---
2 files changed, 11 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 5b2387d8..f38a133e 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -986,6 +986,17 @@ built for the compat arch need to be tagged individually by setting
`PACKAGE_ARCH = "${COMPAT_DISTRO_ARCH}"` in the package recipe. Non-tagged
packages will continue to be built for the primary target architecture.

+### Cross Support for Imagers
+
+If `ISAR_CROSS_COMPILE = "1"`, the imager and optional compression tasks
+run in the host buildchroot instead of the target buildchroot.
+This gives a significant speedup when compressing the generated image,
+as the compression is not emulated.
+
+In case your setup does not support cross-imaging, you can disable this
+just for the particular image by adding `ISAR_CROSS_COMPILE = "0"` to your
+image recipe.
+

## Examining and debugging package generation inside their buildchroot

diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index b9968139..e13d4a3f 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -5,9 +5,6 @@
#
# This file extends the image.bbclass to supply tools for futher imager functions

-# Imager are expected to run natively, thus will use the target buildchroot.
-ISAR_CROSS_COMPILE = "0"
-
inherit buildchroot

IMAGER_INSTALL ??= ""
--
2.30.2

[Henning Schild]

I want a testcase for this where a whole wic gz/xz chain runs in CI with
a CROSS=1.

Henning

Am Tue, 13 Sep 2022 15:38:39 +0200
schrieb Felix Moessbauer <felix.mo...@siemens.com>:

[Moessbauer, Felix]

> -----Original Message-----
> From: Schild, Henning (T CED SES-DE) <henning...@siemens.com>
> Sent: Wednesday, September 14, 2022 10:50 AM
> To: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>
> Cc: isar-...@googlegroups.com; Kiszka, Jan (T CED)
> <jan.k...@siemens.com>; Schmidt, Adriaan (T CED SES-DE)
> <adriaan...@siemens.com>; ami...@ilbers.de; i...@ilbers.de
> Subject: Re: [PATCH v3 1/1] run imager in buildchroot-host on cross
>
> I want a testcase for this where a whole wic gz/xz chain runs in CI with a
> CROSS=1.

The WIC logic is already tested with CROSS=1 as the mc:qemuarm64-bullseye:isar-image-base is built in CI.
What is missing is the compression logic with CROSS=1.
Currently the only test that tests the compression logic at is the "container-amd64" machine.
However, these are not cross.

When trying to add the `wic.xz` to the qemuarm64 machine IMAGE_FSTYPES, we ran out of disk space on many systems as the generated images (and partitions) are rather big.
I'm going to send that as an individual patch so the maintainers can decide if that is really worth it.

Felix

[Henning Schild]

Am Wed, 14 Sep 2022 15:46:22 +0200
schrieb "Moessbauer, Felix (T CED SES-DE)"
<felix.mo...@siemens.com>:

> > -----Original Message-----
> > From: Schild, Henning (T CED SES-DE) <henning...@siemens.com>
> > Sent: Wednesday, September 14, 2022 10:50 AM
> > To: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>
> > Cc: isar-...@googlegroups.com; Kiszka, Jan (T CED)
> > <jan.k...@siemens.com>; Schmidt, Adriaan (T CED SES-DE)
> > <adriaan...@siemens.com>; ami...@ilbers.de; i...@ilbers.de
> > Subject: Re: [PATCH v3 1/1] run imager in buildchroot-host on cross
> >
> > I want a testcase for this where a whole wic gz/xz chain runs in CI
> > with a CROSS=1.
>
> The WIC logic is already tested with CROSS=1 as the
> mc:qemuarm64-bullseye:isar-image-base is built in CI. What is missing
> is the compression logic with CROSS=1. Currently the only test that
> tests the compression logic at is the "container-amd64" machine.
> However, these are not cross.
>
> When trying to add the `wic.xz` to the qemuarm64 machine
> IMAGE_FSTYPES, we ran out of disk space on many systems as the
> generated images (and partitions) are rather big. I'm going to send
> that as an individual patch so the maintainers can decide if that is
> really worth it.

Yes the test should be its own patch anyhow. It will be worth it ... HW
is dead cheap compared to what bad test coverage can cause.

Henning

[Felix Moessbauer]

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series
but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.

Felix Moessbauer (2):

run imager in buildchroot-host on cross

add wic.xz image to qemuarm64-bookworm

doc/user_manual.md | 11 +++++++++++
meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
meta/classes/image-tools-extension.bbclass | 3 ---
testsuite/citest.py | 1 +
4 files changed, 15 insertions(+), 3 deletions(-)

--
2.30.2

[Felix Moessbauer]

--
2.30.2

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
testsuite/citest.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@

MACHINE ?= "qemuarm64"
DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..630b4a65 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -72,6 +72,7 @@ class CrossTest(CIBaseTest):
'mc:qemuarm-buster:isar-image-base',
'mc:qemuarm-bullseye:isar-image-base',
'mc:qemuarm64-bullseye:isar-image-base',
+ 'mc:qemuarm64-bookworm:isar-image-base',
'mc:de0-nano-soc-bullseye:isar-image-base',
'mc:stm32mp15x-buster:isar-image-base'
]
--
2.30.2

[Anton Mikanovich]

14.09.2022 23:21, Felix Moessbauer wrote:
> When globally enabling ISAR_CROSS_COMPILE, the imager and compression
> tasks run in the host buildchroot as well (instead of the target
> buildchroot).
>
> Similar to the cross support for DPKG recipes, this can be toggled
> on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.
>
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

Hello Felix,

This actually brakes SDK building for amd64 stretch targets because
there is no
crossbuild-essential-amd64 there.

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <ami...@ilbers.de>
> Sent: Thursday, September 15, 2022 11:14 AM
> To: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>; isar-
> us...@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning...@siemens.com>; Schmidt,
> Adriaan (T CED SES-DE) <adriaan...@siemens.com>; i...@ilbers.de; Kiszka,
> Jan (T CED) <jan.k...@siemens.com>

Hi,

Interesting finding.
Are there any CI tests that catch this? At least I cannot find a CROSS test that builds the SDK.
Apart from that, Debian stretch is EOL since June 22.

I'm fine with keeping the stretch targets, but then we could simply disable CROSS by setting
ISAR_CROSS_COMPILE_debian-stretch = "0" in the image.bbclass.

If you could point me to a target for testing, I'm happy to add that.

Best regards,
Felix

[Anton Mikanovich]

15.09.2022 12:35, Moessbauer, Felix wrote:
> Hi,
>
> Interesting finding.
> Are there any CI tests that catch this? At least I cannot find a CROSS test that builds the SDK.
> Apart from that, Debian stretch is EOL since June 22.
>
> I'm fine with keeping the stretch targets, but then we could simply disable CROSS by setting
> ISAR_CROSS_COMPILE_debian-stretch = "0" in the image.bbclass.
>
> If you could point me to a target for testing, I'm happy to add that.
>
> Best regards,
> Felix
>

At least test_container_sdk test of our fast CI has do_populate_sdk
target for
stretch. And we still need stretch support for downstreams, so adding
ISAR_CROSS_COMPILE override looks reasonable.

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <ami...@ilbers.de>
> Sent: Thursday, September 15, 2022 12:46 PM
> To: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>; isar-
> us...@googlegroups.com
> Cc: Schild, Henning (T CED SES-DE) <henning...@siemens.com>; Schmidt,
> Adriaan (T CED SES-DE) <adriaan...@siemens.com>; i...@ilbers.de; Kiszka,
> Jan (T CED) <jan.k...@siemens.com>
> Subject: Re: [PATCH v4 1/2] run imager in buildchroot-host on cross
>

Ok, will do that. But then using ` ISAR_CROSS_COMPILE_amd64_debian-stretch = "0"`

One last question before implementing this:
Could it be that this is in fact just a badly written test?
While cross-building for distro-arch == host-arch is technically possible (and uses a different toolchain), it does not look like it provides a reasonable value.
Could it be, that we better simply add ` perform_build_test(...,cross=False)` to the ContainerSdkTest as this is done for all other tests?

Best regards,
Felix

[Henning Schild]

Am Thu, 15 Sep 2022 13:45:35 +0300
schrieb Anton Mikanovich <ami...@ilbers.de>:

Please take bullseye, not stretch. When we drop stretch we would
silently drop the cross/wic/compression test.

And in the long run we need a decoupling from testing multiconfigs to
example multiconfigs. Right now it is a little mixed so we might see
people copying testing bits because they think they are part of the
example, or testing being dropped by dropping a multiconfig that
contained some.

Henning

[Henning Schild]

Am Thu, 15 Sep 2022 13:45:35 +0300
schrieb Anton Mikanovich <ami...@ilbers.de>:

And that bookworm can not be crossed with SDK ... is that written down
somewhere? Maybe that should be a comment in those tests.

Henning

[Anton Mikanovich]

15.09.2022 14:06, Henning Schild wrote:
> Please take bullseye, not stretch. When we drop stretch we would
> silently drop the cross/wic/compression test.

Bullseye don't have any issues with SDK building, when stretch really do.
This is what Felix just asked and this is what my reply about.

[Anton Mikanovich]

15.09.2022 14:08, Henning Schild wrote:
> And that bookworm can not be crossed with SDK ... is that written down
> somewhere? Maybe that should be a comment in those tests.

Why bookworm can't be crossed with SDK?

[Moessbauer, Felix]

> -----Original Message-----
> From: Anton Mikanovich <ami...@ilbers.de>
> Sent: Thursday, September 15, 2022 5:45 PM
> To: Schild, Henning (T CED SES-DE) <henning...@siemens.com>
> Cc: Moessbauer, Felix (T CED SES-DE) <felix.mo...@siemens.com>; isar-
> us...@googlegroups.com; Schmidt, Adriaan (T CED SES-DE)
> <adriaan...@siemens.com>; i...@ilbers.de; Kiszka, Jan (T CED)
> <jan.k...@siemens.com>
> Subject: Re: [PATCH v4 1/2] run imager in buildchroot-host on cross
>

I just checked and for me it worked without any issues.

Still, nobody answered my question about the distro-arch==host-arch question, huh!
Before sending a v5, I'll wait for a clarification.

Felix

[Anton Mikanovich]

15.09.2022 20:08, Moessbauer, Felix wrote:
> I just checked and for me it worked without any issues.
>
> Still, nobody answered my question about the distro-arch==host-arch question, huh!
> Before sending a v5, I'll wait for a clarification.

Running amd64 targets with enabled cross compile really don't have much
sense,
but we have some logic in buildchroot.bbclass which makes Isar to use host
buildchroot in this case (and no such a logic for SDK - is that a bug?).

[Felix Moessbauer]

Changes since v4:

- add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
Note, that the default set to disabled is important.
This fixes the huge disk consumption when running the CI with WIC tests
- add test for WIC_DEPLOY_PARTITIONS logic (p4)
- add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)

Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches
which are only sent as part of this series to fix the disk space blow up in CI.

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.

Felix Moessbauer (5):
do not crossbuild SDK for host-arch eq dist-arch

run imager in buildchroot-host on cross

add option to control deploy of raw wic partitions
add test for wic partition deploy logic

add wic.xz image to qemuarm64-bookworm

RECIPE-API-CHANGELOG.md | 6 +++++
doc/user_manual.md | 11 +++++++++
meta-isar/conf/local.conf.sample | 3 +++
.../conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
meta/classes/image-tools-extension.bbclass | 3 ---
meta/classes/imagetypes_wic.bbclass | 5 ++++
meta/classes/sdk.bbclass | 14 +++++++++--
testsuite/cibase.py | 10 ++++++++
testsuite/cibuilder.py | 5 +++-
testsuite/citest.py | 24 +++++++++++++++++++
10 files changed, 78 insertions(+), 6 deletions(-)

--
2.30.2

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>

[Felix Moessbauer]

This patch adds logic to detect when ISAR_CROSS_COMPILE
is set, but host-arch equals dist-arch.
In this case, we do not add the crossbuild-essential package,
but instead add the build-essential one.

To implement this, we transfer the existing logic into a
anonymous python function to make it better readable.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/classes/sdk.bbclass | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sdk.bbclass b/meta/classes/sdk.bbclass
index 32ee2de7..f42a2ba1 100644
--- a/meta/classes/sdk.bbclass
+++ b/meta/classes/sdk.bbclass
@@ -41,8 +41,18 @@ SDK_PREINSTALL += " \
devscripts \
equivs"

-TOOLCHAIN = "${@'crossbuild-essential-${DISTRO_ARCH}' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' else 'build-essential'}"
-TOOLCHAIN_append_compat-arch = " crossbuild-essential-${COMPAT_DISTRO_ARCH}"
+# Choose the correct toolchain: cross or native
+python __anonymous() {
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch:
+ toolchain = "build-essential"
+ else:
+ toolchain = "crossbuild-essential-" + distro_arch
+ if d.getVar('ISAR_ENABLE_COMPAT_ARCH', True) == "1":
+ toolchain += " crossbuild-essential-" + d.getVar('COMPAT_DISTRO_ARCH')
+ d.setVar('TOOLCHAIN', toolchain)
+}

# rootfs/image overrides for the SDK
ROOTFS_ARCH_class-sdk = "${HOST_ARCH}"
--
2.30.2

[Felix Moessbauer]

This patch adds the WIC_DEPLOY_PARTITIONS option to control
if the raw partition files are deployed by wic.

We also set the default to not deploy to be closer to OE and to
optimize for the average case.
In addition, this significantly reduces the disk size required to
run the CI.

An API changelog entry is added.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 6 ++++++
meta-isar/conf/local.conf.sample | 3 +++
meta/classes/imagetypes_wic.bbclass | 5 +++++
3 files changed, 14 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 621d110e..0c34d9be 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -419,3 +419,9 @@ packages and will be lost after a given build session finishes.

Any package build requirements for the rootfs should be satisfied in the
Debian way via package dependencies.
+
+### Individual WIC partitions are no longer automatically deployed
+
+The individual wic partitions (e.g. *.wic.p1) are no longer automatically
+copied into the `DEPLOY_DIR_IMAGE`.
+To explicitly deploy these files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index ce7b58ee..57d06202 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
# Uncomment the below line to debug WIC.
# WIC_CREATE_EXTRA_ARGS += "-D"

+# Uncomment this to also deploy each wic partition as separate file (e.g. for swupdate)
+#WIC_DEPLOY_PARTITIONS = "1"
+
# Uncomment this to use ccache for custom packages
#USE_CCACHE = "1"
# Uncomment and set own top level ccache directory to share between builds
diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 61a74d4a..ba8aa15c 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"

WIC_CREATE_EXTRA_ARGS ?= ""
+WIC_DEPLOY_PARTITIONS ?= "0"

# taken from OE, do not touch directly
WICVARS += "\
@@ -205,6 +206,10 @@ generate_wic_image() {

sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true
sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}
+ # only keep partitions if requested (ending with .p<x>)
+ if [ "${WIC_DEPLOY_PARTITIONS}" -eq "0" ]; then
+ find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" -delete
+ fi
find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do
suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/')
mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"
--
2.30.2

[Felix Moessbauer]

This patch adds tests to check if the wic partition files
are copied to the deploydir if requested.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibase.py | 10 ++++++++++
testsuite/cibuilder.py | 5 ++++-
testsuite/citest.py | 23 +++++++++++++++++++++++
3 files changed, 37 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 722d7bd2..6239b4de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -17,6 +17,16 @@ class CIBaseTest(CIBuilder):

self.bitbake(targets, **kwargs)

+ def perform_wic_partition_test(self, targets, wic_deploy_parts, **kwargs):
+ self.configure(wic_deploy_parts=wic_deploy_parts, **kwargs)
+ self.bitbake(targets, **kwargs)
+
+ partition_files = set(glob.glob(f'{self.build_dir}/tmp/deploy/images/*/*.wic.p1'))
+ if wic_deploy_parts and len(partition_files) == 0:
+ self.fail('Found raw wic partitions in DEPLOY_DIR')
+ if not wic_deploy_parts and len(partition_files) != 0:
+ self.fail('Did not find raw wic partitions in DEPLOY_DIR')
+
def perform_repro_test(self, targets, signed=False, **kwargs):
gpg_pub_key = os.path.dirname(__file__) + '/keys/base-apt/test_pub.key'
gpg_priv_key = os.path.dirname(__file__) + '/keys/base-apt/test_priv.key'
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index dfb0a376..d5c83b22 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -54,7 +54,7 @@ class CIBuilder(Test):

def configure(self, compat_arch=True, cross=None, debsrc_cache=False,
container=False, ccache=False, sstate=False, offline=False,
- gpg_pub_key=None, **kwargs):
+ gpg_pub_key=None, wic_deploy_parts=False, **kwargs):
# write configuration file and set bitbake_args
# can run multiple times per test case
self.check_init()
@@ -77,6 +77,7 @@ class CIBuilder(Test):
f' ccache = {ccache}\n'
f' sstate = {sstate}\n'
f' gpg_pub_key = {gpg_pub_key}\n'
+ f' wic_deploy_parts = {wic_deploy_parts}\n'
f'===================================================')

# determine bitbake_args
@@ -105,6 +106,8 @@ class CIBuilder(Test):
f.write('IMAGE_INSTALL_remove = "example-module-${KERNEL_NAME} enable-fsck"\n')
if gpg_pub_key:
f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
+ if wic_deploy_parts:
+ f.write('WIC_DEPLOY_PARTITIONS = "1"\n')
if distro_apt_premir:
f.write('DISTRO_APT_PREMIRRORS = "%s"\n' % distro_apt_premir)
if ccache:
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..8381f21c 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -223,6 +223,29 @@ class RebuildTest(CIBaseTest):
finally:
self.restorefile(dpkgbase_file)

+class WicTest(CIBaseTest):
+
+ """
+ Test creation of wic images
+
+ :avocado: tags=wic,full
+ """
+ def test_wic_nodeploy_partitions(self):
+ targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+ self.init()
+ self.delete_from_build_dir('tmp')
+ self.perform_wic_partition_test(targets,
+ wic_deploy_parts=False, debsrc_cache=True, compat_arch=False)
+
+ def test_wic_deploy_partitions(self):
+ targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+ self.init()
+ # reuse artifacts
+ self.perform_wic_partition_test(targets,
+ wic_deploy_parts=True, debsrc_cache=True, compat_arch=False)
+
class ContainerImageTest(CIBaseTest):

"""
--
2.30.2

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
testsuite/citest.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@

MACHINE ?= "qemuarm64"
DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8381f21c..0c0a6284 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Henning Schild]

Am Mon, 19 Sep 2022 13:20:33 +0200
schrieb Felix Moessbauer <felix.mo...@siemens.com>:

I just had a look at OE and they seem to do things differently. They
call wic with a -w "temporary workdir to use for intermediate files"
and later move only the ".direct" to DEPLOY/<image>.wic

So i would suggest to do that as well, and not delete valid files that
maybe a second run or another tool might end up missing. Say you
generate a .wic.xz and a .swu that swu will maybe need the partition
image where the .wic.xz itself would not deploy it.

Henning

[Moessbauer, Felix]

I had a very careful look at the OE wic script as well and the only difference is that they do not copy any temporary files (like the partitions) to the deploy dir.
That's why they don't do this "find" but simply "move" the .direct file [1].
There had been proposed patches to change that (also from Siemens), but they never have been accepted.

The added "-w $tmp_wic" is indeed a divergence, but that is independent of the added / changed logic in this series.
In the end, the OE data in $tmp_wic is never re-used again.

>
> So i would suggest to do that as well, and not delete valid files that maybe a
> second run or another tool might end up missing. Say you generate a .wic.xz and
> a .swu that swu will maybe need the partition image where the .wic.xz itself
> would not deploy it.

If you look carefully at the following lines after the "find", you will see that the
WICTMP is deleted anyways. And in between, there is no extension point where any other tool could integrate and read the temporary files.
By that, I assume that my approach is correct and reliable.

[1] https://git.openembedded.org/openembedded-core/tree/meta/classes/image_types_wic.bbclass?id=958ee0eede859bdba659e3343856b1c226207854#n68

Felix

[Henning Schild]

Am Mon, 19 Sep 2022 13:20:33 +0200
schrieb Felix Moessbauer <felix.mo...@siemens.com>:

> This patch adds the WIC_DEPLOY_PARTITIONS option to control
> if the raw partition files are deployed by wic.
>
> We also set the default to not deploy to be closer to OE and to
> optimize for the average case.
> In addition, this significantly reduces the disk size required to
> run the CI.
>
> An API changelog entry is added.

I think the deployment of the partitions was an accident and never
intended. Cause might be

https://github.com/ilbers/isar/commit/edc10d9361e79f36c8ea5488b1dab4c468213a8d

which wanted to deal only with compressed images when wic itself is
used for compression.

We should maybe just sharpen that to not include partitions, instead of
inventing a new interface to remove them again.

Henning

[Jan Kiszka]

On 20.09.22 09:55, Henning Schild wrote:
> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.mo...@siemens.com>:
>
>> This patch adds the WIC_DEPLOY_PARTITIONS option to control
>> if the raw partition files are deployed by wic.
>>
>> We also set the default to not deploy to be closer to OE and to
>> optimize for the average case.
>> In addition, this significantly reduces the disk size required to
>> run the CI.
>>
>> An API changelog entry is added.
>
> I think the deployment of the partitions was an accident and never
> intended. Cause might be

We might use that in the context of swu (SWUpdate) file generation.
Related recipes can be found in
https://gitlab.com/cip-project/cip-core/isar-cip-core/. But I didn't
check yet if this particular case is related to the SWUpdate, I just
know that we package individual partitions in that use case.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Henning Schild]

Am Tue, 20 Sep 2022 09:55:43 +0200
schrieb Henning Schild <henning...@siemens.com>:

> Am Mon, 19 Sep 2022 13:20:33 +0200
> schrieb Felix Moessbauer <felix.mo...@siemens.com>:
>
> > This patch adds the WIC_DEPLOY_PARTITIONS option to control
> > if the raw partition files are deployed by wic.
> >
> > We also set the default to not deploy to be closer to OE and to
> > optimize for the average case.
> > In addition, this significantly reduces the disk size required to
> > run the CI.
> >
> > An API changelog entry is added.
>
> I think the deployment of the partitions was an accident and never
> intended. Cause might be
>
> https://github.com/ilbers/isar/commit/edc10d9361e79f36c8ea5488b1dab4c468213a8d

I think this patch should simply be reverted, which would bring us in
line with OE. Where also only the ".direct" is taken into deploydir.
While wic does have internal compressor support, people seem to rely on
the external one.

So i suggest you simply revert this one.

Keep reading for a changelog suggestion.

> which wanted to deal only with compressed images when wic itself is
> used for compression.
>
> We should maybe just sharpen that to not include partitions, instead
> of inventing a new interface to remove them again.
>
> Henning
>
> > Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
> > ---
> > RECIPE-API-CHANGELOG.md | 6 ++++++
> > meta-isar/conf/local.conf.sample | 3 +++
> > meta/classes/imagetypes_wic.bbclass | 5 +++++
> > 3 files changed, 14 insertions(+)
> >
> > diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
> > index 621d110e..0c34d9be 100644
> > --- a/RECIPE-API-CHANGELOG.md
> > +++ b/RECIPE-API-CHANGELOG.md
> > @@ -419,3 +419,9 @@ packages and will be lost after a given build
> > session finishes.
> > Any package build requirements for the rootfs should be satisfied
> > in the Debian way via package dependencies.
> > +
> > +### Individual WIC partitions are no longer automatically deployed

We used to copy all temporary WIC files, like the partitions, into the
deploy directory. That was intended actually only for compressed wic
images where wic itself would do the compression.
It was never intended to also deploy those partitions, so that will
also not be done anymore. For compressed wic images IMAGE_FSTYPES
should simply be extended with a compressed wic format, like "wic.xz".

Henning

[Henning Schild]

Am Tue, 20 Sep 2022 10:39:24 +0200
schrieb Jan Kiszka <jan.k...@siemens.com>:

Yes that use-case is "known" but still the patch was never created to
support that. I will happily assume that swupdate users never used
those partitions from that accident. But i still proposed a changelog
entry saying a word or two.

Henning

> Jan
>

[Felix Moessbauer]

This patch adds logic to detect when ISAR_CROSS_COMPILE
is set, but host-arch equals dist-arch.
In this case, we do not add the crossbuild-essential package,
but instead add the build-essential one.

To implement this, we transfer the existing logic into a
anonymous python function to make it better readable.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

Changes since v5:

- before changing WIC deploy logic, revert edc10d93 (p3)
- only reploy requested files (p4)

Changes since v4:

- add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
Note, that the default set to disabled is important.
This fixes the huge disk consumption when running the CI with WIC tests
- add test for WIC_DEPLOY_PARTITIONS logic (p4)
- add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)

Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches which are only sent as part of this series to fix the disk space blow up in CI.

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.

Felix Moessbauer (6):

do not crossbuild SDK for host-arch eq dist-arch
run imager in buildchroot-host on cross

Revert "wic: move out all files ending on "direct*""

add option to control deploy of raw wic partitions

add test for wic partition deploy logic
add wic.xz image to qemuarm64-bookworm

RECIPE-API-CHANGELOG.md | 9 +++++++

doc/user_manual.md | 11 +++++++++
meta-isar/conf/local.conf.sample | 3 +++
.../conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
meta/classes/image-tools-extension.bbclass | 3 ---

meta/classes/imagetypes_wic.bbclass | 16 +++++++++----

meta/classes/sdk.bbclass | 14 +++++++++--
testsuite/cibase.py | 10 ++++++++
testsuite/cibuilder.py | 5 +++-
testsuite/citest.py | 24 +++++++++++++++++++

10 files changed, 88 insertions(+), 10 deletions(-)

--
2.30.2

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

This reverts commit edc10d9361e79f36c8ea5488b1dab4c468213a8d.
---
meta/classes/imagetypes_wic.bbclass | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 61a74d4a..81788faf 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass
@@ -204,11 +204,10 @@ generate_wic_image() {
${WIC_CREATE_EXTRA_ARGS}

sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true

+ WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)

sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}

- find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do
- suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/')
- mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"
- done
+ mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
+ mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap
rm -rf ${BUILDCHROOT_DIR}/${WICTMP}
rm -rf ${IMAGE_ROOTFS}/../pseudo
}
--
2.30.2

[Felix Moessbauer]

This patch adds the WIC_DEPLOY_PARTITIONS option to control
if the raw partition files are deployed by wic.

We also set the default to not deploy to be closer to OE and to
optimize for the average case.
In addition, this significantly reduces the disk size required to
run the CI.

An API changelog entry is added.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 9 +++++++++
meta-isar/conf/local.conf.sample | 3 +++
meta/classes/imagetypes_wic.bbclass | 9 +++++++++
3 files changed, 21 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 621d110e..a5f5abee 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -419,3 +419,12 @@ packages and will be lost after a given build session finishes.

Any package build requirements for the rootfs should be satisfied in the
Debian way via package dependencies.
+
+### Individual WIC partitions are no longer automatically deployed

+
+We used to copy all temporary WIC files, like the partitions, into the deploy directory.
+That was intended actually only for compressed wic images where wic itself would do the compression.
+It was never intended to also deploy those partitions, so that will also not be done (automatically) anymore.
+To explicitly deploy the individual partition files (e.g. for swupdate), set `WIC_DEPLOY_PARTITIONS = "1"`.
+
+For compressed wic images `IMAGE_FSTYPES` should simply be extended with a compressed wic format, like "wic.xz".

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index ce7b58ee..57d06202 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -246,6 +246,9 @@ USER_isar[flags] += "clear-text-password"
# Uncomment the below line to debug WIC.
# WIC_CREATE_EXTRA_ARGS += "-D"

+# Uncomment this to also deploy each wic partition as separate file (e.g. for swupdate)
+#WIC_DEPLOY_PARTITIONS = "1"
+
# Uncomment this to use ccache for custom packages
#USE_CCACHE = "1"
# Uncomment and set own top level ccache directory to share between builds

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 81788faf..c0f3429a 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass

@@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"

WIC_CREATE_EXTRA_ARGS ?= ""
+WIC_DEPLOY_PARTITIONS ?= "0"

# taken from OE, do not touch directly
WICVARS += "\

@@ -208,6 +209,14 @@ generate_wic_image() {

sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}

mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic

mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap

+ # deploy partition files if requested (ending with .p<x>)
+ if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
+ # locate *.direct.p<x> partition files
+ find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
+ suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')
+ mv -f ${f} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}
+ done
+ fi

[Felix Moessbauer]

This patch adds tests to check if the wic partition files
are copied to the deploydir if requested.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 6c8eb26e..8381f21c 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
testsuite/citest.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@

MACHINE ?= "qemuarm64"
DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8381f21c..0c0a6284 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Gylstorff Quirin]

ACK - tested with isar-cip-core.

Quirin

[Anton Mikanovich]

Removing tmp during this test also removes all the images needed for further
VmBootTests. The easiest way to fix that is moving wic test below boot
tests.

[Felix Moessbauer]

Changes since v6:

- reorder tests in testsuite to not break vm_boot tests
- rebase against current next

Changes since v5:

- before changing WIC deploy logic, revert edc10d93 (p3)
- only reploy requested files (p4)

Changes since v4:

- add option WIC_DEPLOY_PARTITIONS = "0" to control if individual wic partitions are deployed (p3)
Note, that the default set to disabled is important.
This fixes the huge disk consumption when running the CI with WIC tests
- add test for WIC_DEPLOY_PARTITIONS logic (p4)
- add logic to handle cross-imaging when distro_arch == host_arch (similar to buildchroot-(target|host)) (p1)

Please note, that the additions around WIC_DEPLOY_PARTITIONS are standalone patches which are only sent as part of this series to fix the disk space blow up in CI.

Changes since v3:

- add test for CROSS=1 wic image including compression
- integrate patch 2 into this series instead of standalone.

Please note, this series supersedes the "add wic.xz image to default qemuarm64 target" series but does only requires little more disk space for the CI tests (compared to ISAR-next).
By that, the CI can still be executed on standard machines.

Changes since v1/v2:

instead of switching the compression logic, this patch adds support to run the whole imager in the buildchroot-host.

Best regards,
Felix

Felix Moessbauer (6):
do not crossbuild SDK for host-arch eq dist-arch
run imager in buildchroot-host on cross
Revert "wic: move out all files ending on "direct*""
add option to control deploy of raw wic partitions
add test for wic partition deploy logic
add wic.xz image to qemuarm64-bookworm

RECIPE-API-CHANGELOG.md | 9 +++++++
doc/user_manual.md | 11 +++++++++
meta-isar/conf/local.conf.sample | 3 +++
.../conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
meta/classes/image-tools-extension.bbclass | 3 ---
meta/classes/imagetypes_wic.bbclass | 16 +++++++++----
meta/classes/sdk.bbclass | 14 +++++++++--
testsuite/cibase.py | 10 ++++++++
testsuite/cibuilder.py | 5 +++-
testsuite/citest.py | 24 +++++++++++++++++++
10 files changed, 88 insertions(+), 10 deletions(-)

--
2.30.2

[Felix Moessbauer]

This patch adds logic to detect when ISAR_CROSS_COMPILE
is set, but host-arch equals dist-arch.
In this case, we do not add the crossbuild-essential package,
but instead add the build-essential one.

To implement this, we transfer the existing logic into a
anonymous python function to make it better readable.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

When globally enabling ISAR_CROSS_COMPILE, the imager and compression
tasks run in the host buildchroot as well (instead of the target
buildchroot).

Similar to the cross support for DPKG recipes, this can be toggled
on a per-image basis by setting ISAR_CROSS_COMPILE in the image recipe.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

[Felix Moessbauer]

This reverts commit edc10d9361e79f36c8ea5488b1dab4c468213a8d.
---
meta/classes/imagetypes_wic.bbclass | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 61a74d4a..81788faf 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass

@@ -204,11 +204,10 @@ generate_wic_image() {
${WIC_CREATE_EXTRA_ARGS}

sudo chown -R $(stat -c "%U" ${LAYERDIR_core}) ${LAYERDIR_core} ${LAYERDIR_isar} ${SCRIPTSDIR} || true
+ WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)

sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}

- find ${BUILDCHROOT_DIR}/${WICTMP} -type f -name "*.direct*" | while read f; do
- suffix=$(basename $f | sed 's/\(.*\)\(\.direct\)\(.*\)/\3/')

- mv -f ${f} "${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}"
- done
+ mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic
+ mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap

[Felix Moessbauer]

This patch adds the WIC_DEPLOY_PARTITIONS option to control
if the raw partition files are deployed by wic.

We also set the default to not deploy to be closer to OE and to
optimize for the average case.
In addition, this significantly reduces the disk size required to
run the CI.

An API changelog entry is added.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

diff --git a/meta/classes/imagetypes_wic.bbclass b/meta/classes/imagetypes_wic.bbclass
index 81788faf..c0f3429a 100644
--- a/meta/classes/imagetypes_wic.bbclass
+++ b/meta/classes/imagetypes_wic.bbclass

@@ -98,6 +98,7 @@ RECIPE_SYSROOT_NATIVE ?= "/"
BUILDCHROOT_DIR = "${BUILDCHROOT_TARGET_DIR}"

WIC_CREATE_EXTRA_ARGS ?= ""
+WIC_DEPLOY_PARTITIONS ?= "0"

# taken from OE, do not touch directly
WICVARS += "\
@@ -208,6 +209,14 @@ generate_wic_image() {

sudo chown -R $(id -u):$(id -g) ${BUILDCHROOT_DIR}/${WICTMP}

mv -f ${WIC_DIRECT} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic

mv -f ${WIC_DIRECT}.bmap ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.bmap

+ # deploy partition files if requested (ending with .p<x>)
+ if [ "${WIC_DEPLOY_PARTITIONS}" -eq "1" ]; then
+ # locate *.direct.p<x> partition files
+ find ${BUILDCHROOT_DIR}/${WICTMP} -type f -regextype sed -regex ".*\.direct.*\.p[0-9]\{1,\}" | while read f; do
+ suffix=$(basename $f | sed 's/.*\.direct\(.*\)/\1/')

+ mv -f ${f} ${DEPLOY_DIR_IMAGE}/${IMAGE_FULLNAME}.wic${suffix}
+ done
+ fi

[Felix Moessbauer]

This patch adds tests to check if the wic partition files
are copied to the deploydir if requested.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

index 6c8eb26e..e8cc299e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -318,3 +318,26 @@ class VmBootTestFull(CIBaseTest):
def test_amd64_focal(self):
self.init()
self.vm_start('amd64','focal')
+

+class WicTest(CIBaseTest):
+
+ """
+ Test creation of wic images
+
+ :avocado: tags=wic,full
+ """
+ def test_wic_nodeploy_partitions(self):
+ targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+ self.init()
+ self.delete_from_build_dir('tmp')

+ self.perform_wic_partition_test(targets,
+ wic_deploy_parts=False, debsrc_cache=True, compat_arch=False)
+
+ def test_wic_deploy_partitions(self):
+ targets = ['mc:qemuarm64-bookworm:isar-image-base']
+
+ self.init()
+ # reuse artifacts
+ self.perform_wic_partition_test(targets,
+ wic_deploy_parts=True, debsrc_cache=True, compat_arch=False)

--
2.30.2

[Felix Moessbauer]

This patch adds generation of a wic.xz image for the mc:qemuarm64-bookworm target.
By that, the cross and non-cross imaging is automatically tested in CI.
Further, we add a qemuarm64-bookworm target to the cross build test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-isar/conf/multiconfig/qemuarm64-bookworm.conf | 3 +++
testsuite/citest.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
index c6b92d2e..a04c9d1f 100644
--- a/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
+++ b/meta-isar/conf/multiconfig/qemuarm64-bookworm.conf
@@ -4,3 +4,6 @@

MACHINE ?= "qemuarm64"
DISTRO ?= "debian-bookworm"
+
+IMAGE_FSTYPES += "wic.xz"
+IMAGER_INSTALL += "${GRUB_BOOTLOADER_INSTALL}"

diff --git a/testsuite/citest.py b/testsuite/citest.py
index e8cc299e..a0e7e8b9 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Anton Mikanovich]

Applied to next, thanks.

[Uladzimir Bely]

In the email from Friday, 7 October 2022 21:53:39 +03 user Anton Mikanovich
wrote:

Now, with "[PATCH v7 5/6] add test for wic partition deploy logic" we lost all
"nocross" CI logs from "tmp/work" (the test deletes "tmp" directory).

While our tests are still sometimes depend on each other, the easiest fix
would be placing "wic" tests before "nocross" ones.

--
Uladzimir Bely

[MOESSBAUER, FELIX JONATHAN]

Hi Uladzimir,

Could you please try to implement this minimal patch.
I have a really hard time to run the full testsuite behind our proxies.

The whole CI currently has a lot of inter-test dependencies which make it really hard to add tests for new features.
Ideally, the test order should not matter at all.
In case the build-time is an issue, we should better rely on the sstate cache than on the artifacts of previous tests.

Felix


>
> --
> Uladzimir Bely
>
>