[PATCH 0/6] Various improvements to the testsuite

[Felix Moessbauer]

This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (6):
testsuite: move targets with custom kernel to separate test
testsuite: enable ccache on kernel tests
testsuite: make prebuilt container a feature test
testsuite: make compat test standalone test
testsuite: handle IMAGE_INSTALL solely in cibuilder.py
testsuite: limit cross_debsrc test to subset of packages

meta-test/conf/local.conf.sample | 10 ----
.../recipes-core/images/isar-image-ci.bb | 2 -
testsuite/cibuilder.py | 24 ++++++--
testsuite/citest.py | 56 +++++++++++++++++--
4 files changed, 71 insertions(+), 21 deletions(-)

--
2.51.0

[Felix Moessbauer]

By moving these tests out of the cross test (which is a fast test), we
significantly reduce the test time on fast.

The tests are now placed in the KernelTests class in a dedicated test,
suitable for applying further optimizations (separate commit).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
testsuite/citest.py | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 76a3df69..cb63a8fe 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -190,7 +190,8 @@ class InstallerTest(CIBaseTest):
class CrossTest(CIBaseTest):

"""
- Start cross build for the defined set of configurations
+ Start cross build for the defined set of configurations.
+ As this is a fast test, ensure to not depend on custom kernels.

:avocado: tags=cross,fast
"""
@@ -199,12 +200,9 @@ class CrossTest(CIBaseTest):
targets = [
'mc:qemuarm-buster:isar-image-ci',
'mc:qemuarm-bullseye:isar-image-ci',
- 'mc:de0-nano-soc-bullseye:isar-image-base',
- 'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',
- 'mc:phyboard-mira-bookworm:isar-image-base',
]

self.init()
@@ -273,6 +271,17 @@ class KernelTests(CIBaseTest):
:avocado: tags=kernel,full
"""

+ def test_kernel_cross(self):
+ """Targets that build a custom kernel"""
+ targets = [
+ 'mc:de0-nano-soc-bullseye:isar-image-base',
+ 'mc:stm32mp15x-bullseye:isar-image-base',
+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)
+
def test_per_kernel(self):
"""Test per-kernel recipe variants for external kernel modules."""

--
2.51.0

[Felix Moessbauer]

The kernels are anyways usually built with ccache enabled to reduce the
buildtime. By enabling it in the test as well, we can significantly
bring down test time on repeated builds.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index cb63a8fe..cba3c008 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -280,7 +280,7 @@ class KernelTests(CIBaseTest):
]

self.init()
- self.perform_build_test(targets)
+ self.perform_build_test(targets, ccache=True)

[Felix Moessbauer]

This reworks how we handle feature tests: Instead of binding a feature
to a target via appends in isar-image-ci, we implement the feature by
using the test setup function. By that, we have fine grained control over
where to test a feature and can avoid testing the same feature over and
over again. This leads to a much cleaner architecture and faster test
execution.

We start implementing this approach with the prebuilt container test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-core/images/isar-image-ci.bb | 2 --
testsuite/citest.py | 17 +++++++++++++++++
2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index 7f2b404d..58aa5738 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,7 +16,6 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
-IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.zst tar.zst"
@@ -52,4 +51,3 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
-IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cba3c008..601eca92 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -265,6 +265,23 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, lines=lines)

+class PrebuiltTest(CIBaseTest):
+ """
+ Tests associated with prebuilt artifacts (containers, debs).
+ :avocado: tags=prebuilt,fast
+ """
+
+ def test_prebuilt_containers(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-image-ci',
+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ ]
+
+ self.init()
+ # perform imaging as well, as needed by VmBootTestFull
+ self.perform_build_test(targets, image_install="prebuilt-docker-img prebuilt-podman-img")
+
+
class KernelTests(CIBaseTest):
"""
Tests associated with kernel builds and development.
--
2.51.0

[Felix Moessbauer]

Currently, enabling compat mode during configuration also changes the
set of packages we build. As this is done with appends, some packages
are unconditionally added, despite not being part of the original test
case. We change this by only enabling compat mode when selecting compat.
To actually test the compat feature, we add a simple and fast test that
builds the hello-isar-compat package.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 4 ----
testsuite/citest.py | 19 +++++++++++++++++++
2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 46bfc546..25e3b6cb 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -180,11 +180,7 @@ class CIBuilder(Test):
if compat_arch:
f.write(
'ISAR_ENABLE_COMPAT_ARCH:amd64 = "1"\n'
- 'IMAGE_INSTALL:remove:amd64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:amd64 = " hello-isar-compat"\n'
'ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n'
- 'IMAGE_INSTALL:remove:arm64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n'
)
if not cross:
f.write('ISAR_CROSS_COMPILE = "0"\n')
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 601eca92..aee138bc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -112,6 +112,25 @@ class DevTest(CIBaseTest):
self.vm_start('arm', 'bookworm', skip_modulecheck=True)


+class CompatTest(CIBaseTest):
+ """
+ Test compilation of recipes for compat architecture.
+ This also tests a custom sbuild chroot for compat.
+ :avocado: tags=compat,fast
+ """
+
+ def test_compat_recipe(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:hello-isar-compat',
+ 'mc:qemuarm64-bookworm:hello-isar-compat',
+ 'mc:qemuamd64-trixie:hello-isar-compat',
+ 'mc:qemuarm64-trixie:hello-isar-compat',
+ ]
+
+ self.init()
+ self.perform_build_test(targets, compat_arch=True)
+
+
class ReproTest(CIBaseTest):

"""
--
2.51.0

[Felix Moessbauer]

By moving the IMAGE_INSTALL parts out of the ci local conf sample, we
make this part reusable across tests. By that, tests can easily
customize this list without either taking it as-is or completely
replacing it. We further get rid of overrides that are not used in the
CI anyways.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/conf/local.conf.sample | 10 ----------
testsuite/cibuilder.py | 20 ++++++++++++++++++++
2 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 092d6ba6..862bea47 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,16 +27,6 @@ BB_DISKMON_DIRS = "\
MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"

-# The default list of extra packages
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
-
-# Container machines don't need example module and enable-fsck.
-IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
-
-# Machines with secure boot should use signed modules
-IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
-IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
-
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 25e3b6cb..26002ade 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -30,6 +30,18 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
import bb

DEF_VM_TO_SEC = 600
+IMAGE_INSTALL_DEFAULT = [
+ 'hello-isar',
+ 'example-raw',
+ 'example-module-${KERNEL_NAME}',
+ 'enable-fsck',
+ 'isar-exclude-docs',
+ 'samefile',
+ 'hello',
+ 'isar-disable-apt-cache',
+ 'cowsay',
+ 'example-prebuilt'
+]

isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
backup_prefix = '.ci-backup'
@@ -222,6 +234,14 @@ class CIBuilder(Test):
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+ else:
+ if container:
+ # strip kernel modules from default package install list
+ _image_install = [p for p in IMAGE_INSTALL_DEFAULT if "-module-" not in p]
+ else:
+ _image_install = IMAGE_INSTALL_DEFAULT
+ f.write('IMAGE_INSTALL = "%s"\n' % ' '.join(_image_install))
+
if fail_on_cleanup == '1':
f.write('ISAR_FAIL_ON_CLEANUP = "1"\n')
if installer_image:
--
2.51.0

[Felix Moessbauer]

To test if the debsrc infrastructure works it is sufficient to build a
single package. This significantly speeds up the CI and reduce the space
needed.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index aee138bc..ab509e22 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -233,7 +233,8 @@ class CrossTest(CIBaseTest):
]

self.init()
- self.perform_build_test(targets, debsrc_cache=True)
+ # only build a single custom package to speedup test
+ self.perform_build_test(targets, debsrc_cache=True, image_install='cowsay')

def test_cross_trixie(self):
targets = [
--
2.51.0

[MOESSBAUER, Felix]

On Wed, 2025-12-17 at 15:08 +0100, Felix Moessbauer wrote:
> This series aims at simplifying and speeding up the testsuite execution.
> This should enable contributors to actually run it and test their patches.

Hi,

I plan to send out a v2 of this series by end of today. This will
contain all "testsuite:" and "ci:" patches recently sent to the ML and
should make applying easier. There are also a couple of fixes in the
more recent parts (not this one), which will be included and marked as
well.

@Anton: Did you already test parts of the series and already have
feedback if it is conceptually ok and also passes your CI?

Best regards,
Felix

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[Anton Mikanovich]

19/12/2025 13:06, MOESSBAUER, Felix wrote:
> Hi,
>
> I plan to send out a v2 of this series by end of today. This will
> contain all "testsuite:" and "ci:" patches recently sent to the ML and
> should make applying easier. There are also a couple of fixes in the
> more recent parts (not this one), which will be included and marked as
> well.
>
> @Anton: Did you already test parts of the series and already have
> feedback if it is conceptually ok and also passes your CI?
>
> Best regards,
> Felix

Hello Felix,

Yes, I've tried this patchset on CI and it fails on full CI.

Following test cases failed:
test_amd64_bookworm_prebuilt_containers
test_arm64_bookworm_prebuilt_containers

It happens because ssh connection got "identification changed" warning
(sshd-regen-keys issue?) and started waiting for "ssh-keygen -f" command.

Regarding the contents on the patchset I like the ways it is done, and I
hope
I will also rebuild my "Additional CI improvements" patchset sometimes,
which
will also speedup our CI.

[MOESSBAUER, Felix]

On Fri, 2025-12-19 at 13:14 +0200, Anton Mikanovich wrote:
> 19/12/2025 13:06, MOESSBAUER, Felix wrote:
> > Hi,
> >
> > I plan to send out a v2 of this series by end of today. This will
> > contain all "testsuite:" and "ci:" patches recently sent to the ML and
> > should make applying easier. There are also a couple of fixes in the
> > more recent parts (not this one), which will be included and marked as
> > well.
> >
> > @Anton: Did you already test parts of the series and already have
> > feedback if it is conceptually ok and also passes your CI?
> >
> > Best regards,
> > Felix
> Hello Felix,
>
> Yes, I've tried this patchset on CI and it fails on full CI.
>
> Following test cases failed:
> test_amd64_bookworm_prebuilt_containers
> test_arm64_bookworm_prebuilt_containers
>
> It happens because ssh connection got "identification changed" warning
> (sshd-regen-keys issue?) and started waiting for "ssh-keygen -f" command.

The issues seems to be a test-ordering issue. Prior to the "testsuite:
make prebuilt container a feature test", the containers (and also sshd-
regen-keys) package have always been added to the image. However, now
some tests only install specific packages, breaking the otherwise
unrelated VM boot tests later on.

The correct way to fix this is to always build the needed image prior
to the VM execution. By that, we can also avoid the imaging step in
most (non VM) tests.

This is something that is really hard to fix as we always have to run
the whole testsuite to expose the hidden inter-test dependencies. I'm
thinking of instead encoding the requirements of the VM tests in a
simple check script that parses the manifest of a image build and then
skips the test if dependencies are not there. By that, we can work on
the rebuilds one-by-one without breaking the testsuite.

I already have a similar patch in my queue to skip the VM tests if the
image itself is not available.

>
> Regarding the contents on the patchset I like the ways it is done, and I
> hope
> I will also rebuild my "Additional CI improvements" patchset sometimes,
> which
> will also speedup our CI.

Sounds good. The other reported issues will be fixed in a v2.

Felix

>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/5510e9b4-168e-4432-a966-832994290b65%40ilbers.de.

[Felix Moessbauer]

The kernels are anyways usually built with ccache enabled to reduce the
buildtime. By enabling it in the test as well, we can significantly
bring down test time on repeated builds.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index cb63a8fe..cba3c008 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Felix Moessbauer]

This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Changes since v1:

- continued work on testsuite refactoring, this supersedes:
- testsuite: fix SignatureTest by avoiding absolute path in bblayers
- testsuite: add rootfs target for rootfs only tests
- Enable SState cache in GitLab CI
- testsuite: fix typo in log message in perform_signature_lint
- completed exclusion / refactoring of db2m sbuild chroot
- GitLab CI: increased after script timeout to give sstate upload more time
- GitLab CI: do not upload sstate artifacts of images (as almost no reuse)
- do not add SignatureTest to fast tag (as otherwise VM tests break...)
- rebuild container test images in VM test to not depend on test order
- rebased onto next

As always, the patches are written in a linear way, so only parts of them
can already be integrated.

That's a wrap for this year, isar developers!
See you next year!

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (17):

testsuite: move targets with custom kernel to separate test
testsuite: enable ccache on kernel tests

testsuite: make prebuilt container a feature test

testsuite: make compat test standalone test
testsuite: handle IMAGE_INSTALL solely in cibuilder.py
testsuite: limit cross_debsrc test to subset of packages

testsuite: forward SSTATE_MIRRORS into CI env on sstate
ci: add support for sstate cache
testsuite: make test_cross_deps more specific
testsuite: fix typo in log message in perform_signature_lint
testsuite: fix SignatureTest by avoiding absolute path in bblayers
testsuite: use more recent distros in SignatureTest
testsuite: make SignatureTest idempotent
testsuite: add rootfs target for rootfs only tests
testsuite: refactor sbom tests to avoid overhead
testsuite: make sbuild-flavor test standalone
testsuite: skip VM tests if images are not available

.gitlab-ci.yml | 36 ++++-
meta-test/conf/bblayers.conf.sample | 8 +-
meta-test/conf/local.conf.sample | 10 --
.../hello-isar/hello-isar.bbappend | 3 +
.../recipes-app/libhello/libhello.bbappend | 5 +
.../recipes-core/images/isar-image-ci.bb | 2 -
.../recipes-core/images/isar-rootfs-ci.bb | 17 +++
testsuite/cibase.py | 34 ++++-
testsuite/cibuilder.py | 49 ++++++-
testsuite/citest.py | 131 ++++++++++++++++--
10 files changed, 255 insertions(+), 40 deletions(-)
create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend
create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb

--
2.51.0

[Felix Moessbauer]

By moving these tests out of the cross test (which is a fast test), we
significantly reduce the test time on fast.

The tests are now placed in the KernelTests class in a dedicated test,
suitable for applying further optimizations (separate commit).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 76a3df69..cb63a8fe 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

@@ -190,7 +190,8 @@ class InstallerTest(CIBaseTest):
class CrossTest(CIBaseTest):

"""
- Start cross build for the defined set of configurations
+ Start cross build for the defined set of configurations.
+ As this is a fast test, ensure to not depend on custom kernels.

:avocado: tags=cross,fast
"""
@@ -199,12 +200,9 @@ class CrossTest(CIBaseTest):
targets = [
'mc:qemuarm-buster:isar-image-ci',
'mc:qemuarm-bullseye:isar-image-ci',
- 'mc:de0-nano-soc-bullseye:isar-image-base',
- 'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',
- 'mc:phyboard-mira-bookworm:isar-image-base',
]

self.init()
@@ -273,6 +271,17 @@ class KernelTests(CIBaseTest):
:avocado: tags=kernel,full
"""

+ def test_kernel_cross(self):
+ """Targets that build a custom kernel"""

+ targets = [

+ 'mc:de0-nano-soc-bullseye:isar-image-base',
+ 'mc:stm32mp15x-bullseye:isar-image-base',

+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)
+

[Felix Moessbauer]

Currently, enabling compat mode during configuration also changes the
set of packages we build. As this is done with appends, some packages
are unconditionally added, despite not being part of the original test
case. We change this by only enabling compat mode when selecting compat.
To actually test the compat feature, we add a simple and fast test that
builds the hello-isar-compat package.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 4 ----
testsuite/citest.py | 19 +++++++++++++++++++

2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 46bfc546..25e3b6cb 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py

@@ -180,11 +180,7 @@ class CIBuilder(Test):
if compat_arch:
f.write(
'ISAR_ENABLE_COMPAT_ARCH:amd64 = "1"\n'
- 'IMAGE_INSTALL:remove:amd64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:amd64 = " hello-isar-compat"\n'
'ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n'
- 'IMAGE_INSTALL:remove:arm64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n'
)
if not cross:
f.write('ISAR_CROSS_COMPILE = "0"\n')

diff --git a/testsuite/citest.py b/testsuite/citest.py
index eec90c6e..5ff4a4ed 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

@@ -112,6 +112,25 @@ class DevTest(CIBaseTest):
self.vm_start('arm', 'bookworm', skip_modulecheck=True)


+class CompatTest(CIBaseTest):
+ """
+ Test compilation of recipes for compat architecture.
+ This also tests a custom sbuild chroot for compat.
+ :avocado: tags=compat,fast
+ """
+
+ def test_compat_recipe(self):

+ targets = [

+ 'mc:qemuamd64-bookworm:hello-isar-compat',
+ 'mc:qemuarm64-bookworm:hello-isar-compat',
+ 'mc:qemuamd64-trixie:hello-isar-compat',

+ 'mc:qemuarm64-trixie:hello-isar-compat',
+ ]
+
+ self.init()

[Felix Moessbauer]

This reworks how we handle feature tests: Instead of binding a feature
to a target via appends in isar-image-ci, we implement the feature by
using the test setup function. By that, we have fine grained control over
where to test a feature and can avoid testing the same feature over and
over again. This leads to a much cleaner architecture and faster test
execution.

We start implementing this approach with the prebuilt container test.

As the images with containers are currently re-used in the
VmBootTestFull, these tests might break depending on the test execution
order. To fix this, we rebuild the needed images in the VM test itself,
which takes less than a minute when running with sstate cache.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

.../recipes-core/images/isar-image-ci.bb | 2 --
testsuite/citest.py | 25 +++++++++++++++++++
2 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index 7f2b404d..58aa5738 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,7 +16,6 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
-IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.zst tar.zst"
@@ -52,4 +51,3 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
-IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

diff --git a/testsuite/citest.py b/testsuite/citest.py
index cba3c008..eec90c6e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -265,6 +265,25 @@ class CrossTest(CIBaseTest):

self.init()
self.perform_build_test(targets, lines=lines)

+class PrebuiltTest(CIBaseTest):
+ """
+ Tests associated with prebuilt artifacts (containers, debs).
+ :avocado: tags=prebuilt,fast
+ """
+
+ def test_prebuilt_containers(self):

+ targets = [

+ 'mc:qemuamd64-bookworm:isar-image-ci',

+ 'mc:qemuarm64-bookworm:isar-image-ci',

+ ]
+
+ self.init()
+ self.perform_build_test(

+ targets,
+ bitbake_cmd='do_rootfs_install',
+ image_install="prebuilt-docker-img prebuilt-podman-img")

+
+
class KernelTests(CIBaseTest):
"""
Tests associated with kernel builds and development.

@@ -984,11 +1003,17 @@ class VmBootTestFull(CIBaseTest):

def test_amd64_bookworm_prebuilt_containers(self):
self.init()
+ self.perform_build_test(

+ ['mc:qemuamd64-bookworm:isar-image-ci'],

+ image_install="prebuilt-docker-img prebuilt-podman-img")
self.vm_start('amd64', 'bookworm', image='isar-image-ci',
script='test_prebuilt_containers.sh')

def test_arm64_bookworm_prebuilt_containers(self):
self.init()
+ self.perform_build_test(

+ ['mc:qemuarm64-bookworm:isar-image-ci'],

+ image_install="prebuilt-docker-img prebuilt-podman-img")
self.vm_start('arm64', 'bookworm', image='isar-image-ci',
script='test_prebuilt_containers.sh')

--
2.51.0

[Felix Moessbauer]

By moving the IMAGE_INSTALL parts out of the ci local conf sample, we
make this part reusable across tests. By that, tests can easily
customize this list without either taking it as-is or completely
replacing it. We further get rid of overrides that are not used in the
CI anyways.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/conf/local.conf.sample | 10 ----------
testsuite/cibuilder.py | 20 ++++++++++++++++++++
2 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 092d6ba6..862bea47 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,16 +27,6 @@ BB_DISKMON_DIRS = "\
MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"

-# The default list of extra packages
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
-
-# Container machines don't need example module and enable-fsck.
-IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
-
-# Machines with secure boot should use signed modules
-IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
-IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
-
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 25e3b6cb..26002ade 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py

[Felix Moessbauer]

When running the testsuite with sstate caching enabled and also the
environment variable SSTATE_MIRRORS is set, add this to the local conf,
so the CI can also use remote sstate caches (important for CI).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 26002ade..310a3836 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -232,6 +232,8 @@ class CIBuilder(Test):
f.write('DL_DIR = "%s"\n' % dl_dir)
if sstate_dir:

f.write('SSTATE_DIR = "%s"\n' % sstate_dir)

+ if sstate and 'SSTATE_MIRRORS' in os.environ:
+ f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS'])

if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)

else:
--
2.51.0

[Felix Moessbauer]

To test if the debsrc infrastructure works it is sufficient to build a
single package. This significantly speeds up the CI and reduce the space
needed.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 5ff4a4ed..3c92f788 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Felix Moessbauer]

The gitlab CI jobs should finish in a timely manner to give quick
feedback to the developer. By running the tests with sstate cache,
simple changes require far less CI time.

We prepare the gitlab ci job description to pick up the sstate cache
configuration from the environment and enable the caching for all jobs
(we always enable it, even if running without sstate cache, as then the
cache is simply thrown away later on). We further introduce a (manual)
info task to show the state of the cache, as well as a cleanup task to
drain the cache.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

.gitlab-ci.yml | 36 ++++++++++++++++++++++++++++++------
1 file changed, 30 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1a7abcb8..caaa56a8 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -6,12 +6,22 @@ variables:

.common-build: &common-build
stage: build
+ variables:
+ RUNNER_AFTER_SCRIPT_TIMEOUT: 15m
before_script:
- export http_proxy=$HTTP_PROXY
- export https_proxy=$HTTPS_PROXY
- export ftp_proxy=$FTP_PROXY
- export no_proxy=$NO_PROXY
- export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
+ after_script:
+ - |
+ if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
+ echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
+ ./scripts/isar-sstate --filter '^(?!isar-image-)' upload "sstate-cache" "${SSTATE_LOCATION}"
+ ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+ fi
+
artifacts:
name: "logs-$CI_JOB_ID"
paths:
@@ -48,7 +58,7 @@ dev-ci:
- *use-default-image
- if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev

fast-ci:
<<: *common-build
@@ -56,7 +66,7 @@ fast-ci:
- *use-default-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast

full-ci:
<<: *common-build
@@ -68,7 +78,7 @@ full-ci:
- PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
- if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
- scripts/ci_build.sh -T full;
+ scripts/ci_build.sh --sstate 1 -T full;
fi
cache:
key: "$CI_COMMIT_REF_SLUG"
@@ -83,7 +93,7 @@ dev-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'dev'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev

fast-ci-isar:
<<: *docker-isar
@@ -92,7 +102,7 @@ fast-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast

full-ci-isar:
<<: *docker-isar
@@ -101,4 +111,18 @@ full-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'full'
script:
- - scripts/ci_build.sh -T full
+ - scripts/ci_build.sh --sstate 1 -T full
+
+sstate-cache-info:
+ stage: build
+ when: manual
+ script:
+ - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+
+sstate-cache-clean:
+ stage: build
+ when: manual
+ variables:
+ SSTATE_MAX_AGE: "0d"
+ script:
+ - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}"
--
2.51.0

[Felix Moessbauer]

We don't need to build all packages to just check the cross dependency
propagation. Just install the single package we are interested in.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3c92f788..52f86486 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -278,12 +278,11 @@ class CrossTest(CIBaseTest):
'mc:qemuarm64-bookworm:isar-image-ci',
]

- lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'",
- f"IMAGE_INSTALL:append = ' test-all-deponlycross'",
- ]
+ lines = [f"IMAGER_BUILD_DEPS:append = ' test-all-depnocross-native'"]

self.init()
- self.perform_build_test(targets, lines=lines)
+ self.perform_build_test(targets, lines=lines,
+ image_install='test-all-deponlycross')

class PrebuiltTest(CIBaseTest):
"""
--
2.51.0

[Felix Moessbauer]

No functional change.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibase.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index e59653a2..5b0139de 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -168,7 +168,7 @@ class CIBaseTest(CIBuilder):
**kwargs,
):
"""
- Generate signature data for target(s) and check for cachability issues
+ Generate signature data for target(s) and check for cacheability issues
"""
self.configure(**kwargs)
self.move_in_build_dir('tmp', 'tmp_before_sstate')
@@ -189,7 +189,7 @@ class CIBaseTest(CIBuilder):
ansi_escape = re.compile(r'\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])')
for line in output.splitlines():
self.log.error(ansi_escape.sub('', line))
- self.fail("Detected cachability issues")
+ self.fail("Detected cacheability issues")

def perform_sstate_test(self, image_target, package_target, **kwargs):
def check_executed_tasks(target, expected):
--
2.51.0

[Felix Moessbauer]

In general, that should not make a difference. However, we better test
things that users actually use than old-old-stable distros. As this test
is just a parser test, it is fast. Hence add the fast tag.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 52f86486..eaa4c440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -708,10 +708,10 @@ class SignatureTest(CIBaseTest):
def test_signature_lint(self):
verbose = bool(int(self.params.get('verbose', default=0)))
targets = [
- 'mc:qemuamd64-bullseye:isar-image-ci',
- 'mc:qemuarm-bullseye:isar-image-base',
- 'mc:qemuarm-bullseye:isar-image-base:do_populate_sdk',
- 'mc:qemuamd64-focal:isar-image-base',
+ 'mc:qemuamd64-trixie:isar-image-ci',
+ 'mc:qemuarm-trixie:isar-image-base',
+ 'mc:qemuarm-trixie:isar-image-base:do_populate_sdk',
+ 'mc:qemuamd64-noble:isar-image-base',
]

self.init()
--
2.51.0

[Felix Moessbauer]

The SignatureTest currently fails, as it correctly detects absolute path
that influence the sstate signatures. These path come from the ISARROOT
substitution done in the isar-init-build-env script. Depending on how
the bblayers is setup (which again depends on who does it), we might end
up with absolute paths.

To fix this, we apply the same strategy as yocto and kas uses: Making
the layer imports relative to the TOPDIR. As we can only guarantee the
correctness of this path within the testsuite, we also only apply the
fix there.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/conf/bblayers.conf.sample | 8 ++++----

1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta-test/conf/bblayers.conf.sample b/meta-test/conf/bblayers.conf.sample
index dcec6cf6..f37625d2 100644
--- a/meta-test/conf/bblayers.conf.sample
+++ b/meta-test/conf/bblayers.conf.sample
@@ -9,10 +9,10 @@ BBPATH = "${TOPDIR}"
BBFILES ?= ""

BBLAYERS ?= " \
- ##ISARROOT##/meta \
- ##ISARROOT##/meta-isar \
- ##ISARROOT##/meta-test \
+ ${TOPDIR}/../meta \
+ ${TOPDIR}/../meta-isar \
+ ${TOPDIR}/../meta-test \
"
BBLAYERS_NON_REMOVABLE ?= " \
- ##ISARROOT##/meta \
+ ${TOPDIR}/../meta \
"
--
2.51.0

[Felix Moessbauer]

The SignatureTest copies the tmp dir for later comparison. However, this
currently fails in case the test was run before as the target dir then
already exists. We fix it by clearing the target dir upfront. We further
reduce the disk consumption by removing the tmp dir copy in case the
test was successfull. On error, we keep the copy for manual
analysis.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibase.py | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5b0139de..5ef1a5b5 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -171,6 +171,7 @@ class CIBaseTest(CIBuilder):

Generate signature data for target(s) and check for cacheability issues
"""
self.configure(**kwargs)

+ self.delete_from_build_dir('tmp_before_sstate')

self.move_in_build_dir('tmp', 'tmp_before_sstate')

self.bitbake(targets, sig_handler='none')

@@ -190,6 +191,9 @@ class CIBaseTest(CIBuilder):

for line in output.splitlines():
self.log.error(ansi_escape.sub('', line))

self.fail("Detected cacheability issues")

+ else:
+ # on success, cleanup temporary copy (keep on failure to inspect)
+ self.delete_from_build_dir('tmp_before_sstate')

[Felix Moessbauer]

As a preparation to speedup the testsuite, we add a rootfs recipe that
behaves similar to the image recipe, however does not carry a initrd or
a kernel which avoids the huge emulation overhead on non native
architectures. This recipe also can be used to check rootfs features
independently.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-core/images/isar-rootfs-ci.bb | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
create mode 100644 meta-test/recipes-core/images/isar-rootfs-ci.bb

diff --git a/meta-test/recipes-core/images/isar-rootfs-ci.bb b/meta-test/recipes-core/images/isar-rootfs-ci.bb
new file mode 100644
index 00000000..a87fd1a8
--- /dev/null
+++ b/meta-test/recipes-core/images/isar-rootfs-ci.bb
@@ -0,0 +1,17 @@
+# CI root filesystem for target installation (without kernel, initrd, ...)
+#
+# This software is a part of ISAR.
+# Copyright (C) 2025 Siemens
+
+# Bill-of-material
+ROOTFS_MANIFEST_DEPLOY_DIR = "${DEPLOY_DIR_IMAGE}"
+
+ROOTFSDIR = "${WORKDIR}/rootfs"
+ROOTFS_FEATURES = "generate-sbom"
+
+inherit multiarch
+inherit rootfs
+
+# behave similar to image class, so we can reuse the testing infrastructure
+DEPENDS += "${IMAGE_INSTALL}"
+ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
--
2.51.0

[Felix Moessbauer]

We currently test the SBOM infrastructure in all image builds, which
adds a significant overhead. We now change this to not generate SBOMs in
general (and by that avoid building the dependencies). To not have a
testing gap, we add a dedicated SBOM test that checks the SBOM creation
for various targets. In addition, we now also check the content of the
SBOM for plausibility.

In the future, the SBOM test can be extended without slowing down the
overall test execution.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibase.py | 26 ++++++++++++++++++++++++++
testsuite/cibuilder.py | 4 ++++
testsuite/citest.py | 33 +++++++++++++++++++++++++++++++++
3 files changed, 63 insertions(+)

diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index 5ef1a5b5..fd6a3df9 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -140,6 +140,32 @@ class CIBaseTest(CIBuilder):
self.delete_from_build_dir('ccache')
self.unconfigure()

+ def perform_sbom_test(self, targets, **kwargs):
+ """
+ Build a rootfs containing a needle package and check if that package
+ is added to the sbom.
+ """
+ import json
+
+ needle_pkg = 'cowsay'
+ self.perform_build_test(
+ targets, image_install=needle_pkg,
+ generate_sbom=True
+ )
+
+ for t in targets:
+ ds, pn, distro, machine = \
+ CIUtils.getVars('DEPLOY_DIR_SBOM', 'PN', 'DISTRO', 'MACHINE',
+ target=t)
+ for t in ["cdx", "spdx"]:
+ sbom_path = os.path.join(ds, f'{pn}-{distro}-{machine}.{t}.json')
+ self.log.info(f"Check {t} SBOM in {sbom_path}")
+ with open(sbom_path) as f:
+ sbom = json.load(f)
+ pkg_key = 'components' if t == 'cdx' else 'packages'
+ if not any(c for c in sbom[pkg_key] if c['name'] == needle_pkg):
+ self.fail(f'{needle_pkg} package not found in SBOM {sbom_path}')
+
def perform_sstate_populate(self, image_target, **kwargs):
# Use a different isar root for populating sstate cache
isar_sstate = f"{isar_root}/isar-sstate"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 310a3836..614a3397 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -126,6 +126,7 @@ class CIBuilder(Test):
installer_distro=None,
installer_device=None,
customizations=None,
+ generate_sbom=False,
lines=None,
**kwargs,
):
@@ -176,6 +177,7 @@ class CIBuilder(Test):
f" image_install = {image_install}\n"
f" installer_image = {installer_image}\n"
f" customizations = {customizations}\n"
+ f" generate_sbom = {generate_sbom}\n"
f" lines = {strlines}\n"
f"==================================================="
)
@@ -275,6 +277,8 @@ class CIBuilder(Test):
'CUSTOMIZATION_FOR_IMAGES:append = " isar-image-ci"\n'
'HOSTNAME:isar-image-ci = "isar-ci"\n'
)
+ if generate_sbom is False:
+ f.write('ROOTFS_FEATURES:remove = "generate-sbom"\n')
if lines is not None:
f.writelines((line + '\n' if not line.endswith('\n') else line) for line in lines)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index eaa4c440..d908f9bc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -694,6 +694,39 @@ class CustomizationsTest(CIBaseTest):
)


+class SbomTest(CIBaseTest):
+ """
+ Test to check if sbom is generated and contains expected packages.
+ Most tests are rootfs tests to avoid costly initrd build and imaging.
+
+ :avocado: tags=sbom,fast
+ """
+
+ def test_sbom_rootfs_generate(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-rootfs-ci',
+ 'mc:qemuarm64-bookworm:isar-rootfs-ci',
+ 'mc:qemuamd64-trixie:isar-rootfs-ci',
+ 'mc:qemuarm64-trixie:isar-rootfs-ci',
+ 'mc:qemuamd64-noble:isar-rootfs-ci',
+ ]
+
+ self.init()
+ self.perform_sbom_test(targets)
+
+ def test_sbom_unsupported(self):
+ targets = [
+ 'mc:qemuamd64-bullseye:isar-rootfs-ci',
+ 'mc:qemuamd64-focal:isar-rootfs-ci',

+ ]
+
+ self.init()
+ self.perform_build_test(

+ targets, bitbake_cmd='do_rootfs', image_install='cowsay',
+ generate_sbom=True
+ )
+
+
class SignatureTest(CIBaseTest):

"""
--
2.51.0

[Felix Moessbauer]

As the hello-isar recipe acts as an example for the SBUILD_FLAVOR
feature, we also pull in a dedicated chroot into almost all tests. This
is very costly and does not add much value. We change this by setting
the SBUILD_FLAVOR of hello-isar to none in the CI layer and add a
dedicated test that just tests the SBUILD_FLAVOR feature.

This only slightly reduces the test coverage, but it significantly
speeds up the test execution.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

.../hello-isar/hello-isar.bbappend | 3 +++
.../recipes-app/libhello/libhello.bbappend | 5 +++++
testsuite/citest.py | 21 +++++++++++++++++++
3 files changed, 29 insertions(+)
create mode 100644 meta-test/recipes-app/libhello/libhello.bbappend

diff --git a/meta-test/recipes-app/hello-isar/hello-isar.bbappend b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
index 44686458..27212262 100644
--- a/meta-test/recipes-app/hello-isar/hello-isar.bbappend
+++ b/meta-test/recipes-app/hello-isar/hello-isar.bbappend
@@ -15,3 +15,6 @@ SRC_URI:append = " \
"
SRC_URI:remove = "file://nonexist-file"
SRC_URI:remove = "git://nonexist-git"
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/meta-test/recipes-app/libhello/libhello.bbappend b/meta-test/recipes-app/libhello/libhello.bbappend
new file mode 100644
index 00000000..3c88a741
--- /dev/null
+++ b/meta-test/recipes-app/libhello/libhello.bbappend
@@ -0,0 +1,5 @@

+# This software is a part of ISAR.

+# Copyright (C) Siemens
+
+# avoid creating a dedicated sbuild chroot
+SBUILD_FLAVOR = ""
diff --git a/testsuite/citest.py b/testsuite/citest.py
index d908f9bc..49fcdec0 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -131,6 +131,27 @@ class CompatTest(CIBaseTest):
self.perform_build_test(targets, compat_arch=True)


+class SbuildFlavor(CIBaseTest):
+ """
+ Test package build with a custom sbuild chroot.
+ :avocado: tags=sbuildflavor,fast
+ """
+
+ def test_sbuild_flavor(self):
+ targets = [
+ 'mc:qemuamd64-trixie:hello-isar',
+ 'mc:qemuarm64-trixie:hello-isar',
+ ]
+
+ lines = [
+ 'SBUILD_FLAVOR:hello-isar = "db2m"',
+ 'SBUILD_FLAVOR:libhello = "db2m"'
+ ]
+
+ self.init()
+ self.perform_build_test(targets, lines=lines)

[Felix Moessbauer]

We currently fail the test if the image is not available. Instead, we
now skip it and report what is missing. By that, we stay semantically
correct by not failing tests when pre-conditions are not fulfilled.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 614a3397..b64d808c 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -24,6 +24,7 @@ from utils import CIUtils
from avocado import Test
from avocado.utils import path
from avocado.utils import process
+from avocado.core import exceptions

sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))

@@ -703,6 +704,7 @@ class CIBuilder(Test):
keep=False,
):
time_to_wait = self.params.get('time_to_wait', default=DEF_VM_TO_SEC)
+ self.skip_if_vm_image_missing(arch, distro, image)

self.log.info("===================================================")
self.log.info(f"Running Isar VM boot test for ({distro}-{arch})")
@@ -800,3 +802,20 @@ class CIBuilder(Test):
self.vm_turn_off(vm)

return stdout, stderr
+
+ def skip_if_vm_image_missing(self, arch, distro, image):
+ (
+ image_fstypes,
+ image_fullname,
+ deploy_dir_image,
+ ) = CIUtils.getVars(
+ 'IMAGE_FSTYPES',
+ 'IMAGE_FULLNAME',
+ 'DEPLOY_DIR_IMAGE',
+ target=f"mc:qemu{arch}-{distro}:{image}",
+ )
+ image_type = image_fstypes.split()[0]
+ rootfs_image = f"{image_fullname}.{image_type}"
+ rootfs_image_path = os.path.join(deploy_dir_image, rootfs_image)
+ if not os.path.exists(rootfs_image_path):
+ raise exceptions.TestSkipError(f'VM image missing: {rootfs_image}')
--
2.51.0