[PATCH 0/6] Various improvements to the testsuite

[Felix Moessbauer]

This series aims at simplifying and speeding up the testsuite execution.
This should enable contributors to actually run it and test their patches.

As pointed out in [1], the testsuite currently tests various features over
and over again, while others are not tested at all. We try to improve this
by focusing on "feature" tests, whereby each test case checks a set of
features instead of a whole image. We further identified long running tasks
(like the kernel builds) and move them to a dedicated test.

The execution time reduction varies greatly depending on the hardware,
network and sstate usage. When applying these changes on recently added
patches, the time was often decreased by more than 50% on "fast".

Note, that more refactorings of the testsuite are needed - and planned,
but this series should already address the biggest pain points.

[1] https://groups.google.com/g/isar-users/c/uZBTIHzLp8Q/m/8k0L1vfMDAAJ

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (6):
testsuite: move targets with custom kernel to separate test
testsuite: enable ccache on kernel tests
testsuite: make prebuilt container a feature test
testsuite: make compat test standalone test
testsuite: handle IMAGE_INSTALL solely in cibuilder.py
testsuite: limit cross_debsrc test to subset of packages

meta-test/conf/local.conf.sample | 10 ----
.../recipes-core/images/isar-image-ci.bb | 2 -
testsuite/cibuilder.py | 24 ++++++--
testsuite/citest.py | 56 +++++++++++++++++--
4 files changed, 71 insertions(+), 21 deletions(-)

--
2.51.0

[Felix Moessbauer]

By moving these tests out of the cross test (which is a fast test), we
significantly reduce the test time on fast.

The tests are now placed in the KernelTests class in a dedicated test,
suitable for applying further optimizations (separate commit).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
testsuite/citest.py | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 76a3df69..cb63a8fe 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -190,7 +190,8 @@ class InstallerTest(CIBaseTest):
class CrossTest(CIBaseTest):

"""
- Start cross build for the defined set of configurations
+ Start cross build for the defined set of configurations.
+ As this is a fast test, ensure to not depend on custom kernels.

:avocado: tags=cross,fast
"""
@@ -199,12 +200,9 @@ class CrossTest(CIBaseTest):
targets = [
'mc:qemuarm-buster:isar-image-ci',
'mc:qemuarm-bullseye:isar-image-ci',
- 'mc:de0-nano-soc-bullseye:isar-image-base',
- 'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',
- 'mc:phyboard-mira-bookworm:isar-image-base',
]

self.init()
@@ -273,6 +271,17 @@ class KernelTests(CIBaseTest):
:avocado: tags=kernel,full
"""

+ def test_kernel_cross(self):
+ """Targets that build a custom kernel"""
+ targets = [
+ 'mc:de0-nano-soc-bullseye:isar-image-base',
+ 'mc:stm32mp15x-bullseye:isar-image-base',
+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)
+
def test_per_kernel(self):
"""Test per-kernel recipe variants for external kernel modules."""

--
2.51.0

[Felix Moessbauer]

The kernels are anyways usually built with ccache enabled to reduce the
buildtime. By enabling it in the test as well, we can significantly
bring down test time on repeated builds.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index cb63a8fe..cba3c008 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -280,7 +280,7 @@ class KernelTests(CIBaseTest):
]

self.init()
- self.perform_build_test(targets)
+ self.perform_build_test(targets, ccache=True)

[Felix Moessbauer]

This reworks how we handle feature tests: Instead of binding a feature
to a target via appends in isar-image-ci, we implement the feature by
using the test setup function. By that, we have fine grained control over
where to test a feature and can avoid testing the same feature over and
over again. This leads to a much cleaner architecture and faster test
execution.

We start implementing this approach with the prebuilt container test.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-core/images/isar-image-ci.bb | 2 --
testsuite/citest.py | 17 +++++++++++++++++
2 files changed, 17 insertions(+), 2 deletions(-)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index 7f2b404d..58aa5738 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,7 +16,6 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
-IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.zst tar.zst"
@@ -52,4 +51,3 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
-IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index cba3c008..601eca92 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -265,6 +265,23 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, lines=lines)

+class PrebuiltTest(CIBaseTest):
+ """
+ Tests associated with prebuilt artifacts (containers, debs).
+ :avocado: tags=prebuilt,fast
+ """
+
+ def test_prebuilt_containers(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:isar-image-ci',
+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ ]
+
+ self.init()
+ # perform imaging as well, as needed by VmBootTestFull
+ self.perform_build_test(targets, image_install="prebuilt-docker-img prebuilt-podman-img")
+
+
class KernelTests(CIBaseTest):
"""
Tests associated with kernel builds and development.
--
2.51.0

[Felix Moessbauer]

Currently, enabling compat mode during configuration also changes the
set of packages we build. As this is done with appends, some packages
are unconditionally added, despite not being part of the original test
case. We change this by only enabling compat mode when selecting compat.
To actually test the compat feature, we add a simple and fast test that
builds the hello-isar-compat package.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 4 ----
testsuite/citest.py | 19 +++++++++++++++++++
2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 46bfc546..25e3b6cb 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -180,11 +180,7 @@ class CIBuilder(Test):
if compat_arch:
f.write(
'ISAR_ENABLE_COMPAT_ARCH:amd64 = "1"\n'
- 'IMAGE_INSTALL:remove:amd64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:amd64 = " hello-isar-compat"\n'
'ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n'
- 'IMAGE_INSTALL:remove:arm64 = "hello-isar"\n'
- 'IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n'
)
if not cross:
f.write('ISAR_CROSS_COMPILE = "0"\n')
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 601eca92..aee138bc 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -112,6 +112,25 @@ class DevTest(CIBaseTest):
self.vm_start('arm', 'bookworm', skip_modulecheck=True)


+class CompatTest(CIBaseTest):
+ """
+ Test compilation of recipes for compat architecture.
+ This also tests a custom sbuild chroot for compat.
+ :avocado: tags=compat,fast
+ """
+
+ def test_compat_recipe(self):
+ targets = [
+ 'mc:qemuamd64-bookworm:hello-isar-compat',
+ 'mc:qemuarm64-bookworm:hello-isar-compat',
+ 'mc:qemuamd64-trixie:hello-isar-compat',
+ 'mc:qemuarm64-trixie:hello-isar-compat',
+ ]
+
+ self.init()
+ self.perform_build_test(targets, compat_arch=True)
+
+
class ReproTest(CIBaseTest):

"""
--
2.51.0

[Felix Moessbauer]

By moving the IMAGE_INSTALL parts out of the ci local conf sample, we
make this part reusable across tests. By that, tests can easily
customize this list without either taking it as-is or completely
replacing it. We further get rid of overrides that are not used in the
CI anyways.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/conf/local.conf.sample | 10 ----------
testsuite/cibuilder.py | 20 ++++++++++++++++++++
2 files changed, 20 insertions(+), 10 deletions(-)

diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index 092d6ba6..862bea47 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -27,16 +27,6 @@ BB_DISKMON_DIRS = "\
MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"

-# The default list of extra packages
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
-
-# Container machines don't need example module and enable-fsck.
-IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
-
-# Machines with secure boot should use signed modules
-IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
-IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
-
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 25e3b6cb..26002ade 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -30,6 +30,18 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
import bb

DEF_VM_TO_SEC = 600
+IMAGE_INSTALL_DEFAULT = [
+ 'hello-isar',
+ 'example-raw',
+ 'example-module-${KERNEL_NAME}',
+ 'enable-fsck',
+ 'isar-exclude-docs',
+ 'samefile',
+ 'hello',
+ 'isar-disable-apt-cache',
+ 'cowsay',
+ 'example-prebuilt'
+]

isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
backup_prefix = '.ci-backup'
@@ -222,6 +234,14 @@ class CIBuilder(Test):
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+ else:
+ if container:
+ # strip kernel modules from default package install list
+ _image_install = [p for p in IMAGE_INSTALL_DEFAULT if "-module-" not in p]
+ else:
+ _image_install = IMAGE_INSTALL_DEFAULT
+ f.write('IMAGE_INSTALL = "%s"\n' % ' '.join(_image_install))
+
if fail_on_cleanup == '1':
f.write('ISAR_FAIL_ON_CLEANUP = "1"\n')
if installer_image:
--
2.51.0

[Felix Moessbauer]

To test if the debsrc infrastructure works it is sufficient to build a
single package. This significantly speeds up the CI and reduce the space
needed.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/citest.py | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index aee138bc..ab509e22 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -233,7 +233,8 @@ class CrossTest(CIBaseTest):
]

self.init()
- self.perform_build_test(targets, debsrc_cache=True)
+ # only build a single custom package to speedup test
+ self.perform_build_test(targets, debsrc_cache=True, image_install='cowsay')

def test_cross_trixie(self):
targets = [
--
2.51.0