[PATCH 0/3] Add 'make world' test case

[Anton Mikanovich]

To be able to test all the targets currently supported by Isar we need
to add separate test case which can build (and run in qemu if possible)
everything with single command.

There are 2 running options possible:

1. Run all the targets with single bitbake command.
This is the fastest way with sharing build dirs between targets.

2. Run every target in separate build dir, one target per testcase.
This allows to check every target separately, but requires external
yaml with target list and varianter plugin for avocado.

Also fix container example-module usage by default for targets, so this
machine can be built without excluding example-module manually.

Anton Mikanovich (3):
CI: Remove container package list hack from testsuite
CI: Add make world test case
CI: Run startvm in world test case

.gitignore | 2 +-
meta-isar/conf/local.conf.sample | 4 +
meta-test/conf/local.conf.sample | 3 +
scripts/ci_build.sh | 8 +-
testsuite/cibuilder.py | 4 -
testsuite/citest.py | 55 +++++++++++++
testsuite/data/targets.yml | 132 +++++++++++++++++++++++++++++++
testsuite/targets_gen.py | 13 +++
testsuite/utils.py | 35 +++++++-
9 files changed, 249 insertions(+), 7 deletions(-)
create mode 100644 testsuite/data/targets.yml
create mode 100755 testsuite/targets_gen.py

--
2.34.1

[Anton Mikanovich]

Add test case which builds all available targets.

Single bitbake usage (directly from mc.conf):
$ avocado run testsuite/citest.py -t world

Multiple bitbakes usage (from generated yaml file):
$ avocado run testsuite/citest.py -t world -m testsuite/data/targets.yml

Or the same with ci_build.sh:

There is also a script to regenerate yaml from mc.conf:
$ ./testsuite/utils/targets_gen.py

This test case requires avocado-framework-plugin-varianter-yaml-to-mux.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
.gitignore | 2 +-
scripts/ci_build.sh | 8 ++-
testsuite/citest.py | 33 ++++++++++
testsuite/data/targets.yml | 132 +++++++++++++++++++++++++++++++++++++
testsuite/targets_gen.py | 13 ++++
testsuite/utils.py | 35 +++++++++-
6 files changed, 220 insertions(+), 3 deletions(-)

create mode 100644 testsuite/data/targets.yml
create mode 100755 testsuite/targets_gen.py

diff --git a/.gitignore b/.gitignore
index 81c09fc1..fff155f1 100644
--- a/.gitignore
+++ b/.gitignore
@@ -3,4 +3,4 @@
__pycache__
.config.yaml*
.kas_shell_history
-build/
+build*
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 338557ad..690dadd4 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -23,6 +23,7 @@ if ! command -v avocado > /dev/null; then
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install avocado-framework==103.0
+ pip install avocado-framework-plugin-varianter-yaml-to-mux==103.0
fi

# Get Avocado build tests path
@@ -49,6 +50,7 @@ show_help() {
echo " -b, --base BASE_DIR set path to base directory. If not set,"
echo " the tests will be started in current path."
echo " -d, --debug enable debug bitbake output."
+ echo " -m, --muxfile input file for yaml-to-mux plugin."
echo " -T, --tags specify basic avocado tags."
echo " --help display this message and exit."
echo
@@ -94,6 +96,10 @@ do
NORUN="1"
echo "warning: deprecated parameter '$key', consider using '-T <TAG>,-startvm' instead"
;;
+ -m|--muxfile)
+ MUXFILE="-m $2"
+ shift
+ ;;
-t|--timeout)
TIMEOUT="-p time_to_wait=$2"
shift
@@ -149,7 +155,7 @@ set -x

avocado ${VERBOSE} run "${TESTSUITE_DIR}/citest.py" \
-t "${TAGS}" --max-parallel-tasks=1 --disable-sysinfo \
- ${SSTATE} ${TIMEOUT} \
+ ${SSTATE} ${MUXFILE} ${TIMEOUT} \
|| ret=$?

python3 ${TESTSUITE_DIR}/cleanup.py
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 9f18a51f..501dbf37 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -1,4 +1,10 @@
#!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2022-2025 ilbers GmbH
+# Copyright (C) 2022-2025 Siemens AG
+#
+# SPDX-License-Identifier: MIT

from avocado import skipUnless
from avocado.core import exceptions
@@ -734,3 +740,30 @@ class VmBootTestFull(CIBaseTest):
self.init()
self.vm_start('amd64-iso', 'bookworm', image='isar-image-ci',
script='test_system_running.sh 30')
+
+
+class World(CIBaseTest):
+
+ """
+ All targets build test
+
+ :avocado: tags=world
+ """
+
+ def test_world(self):
+ name = self.params.get('name')
+ image = self.params.get('image', default='isar-image-ci')
+ targets = []
+
+ if name is None:
+ self.init()
+ for target in CIUtils.get_targets():
+ for image in CIUtils.get_test_images():
+ targets.append(f'mc:{target}:{image}')
+ else:
+ targets.append(f'mc:{name}:{image}')
+ self.init(f'build-{name}')
+
+ self.perform_build_test(
+ targets, container=name and name.startswith('container')
+ )
diff --git a/testsuite/data/targets.yml b/testsuite/data/targets.yml
new file mode 100644
index 00000000..5c9b5196
--- /dev/null
+++ b/testsuite/data/targets.yml
@@ -0,0 +1,132 @@
+a: !mux
+ qemuarm-buster:
+ name: qemuarm-buster
+ qemuarm-bullseye:
+ name: qemuarm-bullseye
+ qemuarm-bookworm:
+ name: qemuarm-bookworm
+ qemuarm-trixie:
+ name: qemuarm-trixie
+ qemuarm64-buster:
+ name: qemuarm64-buster
+ qemuarm64-bullseye:
+ name: qemuarm64-bullseye
+ qemuarm64-bookworm:
+ name: qemuarm64-bookworm
+ qemuarm64-trixie:
+ name: qemuarm64-trixie
+ qemui386-buster:
+ name: qemui386-buster
+ qemui386-bullseye:
+ name: qemui386-bullseye
+ qemui386-bookworm:
+ name: qemui386-bookworm
+ qemuamd64-buster:
+ name: qemuamd64-buster
+ qemuamd64-bullseye:
+ name: qemuamd64-bullseye
+ qemuamd64-sb-bullseye:
+ name: qemuamd64-sb-bullseye
+ qemuamd64-bookworm:
+ name: qemuamd64-bookworm
+ qemuamd64-cip-bookworm:
+ name: qemuamd64-cip-bookworm
+ qemuamd64-iso-bookworm:
+ name: qemuamd64-iso-bookworm
+ qemuamd64-trixie:
+ name: qemuamd64-trixie
+ container-amd64-buster:
+ name: container-amd64-buster
+ container-amd64-bullseye:
+ name: container-amd64-bullseye
+ container-amd64-bookworm:
+ name: container-amd64-bookworm
+ qemumipsel-bullseye:
+ name: qemumipsel-bullseye
+ qemumipsel-bookworm:
+ name: qemumipsel-bookworm
+ qemuriscv64-trixie:
+ name: qemuriscv64-trixie
+ bananapi-buster:
+ name: bananapi-buster
+ bananapi-bullseye:
+ name: bananapi-bullseye
+ bananapi-bookworm:
+ name: bananapi-bookworm
+ beagleplay-bookworm:
+ name: beagleplay-bookworm
+ de0-nano-soc-buster:
+ name: de0-nano-soc-buster
+ de0-nano-soc-bullseye:
+ name: de0-nano-soc-bullseye
+ de0-nano-soc-bookworm:
+ name: de0-nano-soc-bookworm
+ hikey-bullseye:
+ name: hikey-bullseye
+ hikey-bookworm:
+ name: hikey-bookworm
+ imx6-sabrelite-buster:
+ name: imx6-sabrelite-buster
+ imx6-sabrelite-bullseye:
+ name: imx6-sabrelite-bullseye
+ phyboard-mira-bullseye:
+ name: phyboard-mira-bullseye
+ phyboard-mira-bookworm:
+ name: phyboard-mira-bookworm
+ nanopi-neo-buster:
+ name: nanopi-neo-buster
+ nanopi-neo-bullseye:
+ name: nanopi-neo-bullseye
+ nanopi-neo-bookworm:
+ name: nanopi-neo-bookworm
+ nanopi-neo-efi-bookworm:
+ name: nanopi-neo-efi-bookworm
+ stm32mp15x-bullseye:
+ name: stm32mp15x-bullseye
+ virtualbox-bullseye:
+ name: virtualbox-bullseye
+ virtualbox-bookworm:
+ name: virtualbox-bookworm
+ x86-pc-bookworm:
+ name: x86-pc-bookworm
+ rpi-arm-bullseye:
+ name: rpi-arm-bullseye
+ rpi-arm-bookworm:
+ name: rpi-arm-bookworm
+ rpi-arm-v7-bullseye:
+ name: rpi-arm-v7-bullseye
+ rpi-arm-v7-bookworm:
+ name: rpi-arm-v7-bookworm
+ rpi-arm-v7l-bullseye:
+ name: rpi-arm-v7l-bullseye
+ rpi-arm-v7l-bookworm:
+ name: rpi-arm-v7l-bookworm
+ rpi-arm64-v8-bullseye:
+ name: rpi-arm64-v8-bullseye
+ rpi-arm64-v8-bookworm:
+ name: rpi-arm64-v8-bookworm
+ sifive-fu540-trixie:
+ name: sifive-fu540-trixie
+ starfive-visionfive2-trixie:
+ name: starfive-visionfive2-trixie
+ qemuarm64-focal:
+ name: qemuarm64-focal
+ qemuarm64-jammy:
+ name: qemuarm64-jammy
+ qemuarm64-noble:
+ name: qemuarm64-noble
+ qemuamd64-focal:
+ name: qemuamd64-focal
+ qemuamd64-jammy:
+ name: qemuamd64-jammy
+ qemuamd64-noble:
+ name: qemuamd64-noble
+ qemuamd64-sid:
+ name: qemuamd64-sid
+ qemuarm64-sid:
+ name: qemuarm64-sid
+b: !mux
+ base:
+ image: isar-image-base
+ ci:
+ image: isar-image-ci
diff --git a/testsuite/targets_gen.py b/testsuite/targets_gen.py
new file mode 100755
index 00000000..41b4d644
--- /dev/null
+++ b/testsuite/targets_gen.py
@@ -0,0 +1,13 @@
+#!/usr/bin/env python3
+
+"""
+# This software is a part of Isar.
+# Copyright (C) 2025 ilbers GmbH
+
+# targets_gen.py: Generates yaml for yaml-to-mux Avocado varianter plugin.
+"""
+
+from utils import CIUtils
+
+if __name__ == "__main__":
+ CIUtils.gen_targets_yaml()
diff --git a/testsuite/utils.py b/testsuite/utils.py
index ffcb5cc7..9b18999b 100755
--- a/testsuite/utils.py
+++ b/testsuite/utils.py
@@ -1,4 +1,9 @@
#!/usr/bin/env python3
+#
+# This software is a part of ISAR.
+# Copyright (C) 2024-2025 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT

import os
import sys
@@ -10,8 +15,10 @@ sys.path.append(os.path.join(os.path.dirname(__file__), '../bitbake/lib'))
import bb
import bb.tinfoil

+isar_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))

-class CIUtils():
+
+class CIUtils:
@staticmethod
def getVars(*vars, target=None):
def fixStream(stream):
@@ -53,3 +60,29 @@ class CIUtils():
return tar.getnames()
except Exception:
return []
+
+ @staticmethod
+ def get_test_images():
+ return ['isar-image-base', 'isar-image-ci']
+
+ @staticmethod
+ def get_targets():
+ d = bb.data.init()
+ d.setVar('BBPATH', os.path.join(isar_root, 'meta-isar'))
+ d = bb.cookerdata.parse_config_file('conf/mc.conf', d, False)
+ return d.getVar('BBMULTICONFIG').split()
+
+ @staticmethod
+ def gen_targets_yaml(fn='targets.yml'):
+ targetsfile = os.path.join(os.path.dirname(__file__), 'data', fn)
+ with open(targetsfile, 'w') as f:
+ f.write('a: !mux\n')
+ for target in CIUtils.get_targets():
+ f.write(f' {target}:\n name: {target}\n')
+ f.write('b: !mux\n')
+ prefix = 'isar-image-'
+ for image in CIUtils.get_test_images():
+ nodename = image
+ if image.startswith(prefix):
+ nodename = image[len(prefix):]
+ f.write(f' {nodename}:\n image: {image}\n')
--
2.34.1

[Anton Mikanovich]

As IMAGE_INSTALL list is set in local.conf removing items should be
also done in local.conf.
This will allow to build container targets just like any other.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 4 ++++
meta-test/conf/local.conf.sample | 3 +++
testsuite/cibuilder.py | 4 ----
3 files changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 52ea2fb5..9bc95dfd 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -154,6 +154,10 @@ CONF_VERSION = "1"
# The default list of extra packages to be installed.
IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"

+#
+# Container machines don't need example module and enable-fsck.
+IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
+
#
# Machines with secure boot should use signed modules.
IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index f692f533..092d6ba6 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -30,6 +30,9 @@ MIRRORS += "https?://cdn\.kernel\.org/.* https://mirrors.edge.kernel.org/PATH"
# The default list of extra packages
IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"

+# Container machines don't need example module and enable-fsck.
+IMAGE_INSTALL:remove:container-amd64 = "example-module-${KERNEL_NAME} enable-fsck"
+
# Machines with secure boot should use signed modules
IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index c2520f7b..7d702b35 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -199,10 +199,6 @@ class CIBuilder(Test):
f.write('BB_NO_NETWORK = "1"\n')
if container:
f.write('SDK_FORMATS = "docker-archive"\n')
- f.write(
- 'IMAGE_INSTALL:remove = '
- '"example-module-${KERNEL_NAME} enable-fsck"\n'
- )
if gpg_pub_key:
f.write('BASE_REPO_KEY="file://' + gpg_pub_key + '"\n')
if wic_deploy_parts:
--
2.34.1

[Anton Mikanovich]

To cover also QEMU running for qemu- targets.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 501dbf37..e7ba1720 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -767,3 +767,25 @@ class World(CIBaseTest):
self.perform_build_test(

targets, container=name and name.startswith('container')

)
+
+ def test_runworld(self):

+ name = self.params.get('name')
+ image = self.params.get('image', default='isar-image-ci')
+ targets = []
+
+ if name is None:
+ self.init()
+ for target in CIUtils.get_targets():
+ for image in CIUtils.get_test_images():
+ targets.append(f'mc:{target}:{image}')
+ else:
+ targets.append(f'mc:{name}:{image}')
+ self.init(f'build-{name}')
+

+ for target in targets:
+ t_list = target.split(':', 2)
+ if not t_list[1].startswith('qemu'):
+ self.cancel(f"{t_list[1]} skipped as non qemu")
+ distro = t_list[1].split('-')[-1]
+ arch = t_list[1].removeprefix('qemu').removesuffix(f"-{distro}")
+ self.vm_start(arch, distro, image=t_list[2])
--
2.34.1

[Baurzhan Ismagulov]

On 2025-05-12 11:32, Anton Mikanovich wrote:
> To be able to test all the targets currently supported by Isar we need
> to add separate test case which can build (and run in qemu if possible)
> everything with single command.

Applied to next, thanks.

With kind regards,
Baurzhan