[PATCH 0/4] Speedup testsuites

[Anton Mikanovich]

Isar CI running takes too much time already, so speed up optimizations
need to be done. Remove some test case duplications and minimize time
consuming parts keeping the same test coverage.

On the test machine improvement is ~50%:
dev: 30min -> 22min
fast: 168min -> 113min
full: TBD

Based on `Additional CI improvements` patchset.

Anton Mikanovich (4):
CI: Remove cleanup before nocross build
CI: Minimize debsrc_cache checking
CI: Minimize kselftest checking
CI: Minimize testsuites duplication

testsuite/cibuilder.py | 1 -
testsuite/citest.py | 64 +++++++++++++++++++++++++++++++++---------
2 files changed, 50 insertions(+), 15 deletions(-)

--
2.34.1

[Anton Mikanovich]

Long time ago Isar was failing if running QEMU building right after
cross build test case. To mask that failure cleanup between nocross
and cross test cases was introduced. There is no more such an issue,
so we can safely remove this hack from the testsuite.

If nocross building will not work after switching from cross mode it
should be considered as a bug.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
testsuite/citest.py | 2 --
1 file changed, 2 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 18d3af97..3f98fb3b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -230,8 +230,6 @@ class NoCrossTest(CIBaseTest):
]

self.init()
- # Cleanup after cross build
- self.move_in_build_dir('tmp', 'tmp_before_nocross')
self.perform_build_test(targets, cross=False, debsrc_cache=True)

def test_nocross_rpi(self):
--
2.34.1

[Anton Mikanovich]

Enabling debsrc_cache for such a big set of targets is not needed for
checking this functionality. Speedup testsuites by enabling
debsrc_cache partially.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 38 ++++++++++++++++++++++++++++++--------
1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3f98fb3b..336eea4e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -137,12 +137,19 @@ class CrossTest(CIBaseTest):
'mc:de0-nano-soc-bullseye:isar-image-base',
'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
- 'mc:qemuarm64-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',
'mc:phyboard-mira-bookworm:isar-image-base',
]

+ self.init()
+ self.perform_build_test(targets)
+
+ def test_cross_debsrc(self):
+ targets = [
+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ ]
+
self.init()
self.perform_build_test(targets, debsrc_cache=True)

@@ -153,7 +160,7 @@ class CrossTest(CIBaseTest):

self.init()
try:
- self.perform_build_test(targets, debsrc_cache=True)
+ self.perform_build_test(targets)
except exceptions.TestFail:
self.cancel('KFAIL')

@@ -174,7 +181,6 @@ class WicTest(CIBaseTest):
self.perform_wic_partition_test(
targets,
wic_deploy_parts=False,
- debsrc_cache=True,
compat_arch=False,
)

@@ -186,7 +192,6 @@ class WicTest(CIBaseTest):
self.perform_wic_partition_test(
targets,
wic_deploy_parts=True,
- debsrc_cache=True,
compat_arch=False,
)

@@ -218,17 +223,24 @@ class NoCrossTest(CIBaseTest):
'mc:bananapi-bookworm:isar-image-base',
'mc:nanopi-neo-bullseye:isar-image-base',
'mc:nanopi-neo-bookworm:isar-image-base',
- 'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuamd64-focal:isar-image-ci',
'mc:qemuamd64-bookworm:isar-image-ci',
- 'mc:qemuarm-bookworm:isar-image-ci',
'mc:qemui386-bookworm:isar-image-base',
'mc:qemumipsel-bookworm:isar-image-ci',
'mc:hikey-bookworm:isar-image-base',
- 'mc:de0-nano-soc-bookworm:isar-image-base',
'mc:beagleplay-bookworm:isar-image-base',
]

+ self.init()
+ self.perform_build_test(targets, cross=False)
+
+ def test_nocross_debsrc(self):
+ targets = [
+ 'mc:qemuarm-bookworm:isar-image-ci',
+ 'mc:stm32mp15x-bullseye:isar-image-base',
+ 'mc:de0-nano-soc-bookworm:isar-image-base',
+ ]
+
self.init()
self.perform_build_test(targets, cross=False, debsrc_cache=True)

@@ -238,10 +250,20 @@ class NoCrossTest(CIBaseTest):
'mc:rpi-arm-v7-bullseye:isar-image-base',
'mc:rpi-arm-v7l-bullseye:isar-image-base',
'mc:rpi-arm64-v8-bullseye:isar-image-base',
+ 'mc:rpi-arm64-v8-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ try:
+ self.perform_build_test(targets, cross=False)
+ except exceptions.TestFail:
+ self.cancel('KFAIL')
+
+ def test_nocross_rpi_debsrc(self):
+ targets = [
'mc:rpi-arm-bookworm:isar-image-base',
'mc:rpi-arm-v7-bookworm:isar-image-base',
'mc:rpi-arm-v7l-bookworm:isar-image-base',
- 'mc:rpi-arm64-v8-bookworm:isar-image-base',
]

self.init()
--
2.34.1

[Anton Mikanovich]

There is no need in building kselftest for all the targets.
Move it to the separate test case.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/cibuilder.py | 1 -
testsuite/citest.py | 16 ++++++++++++++++
2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3f77cd42..2311631d 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -167,7 +167,6 @@ class CIBuilder(Test):
f.write('ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n')
f.write('IMAGE_INSTALL:remove:arm64 = "hello-isar"\n')
f.write('IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n')
- f.write('IMAGE_INSTALL += "kselftest"\n')
if cross:
f.write('ISAR_CROSS_COMPILE = "1"\n')
f.write(
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 336eea4e..1f7e854d 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -153,6 +153,22 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, debsrc_cache=True)

+ def test_cross_kselftest(self):
+ targets = [
+ 'mc:qemuarm-buster:isar-image-ci',
+ 'mc:qemuarm-bullseye:isar-image-ci',
+ 'mc:de0-nano-soc-bullseye:isar-image-base',

+ 'mc:stm32mp15x-bullseye:isar-image-base',
+ 'mc:qemuarm-bookworm:isar-image-ci',

+ 'mc:qemuarm64-bookworm:isar-image-ci',
+ 'mc:qemuarm64-focal:isar-image-base',
+ 'mc:nanopi-neo-efi-bookworm:isar-image-base',
+ 'mc:phyboard-mira-bookworm:isar-image-base',
+ ]
+
+ self.init()
+ self.perform_build_test(targets, image_install="kselftest")
+
def test_cross_rpi(self):
targets = [
'mc:rpi-arm-v7-bullseye:isar-image-base',
--
2.34.1

[Anton Mikanovich]

Separate test cases between fast and full testsuites to avoid
duplication.
From now on full testsuite will not also execute dev and fast
test cases. Container test cases will be executed only in full.
The same coverage should be done by running several testsuites.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 1f7e854d..278e0bd3 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -23,7 +23,7 @@ class DevTest(CIBaseTest):
"""
Developer's test

- :avocado: tags=dev,fast,full
+ :avocado: tags=dev,fast
"""

def test_dev(self):
@@ -127,7 +127,7 @@ class CrossTest(CIBaseTest):
"""
Start cross build for the defined set of configurations

- :avocado: tags=cross,fast,full
+ :avocado: tags=cross,fast
"""

def test_cross(self):
@@ -320,7 +320,7 @@ class ContainerImageTest(CIBaseTest):
"""
Test containerized images creation

- :avocado: tags=containerbuild,fast,full,container
+ :avocado: tags=containerbuild,full,container
"""

@skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
@@ -340,7 +340,7 @@ class ContainerSdkTest(CIBaseTest):
"""
Test SDK container image creation

- :avocado: tags=containersdk,fast,full,container
+ :avocado: tags=containersdk,full,container
"""

@skipUnless(UMOCI_AVAILABLE and SKOPEO_AVAILABLE, 'umoci/skopeo not found')
--
2.34.1

[Anton Mikanovich]

Isar CI running takes too much time already, so speed up optimizations
need to be done. Remove some test case duplications and minimize time
consuming parts keeping the same test coverage.

On the test machine improvement is ~30-50%:

dev: 30min -> 22min
fast: 168min -> 113min

full: 636min -> 483min

Based on `Additional CI improvements` patchset.

Changes since v1:
- Fixed kselftest test case.
- Add full testsuite time improve statistics.

[Anton Mikanovich]

Long time ago Isar was failing if running QEMU building right after
cross build test case. To mask that failure cleanup between nocross
and cross test cases was introduced. There is no more such an issue,
so we can safely remove this hack from the testsuite.

If nocross building will not work after switching from cross mode it
should be considered as a bug.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 2 --
1 file changed, 2 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 18d3af97..3f98fb3b 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Anton Mikanovich]

Enabling debsrc_cache for such a big set of targets is not needed for
checking this functionality. Speedup testsuites by enabling
debsrc_cache partially.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 38 ++++++++++++++++++++++++++++++--------
1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 3f98fb3b..336eea4e 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

@@ -137,12 +137,19 @@ class CrossTest(CIBaseTest):
'mc:de0-nano-soc-bullseye:isar-image-base',
'mc:stm32mp15x-bullseye:isar-image-base',
'mc:qemuarm-bookworm:isar-image-ci',
- 'mc:qemuarm64-bookworm:isar-image-ci',
'mc:qemuarm64-focal:isar-image-base',
'mc:nanopi-neo-efi-bookworm:isar-image-base',

'mc:phyboard-mira-bookworm:isar-image-base',
]

+ self.init()
+ self.perform_build_test(targets)
+
+ def test_cross_debsrc(self):

+ targets = [

+ targets = [

+ 'mc:qemuarm-bookworm:isar-image-ci',
+ 'mc:stm32mp15x-bullseye:isar-image-base',

+ 'mc:de0-nano-soc-bookworm:isar-image-base',
+ ]
+
self.init()
self.perform_build_test(targets, cross=False, debsrc_cache=True)

@@ -238,10 +250,20 @@ class NoCrossTest(CIBaseTest):
'mc:rpi-arm-v7-bullseye:isar-image-base',
'mc:rpi-arm-v7l-bullseye:isar-image-base',
'mc:rpi-arm64-v8-bullseye:isar-image-base',

+ 'mc:rpi-arm64-v8-bookworm:isar-image-base',
+ ]
+
+ self.init()

[Anton Mikanovich]

There is no need in building kselftest for all the targets.
Move it to the separate test case.

Use the same targets set as cross build test case.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/cibuilder.py | 1 -
testsuite/citest.py | 16 ++++++++++++++++
2 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 3f77cd42..2311631d 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -167,7 +167,6 @@ class CIBuilder(Test):
f.write('ISAR_ENABLE_COMPAT_ARCH:arm64 = "1"\n')
f.write('IMAGE_INSTALL:remove:arm64 = "hello-isar"\n')
f.write('IMAGE_INSTALL:append:arm64 = " hello-isar-compat"\n')
- f.write('IMAGE_INSTALL += "kselftest"\n')
if cross:
f.write('ISAR_CROSS_COMPILE = "1"\n')
f.write(

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 336eea4e..29641d38 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

@@ -153,6 +153,22 @@ class CrossTest(CIBaseTest):
self.init()
self.perform_build_test(targets, debsrc_cache=True)

+ def test_cross_kselftest(self):

+ targets = [
+ 'mc:qemuarm-buster:kselftest',
+ 'mc:qemuarm-bullseye:kselftest',
+ 'mc:de0-nano-soc-bullseye:kselftest',
+ 'mc:stm32mp15x-bullseye:kselftest',
+ 'mc:qemuarm-bookworm:kselftest',
+ 'mc:qemuarm64-bookworm:kselftest',
+ 'mc:qemuarm64-focal:kselftest',
+ 'mc:nanopi-neo-efi-bookworm:kselftest',
+ 'mc:phyboard-mira-bookworm:kselftest',
+ ]
+
+ self.init()
+ self.perform_build_test(targets)

[Anton Mikanovich]

Separate test cases between fast and full testsuites to avoid
duplication.
From now on full testsuite will not also execute dev and fast
test cases. Container test cases will be executed only in full.
The same coverage should be done by running several testsuites.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

testsuite/citest.py | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/testsuite/citest.py b/testsuite/citest.py
index 29641d38..82affb1a 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py

[Uladzimir Bely]

Applied to next.

--
Best regards,
Uladzimir.