[PATCH 0/2] Enable SState cache in GitLab CI

[Felix Moessbauer]

By running the GitLab CI jobs with SState cache enabled, huge speedups
can be achieved. Example:

Rebuild with 100% cache hits:

+ avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1
JOB ID : 3070455808c8c6c4fa78adac9e4c4508fe001069
JOB LOG : /build/build/job-results/job-2025-12-17T16.59-3070455/job.log
(1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED
(1/6) /build/testsuite/citest.py:DevTest.test_dev: PASS (449.91 s)
(2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED
(2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: PASS (307.69 s)
(3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED
(3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: PASS (249.28 s)
(4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
(4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (96.88 s)
(5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
(5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (61.12 s)
(6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
(6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: PASS (64.81 s)
RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 1233.72 s

Rebuild without cache:

+ avocado run /build/testsuite/citest.py -t dev --max-parallel-tasks=1 --disable-sysinfo -p sstate=1
JOB ID : 38c1d4fa508ab774c7774ad404a3575343c59979
JOB LOG : /build/build/job-results/job-2025-12-17T13.29-38c1d4f/job.log
(1/6) /build/testsuite/citest.py:DevTest.test_dev: STARTED
(1/6) /build/testsuite/citest.py:DevTest.test_dev: PASS (1559.37 s)
(2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: STARTED
(2/6) /build/testsuite/citest.py:DevTest.test_dev_apps: PASS (1429.77 s)
(3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: STARTED
(3/6) /build/testsuite/citest.py:DevTest.test_dev_rebuild: PASS (383.64 s)
(4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: STARTED
(4/6) /build/testsuite/citest.py:DevTest.test_dev_run_amd64_bookworm: PASS (99.56 s)
(5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: STARTED
(5/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm64_bookworm: PASS (62.24 s)
(6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: STARTED
(6/6) /build/testsuite/citest.py:DevTest.test_dev_run_arm_bookworm: PASS (65.56 s)
RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME : 3604.21 s

Note: the measurements are on top of the "Various improvements to the testsuite" series,
but the patches also work independently.

Best regards,
Felix Moessbauer

Felix Moessbauer (2):
testsuite: forward SSTATE_MIRRORS into CI env on sstate
ci: add support for sstate cache

.gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------
testsuite/cibuilder.py | 2 ++
2 files changed, 30 insertions(+), 6 deletions(-)

--
2.51.0

[Felix Moessbauer]

The gitlab CI jobs should finish in a timely manner to give quick
feedback to the developer. By running the tests with sstate cache,
simple changes require far less CI time.

We prepare the gitlab ci job description to pick up the sstate cache
configuration from the environment and enable the caching for all jobs
(we always enable it, even if running without sstate cache, as then the
cache is simply thrown away later on). We further introduce a (manual)
info task to show the state of the cache, as well as a cleanup task to
drain the cache.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
.gitlab-ci.yml | 34 ++++++++++++++++++++++++++++------
1 file changed, 28 insertions(+), 6 deletions(-)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1a7abcb8..9bd8ff07 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -12,6 +12,14 @@ variables:
- export ftp_proxy=$FTP_PROXY
- export no_proxy=$NO_PROXY
- export DISTRO_APT_PREMIRRORS=$DISTRO_APT_PREMIRRORS
+ after_script:
+ - |
+ if [[ -n "${SSTATE_LOCATION}" ]] && [[ -d "sstate-cache" ]]; then
+ echo "=== Upload sstate artifacts to ${SSTATE_LOCATION} ==="
+ ./scripts/isar-sstate upload "sstate-cache" "${SSTATE_LOCATION}"
+ ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+ fi
+
artifacts:
name: "logs-$CI_JOB_ID"
paths:
@@ -48,7 +56,7 @@ dev-ci:
- *use-default-image
- if: $TESTSUITE == 'dev' || $CI_PIPELINE_SOURCE != 'schedule'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev

fast-ci:
<<: *common-build
@@ -56,7 +64,7 @@ fast-ci:
- *use-default-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast

full-ci:
<<: *common-build
@@ -68,7 +76,7 @@ full-ci:
- PREVIOUS_SHA="$(cat .CI_COMMIT_SHA || true)"
- if [ "$CI_COMMIT_SHA" != "$PREVIOUS_SHA" ]; then
echo "$CI_COMMIT_SHA" > .CI_COMMIT_SHA;
- scripts/ci_build.sh -T full;
+ scripts/ci_build.sh --sstate 1 -T full;
fi
cache:
key: "$CI_COMMIT_REF_SLUG"
@@ -83,7 +91,7 @@ dev-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'dev'
script:
- - scripts/ci_build.sh -T dev
+ - scripts/ci_build.sh --sstate 1 -T dev

fast-ci-isar:
<<: *docker-isar
@@ -92,7 +100,7 @@ fast-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'fast'
script:
- - scripts/ci_build.sh -T fast
+ - scripts/ci_build.sh --sstate 1 -T fast

full-ci-isar:
<<: *docker-isar
@@ -101,4 +109,18 @@ full-ci-isar:
- *use-docker-isar-image
- if: $TESTSUITE == 'full'
script:
- - scripts/ci_build.sh -T full
+ - scripts/ci_build.sh --sstate 1 -T full
+
+sstate-cache-info:
+ stage: build
+ when: manual
+ script:
+ - ./scripts/isar-sstate info -v "${SSTATE_LOCATION}"
+
+sstate-cache-clean:
+ stage: build
+ when: manual
+ variables:
+ SSTATE_MAX_AGE: "0d"
+ script:
+ - ./scripts/isar-sstate clean "${SSTATE_LOCATION}" --max-age "${SSTATE_MAX_AGE}"
--
2.51.0

[Felix Moessbauer]

When running the testsuite with sstate caching enabled and also the
environment variable SSTATE_MIRRORS is set, add this to the local conf,
so the CI can also use remote sstate caches (important for CI).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

testsuite/cibuilder.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 26002ade..310a3836 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -232,6 +232,8 @@ class CIBuilder(Test):
f.write('DL_DIR = "%s"\n' % dl_dir)
if sstate_dir:
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
+ if sstate and 'SSTATE_MIRRORS' in os.environ:
+ f.write('SSTATE_MIRRORS = "%s"\n' % os.environ['SSTATE_MIRRORS'])
if image_install is not None:
f.write('IMAGE_INSTALL = "%s"\n' % image_install)
else:
--
2.51.0

[MOESSBAUER, Felix]

We need to increase the timeout of the after_script, as otherwise not
all artifacts might be uploaded on larger runs like "-t fast". We need
to add [1]

variables:
RUNNER_AFTER_SCRIPT_TIMEOUT: 15m

I'll add it in case a v2 is needed. Otherwise it can also be added
later one, when more evidence around resonable timeouts are gathered.

[1]
https://docs.gitlab.com/ci/runners/configure_runners/#set-script-and-after_script-timeouts

Felix

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany