[RFC][PATCH 0/3] Introduce container fetcher and pre-loader

[Jan Kiszka]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

This still needs at least documentation and CI hook-up, possibly more,
but I'd like to invite colleagues and the community in general to early
feedback.

BTW, patch 1 also introduces a fairly nice pattern to register a real
bitbake fetcher, rather than meddling with SRC_URI like we do for apt://
right now. Therefore, I'm planning to refactor the apt fetcher soon as
well, maybe even with progress reporting...

Jan

Jan Kiszka (3):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers

kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 10 +++
.../prebuilt-podman-img_0.1.bb | 10 +++
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 +++++++++++++++++++
.../container-loader/container-loader.inc | 76 ++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 11 +++
.../files/container-loader.sh.tmpl | 13 +++
.../container-loader/podman-loader.inc | 10 +++
12 files changed, 273 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
create mode 100644 meta/lib/container_fetcher.py
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 10 ++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 57 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
default "kas/package/pkg_sshd-regen-keys.yaml"
depends on PACKAGE_ISAR_CI_SSH_SETUP

+
+config PACKAGE_PREBUILT_DOCKER_IMG
+ bool "prebuilt-docker-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+ string
+ default "kas/package/pkg_prebuilt-docker-img.yaml"
+ depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+ bool "prebuilt-podman-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+ string
+ default "kas/package/pkg_prebuilt-podman-img.yaml"
+ depends on PACKAGE_PREBUILT_PODMAN_IMG
+
endmenu

config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..3f337d92
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;tag=latest \
+ "
--
2.43.0

[Jan Kiszka]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):

Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers

ci: Add test cases for container fetching and loading
doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 60 ++++++++++++

kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++

.../prebuilt-docker-img_0.1.bb | 12 +++
.../prebuilt-podman-img_0.1.bb | 10 ++
.../recipes-core/images/isar-image-ci.bb | 2 +

meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++

.../container-loader/container-loader.inc | 94 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
testsuite/citest.py | 21 +++++
15 files changed, 382 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+)
create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+import os
+import re
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd
+from bb.progress import LineFilterProgressHandler
+
+class SkopeoProgressHandler(LineFilterProgressHandler):
+ def __init__(self, d):
+ super(SkopeoProgressHandler, self).__init__(d)
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line)
+ if match:
+ state = match[0].split('/')
+ progress = (int(state[0]) * 100) / int(state[1])
+ self.update(progress)
+ return True
+
+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+
+ container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = container_name + \
+ ("@" + ud.digest if ud.digest else ":" + ud.tag)
+ ud.localname = container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.localname + \
+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag)
+
+ def download(self, ud, d):
+ progresshandler = SkopeoProgressHandler(d)
+ runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}",
+ d, log=progresshandler)
+
+ if ud.digest:
+ return
+
+ checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+ checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+ strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+ # If strict checking enabled and neither sum defined, raise error
+ if strict == "1":
+ raise NoChecksumError(checksum_line)
+
+ checksum_event = {"sha256sum": checksum}
+ bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+ if strict == "ignore":
+ return
+
+ # Log missing digest so user can more easily add it
+ logger.warning(
+ f"Missing checksum for '{ud.localpath}', consider using this " \
+ f"SRC_URI in the recipe:\n{checksum_line}")
+
+ def unpack(self, ud, rootdir, d):
+ arch = d.getVar('PACKAGE_ARCH')
+ variant_opt = ""
+ if arch == "armhf":
+ arch = "arm"
+ variant_opt = "--override-variant v7"
+ elif arch == "armel":
+ arch = "arm"
+ variant_opt = "--override-variant v6"
+ runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \
+ f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d)
+
+ def clean(self, ud, d):
+ bb.utils.remove(ud.localpath, recurse=True)
--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.../container-loader/container-loader.inc | 94 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++

5 files changed, 144 insertions(+)

create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..a0c2ddb3
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,94 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+
+CONTAINER_COMPRESSION ?= "zst"
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += " \
+ ${CONTAINER_ENGINE_PACKAGES} \
+ ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ ''}"
+
+CONTAINER_COMPRESSOR_CMD = "${@ \
+ 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+CONTAINER_DECOMPRESSOR_CMD = "${@ \
+ 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \
+ CONTAINER_DECOMPRESSOR_CMD \
+ CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+ import os
+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+
+ image_name = host + (path if path != "/" else "")
+ unpacked_image = workdir + "/" + image_name.replace('/', '.')
+ dest_dir = D + "/usr/share/" + BPN + "/images"
+ tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
+ docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
+
+ bb.utils.remove(tar_image)
+ cmd = f"skopeo copy dir:{unpacked_image} " \
+ f"docker-archive:{tar_image}:{image_name}{docker_ref}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
+ image_name + docker_ref
+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#

+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..733b2b30 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provides a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
+an when fetching as it also allows to validate its integrity. If tag is not
+specified, `latest` is used as tag name.
+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+
+The fetched container image are stored in a directory in the `WORKDIR` of the
+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them into an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates debian package with will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, thus ensure that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
+```
--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++

.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml

new file mode 100644

index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb

new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT

+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.../recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 21 +++++++++++++++++++
2 files changed, 23 insertions(+)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
self.init()
self.vm_start('mipsel','bookworm', image='isar-image-ci',
script='test_kernel_module.sh example_module')
+
+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
--
2.43.0

[Niedermayr, BENEDIKT]

On Mon, 2024-07-15 at 12:08 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
>  doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 60 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 776ae52c..733b2b30 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
>  
>  Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
>  To install additional packages into the sbuild chroot, add them to
> `SBUILD_CHROOT_PREINSTALL_EXTRA`.
> +
> +## Pre-install container images
> +
> +If an isar-generated image shall provides a container runtime, it may also be

If an isar-generated image shall provide a container runtime...

> +desirable to pre-install container images to avoid having to download them on
> +first boot or because they may not be accessible outside of the build
> +environment. Isar supports this scenario via two services, a container fetcher
> +and a container loader.
> +
> +### Bitbake fetcher for containers
> +
> +The bitbake fetching protocol "docker://" allows to download pre-built images
> +from container registries. The URL consists of the image path, followed by
> +a recommened digests in the form `digest=sha256:<sha256sum>` and an optional

a recommended...

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not

... to indentify the ?an? when fetching...

Benedikt

[Jan Kiszka]

Thanks for actually reading it :). Update will follow.

Jan

--
Siemens AG, Technology
Linux Expert Center

[Schmidt, Adriaan]

'Jan Kiszka' via isar-users <isar-...@googlegroups.com>, Montag, 15. Juli 2024 12:09:
> Subject: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader

recommended *digest

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not
> +specified, `latest` is used as tag name.
> +
> +When specifying a multi-arch image, the fetcher will download the images for
> +all available architectures. If this is not desired, directly specify the
> +digest of the desired architecture manifest instead of that the manifest list.
> +
> +The fetched container image are stored in a directory in the `WORKDIR` of the

*images

> +requesting recipe. When a multi-arch image was specified, only the image
> +matching `PACKAGE_ARCH` will be stored. The name of the image directory is
> +derived from the container image name, replacing all `/` with `.`.
> +
> +### Container loader helpers
> +
> +To create a Debian package which can carry container images and load them into
> +local storage of docker or podman, there is a set of helpers available. To use
> +them into an own recipe, add

them *in an own

> +`require recipes-support/container-loader/docker-loader.inc` when using docker
> +and `require recipes-support/container-loader/podman-loader.inc` when using
> +podman. The loader will try to transfer the packaged image into the container
> +runtime storage on boot, but only if no container image of the same name and
> +tag is present already.
> +
> +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container
> images
> +remain by default available and may be used again for loading the storage after
> +it may have been emptied later on (factory reset).
> +
> +Source container images may either be fetched as binaries from a registry, see
> +above, or built via isar as well.
> +
> +### Example
> +
> +This creates debian package with will download, package and then load the

*which will download

> +`debian:bookworm-20240701-slim` container image into the docker container
> +storage. The package will depend on `docker.io`, thus ensure that that basic

*ensuring?

Adriaan

> +runtime services are installed on the target as well. The packaged image will
> +be deleted from the target device's rootfs after successful import.
> +
> +```
> +require recipes-support/container-loader/docker-loader.inc
> +
> +CONTAINER_DELETE_AFTER_LOAD = "1"
> +
> +SRC_URI +=
> "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75
> 510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
> +```
> --
> 2.43.0
>

> --
> You received this message because you are subscribed to the Google Groups "isar-
> users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> users/2e1537004e7ad48f33cd3d3daf2b8f7c3294d31a.1721038111.git.jan.kiszka%40s
> iemens.com.

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.../container-loader/container-loader.inc | 101 ++++++++++++++++++

.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++

5 files changed, 151 insertions(+)

create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc

new file mode 100644
index 00000000..e97e829b
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,101 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF

+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc

new file mode 100644

index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc

@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl

new file mode 100644

index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#

+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc

new file mode 100644

index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc

@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

[Jan Kiszka]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v3:
- do not compress container deb packages [Benedikt]
- fix various spelling issues in the doc [Adriaan, Benedikt]

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers
ci: Add test cases for container fetching and loading

doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 60 +++++++++++

kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 12 +++
.../prebuilt-podman-img_0.1.bb | 10 ++
.../recipes-core/images/isar-image-ci.bb | 2 +
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 ++++++++++++++++

.../container-loader/container-loader.inc | 101 ++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++

testsuite/citest.py | 21 ++++
15 files changed, 389 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

create mode 100644 meta/lib/container_fetcher.py

create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+)
create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py

new file mode 100644

index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py

@@ -0,0 +1,90 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md

index 776ae52c..e97a2cd5 100644

--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+

+If an isar-generated image shall provide a container runtime, it may also be

+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by

+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.

+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+

+The fetched container images are stored in a directory in the `WORKDIR` of the

+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use

+them in an own recipe, add

+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+

+This creates a debian package which will download, package and then load the

+`debian:bookworm-20240701-slim` container image into the docker container

+storage. The package will depend on `docker.io`, insuring that that basic

+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++

5 files changed, 59 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml

new file mode 100644

index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb

new file mode 100644

index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb

@@ -0,0 +1,12 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT

+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

@@ -0,0 +1,10 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

[Niedermayr, BENEDIKT]

Add do_install_fetched_containers[vardeps] += "CONTAINER_COMPRESSOR_CMD" here.

By the way, for this to work the implementation in bitbake.conf needs to be fixed as well.
The vardepsexclude for ZSTD_DEFAULTS should not contain "ZSTD_LEVEL".

I could sent out a patch for this.

Benedikt

[MOESSBAUER, Felix]

On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:

> From: Jan Kiszka <jan.k...@siemens.com>
>
> This bitbake fetcher allows to pull container images from registries,
> store them in the download cache and transfer them into the workdir
> of
> recipes requesting the image. The format of the URL is
>
> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>
> Fetching without digest is supported but will cause a warning, just
> like
> downloading via wget without a checksum.

This is fine, as long as the upstream artifact is expected to be
stable.

>
> If tag is left out, "latest" is
> used.

The tag should be mandatory and it should be clear that - even without
digest - the artifact needs to be stable. Floating tags create all
kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
vote for just not allowing this.

>
> The fetcher will try to pull all available variants of a multi-arch
> image. If this is not needed, you can also directly specify the image
> digest of a specific architecture.

In most cases this does not make sense. I propose to always limit the
fetching to the current architecture. For that, we either need a
mapping between the debian architecture and the OCI architectures (what
is specified in the application/vnd.oci.image.index.v1+json manifest),
or we simply don't support index manifests at all and force people to
use image manifests (application/vnd.oci.image.manifest.v1+json).

Felix

[Jan Kiszka]

On 17.07.24 13:50, Moessbauer, Felix (T CED OES-DE) wrote:
> On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.k...@siemens.com>
>>
>> This bitbake fetcher allows to pull container images from registries,
>> store them in the download cache and transfer them into the workdir
>> of
>> recipes requesting the image. The format of the URL is
>>
>> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>>
>> Fetching without digest is supported but will cause a warning, just
>> like
>> downloading via wget without a checksum.
>
> This is fine, as long as the upstream artifact is expected to be
> stable.
>
>>
>> If tag is left out, "latest" is
>> used.
>
> The tag should be mandatory and it should be clear that - even without
> digest - the artifact needs to be stable. Floating tags create all
> kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
> vote for just not allowing this.

I cannot follow: Leaving out the tag has nothing to do with getting a
stable image. It may just lead to something being tagged "latest" on the
device that had a different tag (or none) in the registry.

>
>>
>> The fetcher will try to pull all available variants of a multi-arch
>> image. If this is not needed, you can also directly specify the image
>> digest of a specific architecture.
>
> In most cases this does not make sense. I propose to always limit the
> fetching to the current architecture. For that, we either need a
> mapping between the debian architecture and the OCI architectures (what
> is specified in the application/vnd.oci.image.index.v1+json manifest),
> or we simply don't support index manifests at all and force people to
> use image manifests (application/vnd.oci.image.manifest.v1+json).

Don't worry, I already have ideas of redesigning this into per-arch
fetches. Background is using less different formats on the build system
and, where possible, even hard-link between them to save space and time
with larger images.

Jan

[MOESSBAUER, Felix]

Ok, that means the image is just tagged on the device. Then it should
be fine.

Felix

[Jan Kiszka]

We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v4
- fetch per architecture, no longer full multi-arch
- directly archive and compress in download task
- use hard links where possible to reduce disk usage
- always use zstd, remove configurability

Changes in v3:
- do not compress container deb packages [Benedikt]
- fix various spelling issues in the doc [Adriaan, Benedikt]

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers
ci: Add test cases for container fetching and loading
doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 58 +++++++++++++

kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 12 +++

.../prebuilt-podman-img_0.1.bb | 10 +++

.../recipes-core/images/isar-image-ci.bb | 2 +
meta/classes/dpkg-base.bbclass | 6 ++

meta/lib/container_fetcher.py | 86 +++++++++++++++++++
.../container-loader/container-loader.inc | 73 ++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++

.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++

.../container-loader/podman-loader.inc | 10 +++
testsuite/citest.py | 21 +++++
15 files changed, 355 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

create mode 100644 meta/lib/container_fetcher.py

create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are

compressed with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.../container-loader/container-loader.inc | 73 +++++++++++++++++++

.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 12 +++

.../files/container-loader.sh.tmpl | 18 +++++
.../container-loader/podman-loader.inc | 10 +++
5 files changed, 123 insertions(+)

create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..5fd8d23c
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,73 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+

+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd"

+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \

+ CONTAINER_DELETE_AFTER_LOAD"

+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {

+ from oe.path import copyhardlink

+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+

+ tag = parm["tag"] if "tag" in parm else "latest"
+ image_name = host + (path if path != "/" else "")
+ image_file = image_name.replace('/', '.') + \
+ ":" + tag + ".zst"

+ dest_dir = D + "/usr/share/" + BPN + "/images"
+

+ copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file)
+
+ line = f"{image_file} {image_name}:{tag}"

+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build

+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+
+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc

new file mode 100644

index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc

@@ -0,0 +1,10 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl

new file mode 100644

index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755

index 00000000..2356e31c

--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#

+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then

+ pzstd -c -d /usr/share/${BPN}/images/"$image" | \

+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc

new file mode 100644

index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc

@@ -0,0 +1,10 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"

--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++

5 files changed, 59 insertions(+)

create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml

new file mode 100644

index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml

@@ -0,0 +1,9 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024

+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb

new file mode 100644

index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb

@@ -0,0 +1,12 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644

index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

doc/user_manual.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..2bdacbec 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,61 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provide a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.
+

+In case a multi-arch image is specified, the fetcher will only pull for the
+package architecture of the requesting recipe (`PACKAGE_ARCH`). The fetched
+images are stored as zstd-compressed in docker-archive format in the
+`WORKDIR` of the recipe. The name of the image is derived from the container
+image name, replacing all `/` with `.` and appending `:<tag>.zst`. Example:
+`docker://debian;tag=bookworm` will be saved as `debian:bookworm.zst`.

+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like

downloading via wget without a checksum. If tag is left out, "latest" is
used.

In case a multi-arch image is specified, the fetcher will only pull for

the package architecture of the requesting recipe. The image is stored
compressed in docker-archive format and, wherever possible, hard-linked
from DL_DIR to WORKDIR. Future versions may also introduce full

unpacking of the fetched container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)

create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py

new file mode 100644
index 00000000..0d659154
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,86 @@

+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

+import oe.path
+import os
+import tempfile

+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd

+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+

+ ud.arch = d.getVar('PACKAGE_ARCH')
+ ud.variant = None
+ if ud.arch == "armhf":
+ ud.arch = "arm"
+ ud.variant = "v7"
+ elif ud.arch == "armel":
+ ud.arch = "arm"
+ ud.variant = "v6"
+
+ ud.container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = ud.container_name + \

+ ("@" + ud.digest if ud.digest else ":" + ud.tag)

+ ud.localname = ud.container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.arch + "/" + \
+ (ud.variant + "/" if ud.variant else "") + ud.localname + \

+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \

+ ".zst"

+
+ def download(self, ud, d):

+ tarball = ud.localfile[:-len('.zst')]
+ with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
+ # Take a two steps for downloading into a docker archive because
+ # not all source may have the required Docker schema 2 manifest.
+ runfetchcmd("skopeo copy --preserve-digests " + \
+ f"--override-arch {ud.arch} " + \
+ (f"--override-variant {ud.variant} " if ud.variant else "") + \
+ f"docker://{ud.container_src} dir:{tmpdir}", d)
+ runfetchcmd(f"skopeo copy dir:{tmpdir} " + \
+ f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d)
+ zstd_defaults = d.getVar('ZSTD_DEFAULTS')
+ runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d)

+ image_file = ud.localname + ":" + ud.tag + ".zst"
+ oe.path.remove(rootdir + "/" + image_file)
+ oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)
--
2.43.0

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

[Niedermayr, BENEDIKT]

do_fetch doesn't get triggered if "ZSTD_LEVEL" changes, but it should, since the
output zst file changes with a different ZSTD_LEVEL.

Even adding a do_fetch[vardeps] += "ZSTD_DEFAULTS" doesn't lead to the desired behavior:

When ZSTD_LEVEL changes, the fetch task is run but does not actually fetch anything and
immediately proceeds with do_unpack. More precisely, the download() Method of the Container()
class is not getting executed, maybe due to some logic in the upper/surrounding Fetch() class.
That would be fine if the compression task didn't implement the compression...

Another thing to mention is that he whole fetch task is run again only if the compression changes.
Skopeo seems to override already downloaded layers rather than skipping them, which means that
changing the compression also means a complete new download process.

Maybe the unpack() task for packing and hardlinking the images would be better, even though it
sounds strange to put an compression task into the unpack task.

Benedikt

[Niedermayr, BENEDIKT]

Sorry wrong wording:

That would be fine if the FETCH task didn't implement the compression...

Benedikt

[Uladzimir Bely]

This appears to fail in CI:

[stdlog] 2024-07-30 07:29:37,721 avocado.app ERROR| qemu-system-
aarch64: -drive
file=/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm6
4/isar-image-ci-debian-bookworm-
qemuarm64.ext4,if=none,format=raw,id=hd0: Could not open
'/workspace/build/isar_ub_devel/49/build/tmp/deploy/images/qemuarm64/is
ar-image-ci-debian-bookworm-qemuarm64.ext4': No such file or directory

Should be either changed the bullseye here (not yet checked if it
works). Or, we need to build arm64-bullseye image in build tests
before.

> --
> 2.43.0
>

--
Best regards,
Uladzimir.

[Jan Kiszka]

bullseye is legacy.

Unfortunately, the testsuite lacks a dependency model. Here,
VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
was not the case so far.

I'm still wondering why we are using such an unhandy and non-standard
test framework if not even such basic things are expressed - or are even
expressible?

Jan

[Uladzimir Bely]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
.../recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 24 +++++++++++++++++++
2 files changed, 26 insertions(+)

This is a drop-in replacement of patch 4 from "[PATCH v4 0/5] Introduce
container fetcher and pre-loader" series:
- Fixed syntax errors (incorrectly escaped '\$')
- Fixed long lines in order to pass flake8

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb

index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"

+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"

+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7064c1e4..4a248a49 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -609,3 +609,27 @@ class VmBootTestFull(CIBaseTest):
image='isar-image-ci',
script='test_kernel_module.sh example_module',
)

+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start(

+ 'amd64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'PATH=$PATH:/usr/sbin;'
+ 'for n in $(seq 30);'
+ ' do docker images | grep -q alpine && break; sleep 10; done;'

+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && '

+ 'for n in $(seq 30);'
+ ' do podman images | grep -q alpine && break; sleep 10; done;'

+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start(

+ 'arm64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'PATH=$PATH:/usr/sbin;'
+ 'for n in $(seq 30);'
+ ' do docker images | grep -q alpine && break; sleep 10; done;'

+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && '

+ 'for n in $(seq 30);'
+ ' do podman images | grep -q alpine && break; sleep 10; done;'

+ 'podman run --rm quay.io/libpod/alpine:latest true\'')

--
2.44.2

[Uladzimir Bely]

Hello.

I rebased the patchset on top of "[PATCH] testsuite: Build bookworm
arm64 image in no-cross test". So, now this test doesn't fail this way.

But there were other issues related to the backslashed '\$'. I couldn't
catch this until applied one more useful patch: "[PATCH] testsuite:
Save ssh command stdout/stderr in case of failure" that allowed to see
syntax error here.

Also, flake8 appeared to recognize this as "W605 invalid escape
sequence '\$'".

I've just sent a replacement of the patch that fixes the issue and
python style here.

You could prepare pathset v5, or I could merge it as is after testing.

> Unfortunately, the testsuite lacks a dependency model. Here,
> VmBootTestFull depends on CrossTest and NoCrossTest - apparently this
> was not the case so far.
>
> I'm still wondering why we are using such an unhandy and non-standard
> test framework if not even such basic things are expressed - or are
> even
> expressible?
>
> Jan
>

--
Best regards,
Uladzimir.

[Jan Kiszka]

On 05.08.24 09:16, Uladzimir Bely wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> This plugs the two example recipes for loading container images into
> VM-based testing. The test consists of running 'true' in the installed
> alpine images.
>
> Rather than enabling the ci user to do password-less sudo, this uses su
> with the piped-in password. Another trick needed is to poll for the
> images because loading is performed asynchronously.
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
> ---
> .../recipes-core/images/isar-image-ci.bb | 2 ++
> testsuite/citest.py | 24 +++++++++++++++++++
> 2 files changed, 26 insertions(+)
>
> This is a drop-in replacement of patch 4 from "[PATCH v4 0/5] Introduce
> container fetcher and pre-loader" series:
> - Fixed syntax errors (incorrectly escaped '\$')

IIRC, we do need the escape inside the shell (sh -c '...'). So, you
likely rather need to escape the escape character.

Jan

[Uladzimir Bely]

I just tried to make a simple check:

```
$ su -c 'for i in $(seq 3); do echo $i; done'
Password:
1
2
3

$ su -c 'for i in \$(seq 3); do echo $i; done'
Password:
bash: -c: line 1: syntax error near unexpected token `('
bash: -c: line 1: `for i in \$(seq 3); do echo $i; done'

$ su -c 'for i in \\$(seq 3); do echo $i; done'
Password:
\1
2
3
```

We are likely don't need escaping at all.

Anyway, we could just convert the tests from "cmd=<long_command"
to "script=test_prebuild_container.sh" and have test logic in a human-
readable form.

Best regards,
Uladzimir.

[Jan Kiszka]

Interesting - anyway, if this sequence is not properly resolved, the
test will fail. And I assume you had it running successfully, so we must
be fine.

>
> Anyway, we could just convert the tests from "cmd=<long_command"
> to "script=test_prebuild_container.sh" and have test logic in a human-
> readable form.
>

Also fine with me.

Jan

[Uladzimir Bely]

OK, I've already prepared the script internally and will check in CI
with it.

--
Best regards,
Uladzimir.

[Uladzimir Bely]

... and still having problems with running commands inside arm64
container.

I manually run (with same command-line as CI does) qemuamd64 and
qemuarm64 images.

Running prebuilt container in amd64 machine works well:

```
root@isar:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/alpine 3.10.2 961769676411 4 years ago 5.58MB
root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
[ 61.233873] docker0: port 1(veth1c2b6f9) entered blocking state
[ 61.234280] docker0: port 1(veth1c2b6f9) entered disabled state
[ 61.240243] device veth1c2b6f9 entered promiscuous mode
[ 62.650328] eth0: renamed from veth2aff680
[ 62.664713] IPv6: ADDRCONF(NETDEV_CHANGE): veth1c2b6f9: link becomes
ready
[ 62.665407] docker0: port 1(veth1c2b6f9) entered blocking state
[ 62.665656] docker0: port 1(veth1c2b6f9) entered forwarding state
[ 62.666394] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes
ready
[ 63.220542] docker0: port 1(veth1c2b6f9) entered disabled state
[ 63.229530] veth2aff680: renamed from eth0
[ 63.308290] docker0: port 1(veth1c2b6f9) entered disabled state
[ 63.311282] device veth1c2b6f9 left promiscuous mode
[ 63.311507] docker0: port 1(veth1c2b6f9) entered disabled state
root@isar:~# echo $?
0
root@isar:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/alpine latest 961769676411 4 years ago 5.85 MB
root@isar:~# podman run --rm quay.io/libpod/alpine:latest true
[ 78.274955] cni-podman0: port 1(vethf6fde03e) entered blocking state
[ 78.275225] cni-podman0: port 1(vethf6fde03e) entered disabled state
[ 78.277667] device vethf6fde03e entered promiscuous mode
[ 78.626628] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 78.627038] IPv6: ADDRCONF(NETDEV_CHANGE): vethf6fde03e: link
becomes ready
[ 78.627313] cni-podman0: port 1(vethf6fde03e) entered blocking state
[ 78.627513] cni-podman0: port 1(vethf6fde03e) entered forwarding
state
[ 79.690462] audit: type=1400 audit(1722919083.116:6):
apparmor="STATUS" operation="profile_load" profile="unconfined"
name="containers-default-0.50.1" pid=750 comm="apparmor_parser"
[ 80.574314] cni-podman0: port 1(vethf6fde03e) entered disabled state
[ 80.575874] device vethf6fde03e left promiscuous mode
[ 80.576060] cni-podman0: port 1(vethf6fde03e) entered disabled state
root@isar:~# echo $?
0
```

The same under arm64 fails:

```
root@isar:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/alpine 3.10.2 915beeae4675 4 years ago 5.33MB
root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true
[ 407.689016] docker0: port 1(veth81a2857) entered blocking state
[ 407.689231] docker0: port 1(veth81a2857) entered disabled state
[ 407.698637] device veth81a2857 entered promiscuous mode
[ 410.003030] eth0: renamed from vethbe8a124
[ 410.026357] IPv6: ADDRCONF(NETDEV_CHANGE): veth81a2857: link becomes
ready
[ 410.026727] docker0: port 1(veth81a2857) entered blocking state
[ 410.026872] docker0: port 1(veth81a2857) entered forwarding state
[ 410.767475] docker0: port 1(veth81a2857) entered disabled state
[ 410.788277] vethbe8a124: renamed from eth0
[ 410.941958] docker0: port 1(veth81a2857) entered disabled state
[ 410.944534] device veth81a2857 left promiscuous mode
[ 410.944676] docker0: port 1(veth81a2857) entered disabled state
docker: Error response from daemon: failed to create shim task: OCI
runtime create failed: runc create failed: unable to start container
process: exec: "true": executable file not found in $PATH: unknown.
root@isar:~# echo $?
127
root@isar:~# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/alpine latest 915beeae4675 4 years ago 5.59 MB
root@isar:~# podman run --rm quay.io/libpod/alpine:latest true
[ 423.567388] cni-podman0: port 1(veth29135974) entered blocking state
[ 423.567593] cni-podman0: port 1(veth29135974) entered disabled state
[ 423.569719] device veth29135974 entered promiscuous mode
[ 423.754420] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 423.754765] IPv6: ADDRCONF(NETDEV_CHANGE): veth29135974: link
becomes ready
[ 423.755036] cni-podman0: port 1(veth29135974) entered blocking state
[ 423.755183] cni-podman0: port 1(veth29135974) entered forwarding
state
[ 426.090252] cni-podman0: port 1(veth29135974) entered disabled state
[ 426.098292] device veth29135974 left promiscuous mode
[ 426.098455] cni-podman0: port 1(veth29135974) entered disabled state
Error: runc: runc create failed: unable to start container process:
exec: "true": executable file not found in $PATH: OCI runtime attempted
to invoke a command that was not found
root@isar:~# echo $?
127
```

At first glance this looks like arm64 images are not functional.
Continue debugging.


--
Best regards,
Uladzimir.

[Uladzimir Bely]

After some debugging I can see that something makes docker prebuilt
image inside qemu broken. But removing it from and loading to docker
engine again helps:

```
root@isar:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
quay.io/libpod/alpine 3.10.2 915beeae4675 4 years ago 5.33MB

root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true

[ 902.770874] docker0: port 1(veth8275b2c) entered blocking state
[ 902.771066] docker0: port 1(veth8275b2c) entered disabled state
[ 902.777051] device veth8275b2c entered promiscuous mode
[ 904.813519] eth0: renamed from veth2f2256f
[ 904.830269] IPv6: ADDRCONF(NETDEV_CHANGE): veth8275b2c: link becomes
ready
[ 904.830857] docker0: port 1(veth8275b2c) entered blocking state
[ 904.830997] docker0: port 1(veth8275b2c) entered forwarding state
[ 904.831407] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes
ready
[ 905.372753] docker0: port 1(veth8275b2c) entered disabled state
[ 905.385163] veth2f2256f: renamed from eth0
[ 905.487707] docker0: port 1(veth8275b2c) entered disabled state
[ 905.491396] device veth8275b2c left promiscuous mode
[ 905.491533] docker0: port 1(veth8275b2c) entered disabled state

docker: Error response from daemon: failed to create shim task: OCI
runtime create failed: runc create failed: unable to start container
process: exec: "true": executable file not found in $PATH: unknown.

ERRO[0003] error waiting for container: context canceled

root@isar:~# echo $?
127

root@isar:~# docker image rm 915beeae4675
Untagged: quay.io/libpod/alpine:3.10.2
Deleted:
sha256:915beeae46751fc564998c79e73a1026542e945ca4f73dc841d09ccc6c2c0672
Deleted:
sha256:5e0d8111135538b8a86ce5fc969849efce16c455fd016bb3dc53131bcedc4da5

root@isar:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE

root@isar:~# pzstd -c -d /usr/share/prebuilt-docker-
img/images/quay.io.libpod.alpine\:3.10.2.zst | docker load
/usr/share/prebuilt-docker-img/images/quay.io.libpod.alpine:3.10.2.zst:
5598720 bytes
5e0d81111355: Loading layer 5.59MB/5.59MB
Loaded image: quay.io/libpod/alpine:3.10.2

root@isar:~# docker run --rm quay.io/libpod/alpine:3.10.2 true

[ 1023.800568] docker0: port 1(veth3eb45d3) entered blocking state
[ 1023.800790] docker0: port 1(veth3eb45d3) entered disabled state
[ 1023.805585] device veth3eb45d3 entered promiscuous mode
[ 1025.295999] eth0: renamed from veth7e4183e
[ 1025.310388] IPv6: ADDRCONF(NETDEV_CHANGE): veth3eb45d3: link becomes
ready
[ 1025.310681] docker0: port 1(veth3eb45d3) entered blocking state
[ 1025.310801] docker0: port 1(veth3eb45d3) entered forwarding state
[ 1025.979813] docker0: port 1(veth3eb45d3) entered disabled state
[ 1025.990858] veth7e4183e: renamed from eth0
[ 1026.087161] docker0: port 1(veth3eb45d3) entered disabled state
[ 1026.088367] device veth3eb45d3 left promiscuous mode
[ 1026.088471] docker0: port 1(veth3eb45d3) entered disabled state

root@isar:~# echo $?
0

```

This looks strange. Nothing changed (image hash is the same), but the
second run works well. After rebooting qemu machine it still works.

Podman prebuilt image looks unaffected - it works from the beginning.

--
Best regards,
Uladzimir.

[Jan Kiszka]

Strange, all that used to work. You manually reproduced this as well,
not only via the testsuite, right? Let me test again locally...

[Uladzimir Bely]

For manual tests I used images taken from CI (that failed). As I could
see, the issue in my case was caused by zero-size "/bin/busybox"
somewhere in /var/lib/docker/overlay2/. The file was broken and
reinstalling the container fixed this.

But I guess this was caused by already "spoiled" image that was tested
in CI. When I just built (on a local machine) a new image and didn't
try to run qemu with it (e.g., didn't modify it), manual running docker
image in it worked well.. The busybox binary from alpine container was
OK in that case.

Continue debugging ...

--
Best regards,
Uladzimir.

[Uladzimir Bely]

So, there was my logical error I did in the test script. After polling
for docker images I wrongly got an error code so "docker run" was not
even started. This made CI test fail, qemu machine was interrupted and
this broke busybox binary (ext4 was not synced). So, on the next
(manual) boot it had a size 0 and nothing worked.

Currently I have a proper script that was run OK at least on three
different build machines, so I'll resend new patch soon.

--
Best regards,
Uladzimir.

[Uladzimir Bely]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta-test/recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 10 ++++++++++
testsuite/scripts/test_prebuilt_containers.sh | 16 ++++++++++++++++
3 files changed, 28 insertions(+)
create mode 100755 testsuite/scripts/test_prebuilt_containers.sh

This is a drop-in replacement of patch 4 from "[PATCH v4 0/5] Introduce
container fetcher and pre-loader" series:
- Fixed syntax errors (incorrectly escaped '\$')

- Fixed long lines in order to pass flake8

Changes since v1:
- Switched from running command to script for better readability.

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb

index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"

+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"

+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 7064c1e4..87f34846 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -609,3 +609,13 @@ class VmBootTestFull(CIBaseTest):

image='isar-image-ci',
script='test_kernel_module.sh example_module',
)
+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()

+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ script='test_prebuilt_containers.sh')

+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()

+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ script='test_prebuilt_containers.sh')
diff --git a/testsuite/scripts/test_prebuilt_containers.sh b/testsuite/scripts/test_prebuilt_containers.sh
new file mode 100755
index 00000000..1700e653
--- /dev/null
+++ b/testsuite/scripts/test_prebuilt_containers.sh
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+echo root | su -c '\
+ set -e
+ export PATH=$PATH:/usr/sbin
+ for n in $(seq 30); do
+ docker images | grep -q alpine && break
+ sleep 10
+ done
+ docker run --rm quay.io/libpod/alpine:3.10.2 true
+ for n in $(seq 30); do
+ podman images | grep -q alpine && break
+ sleep 10
+ done
+ podman run --rm quay.io/libpod/alpine:latest true
+'
--
2.44.2

[Jan Kiszka]

Nice!

Reviewed-by: Jan Kiszka <jan.k...@siemens.com>

Thanks,

[Uladzimir Bely]

From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
meta-test/recipes-core/images/isar-image-ci.bb | 2 ++

testsuite/citest.py | 13 ++++++++++++-
testsuite/scripts/test_prebuilt_containers.sh | 16 ++++++++++++++++
3 files changed, 30 insertions(+), 1 deletion(-)

create mode 100755 testsuite/scripts/test_prebuilt_containers.sh


This is a drop-in replacement of patch 4 from "[PATCH v4 0/5] Introduce
container fetcher and pre-loader" series:
- Fixed syntax errors (incorrectly escaped '\$')
- Fixed long lines in order to pass flake8

Changes since v2:
- Run the tests in the VM that was previously run. This:
- reduces test execution time;
- avoids filesystem failures in the VM image caused by previously
killed machine.

Changes since v1:
- Switched from running command to script for better readability.


diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py

index 7064c1e4..18d3af97 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -539,6 +539,7 @@ class VmBootTestFull(CIBaseTest):

'bookworm',
image='isar-image-ci',

script='test_kernel_module.sh example_module',
+ keep=True,
)

def test_i386_buster(self):
@@ -577,7 +578,7 @@ class VmBootTestFull(CIBaseTest):

def test_amd64_bookworm(self):
self.init()
- self.vm_start('amd64', 'bookworm', image='isar-image-ci')
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci', keep=True)

def test_arm_bookworm(self):
self.init()
@@ -609,3 +610,13 @@ class VmBootTestFull(CIBaseTest):

--
2.44.2

[Uladzimir Bely]

On Fri, 2024-07-19 at 18:38 +0200, 'Jan Kiszka' via isar-users wrote:

Applied to next (w/o patch 4 for CI), thanks.

--
Best regards,
Uladzimir.

[Uladzimir Bely]

Applied to next.

--
Best regards,
Uladzimir.