[RFC][PATCH 0/3] Introduce container fetcher and pre-loader

28 views
Skip to first unread message

Jan Kiszka

unread,
Jul 9, 2024, 1:31:44 PMJul 9
to isar-users
We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

This still needs at least documentation and CI hook-up, possibly more,
but I'd like to invite colleagues and the community in general to early
feedback.

BTW, patch 1 also introduces a fairly nice pattern to register a real
bitbake fetcher, rather than meddling with SRC_URI like we do for apt://
right now. Therefore, I'm planning to refactor the apt fetcher soon as
well, maybe even with progress reporting...

Jan

Jan Kiszka (3):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers

kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 10 +++
.../prebuilt-podman-img_0.1.bb | 10 +++
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 +++++++++++++++++++
.../container-loader/container-loader.inc | 76 ++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 11 +++
.../files/container-loader.sh.tmpl | 13 +++
.../container-loader/podman-loader.inc | 10 +++
12 files changed, 273 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
create mode 100644 meta/lib/container_fetcher.py
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

Jan Kiszka

unread,
Jul 9, 2024, 1:31:44 PMJul 9
to isar-users
From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 10 ++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 57 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 35ba7cf1..395c3a08 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -146,6 +146,25 @@ config KAS_INCLUDE_PACKAGE_ISAR_CI_SSH_SETUP
default "kas/package/pkg_sshd-regen-keys.yaml"
depends on PACKAGE_ISAR_CI_SSH_SETUP

+
+config PACKAGE_PREBUILT_DOCKER_IMG
+ bool "prebuilt-docker-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_DOCKER_IMG
+ string
+ default "kas/package/pkg_prebuilt-docker-img.yaml"
+ depends on PACKAGE_PREBUILT_DOCKER_IMG
+
+config PACKAGE_PREBUILT_PODMAN_IMG
+ bool "prebuilt-podman-img"
+ default y
+
+config KAS_INCLUDE_PACKAGE_PREBUILT_PODMAN_IMG
+ string
+ default "kas/package/pkg_prebuilt-podman-img.yaml"
+ depends on PACKAGE_PREBUILT_PODMAN_IMG
+
endmenu

config KAS_IMAGE_PREINSTALL
diff --git a/kas/package/pkg_prebuilt-docker-img.yaml b/kas/package/pkg_prebuilt-docker-img.yaml
new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..3f337d92
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/podman-loader.inc
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;tag=latest \
+ "
--
2.43.0

Jan Kiszka

unread,
Jul 15, 2024, 6:08:36 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers
ci: Add test cases for container fetching and loading
doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 60 ++++++++++++
kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 12 +++
.../prebuilt-podman-img_0.1.bb | 10 ++
.../recipes-core/images/isar-image-ci.bb | 2 +
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++
.../container-loader/container-loader.inc | 94 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
testsuite/citest.py | 21 +++++
15 files changed, 382 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

Jan Kiszka

unread,
Jul 15, 2024, 6:08:36 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+)
create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import os
+import re
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd
+from bb.progress import LineFilterProgressHandler
+
+class SkopeoProgressHandler(LineFilterProgressHandler):
+ def __init__(self, d):
+ super(SkopeoProgressHandler, self).__init__(d)
+ self._fire_progress(0)
+
+ def writeline(self, line):
+ match = re.findall(r'^Copying image .*\(([0-9]+/[0-9]+)\)$', line)
+ if match:
+ state = match[0].split('/')
+ progress = (int(state[0]) * 100) / int(state[1])
+ self.update(progress)
+ return True
+
+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+
+ container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = container_name + \
+ ("@" + ud.digest if ud.digest else ":" + ud.tag)
+ ud.localname = container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.localname + \
+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag)
+
+ def download(self, ud, d):
+ progresshandler = SkopeoProgressHandler(d)
+ runfetchcmd(f"skopeo copy --preserve-digests --all docker://{ud.container_src} dir:{ud.localfile}",
+ d, log=progresshandler)
+
+ if ud.digest:
+ return
+
+ checksum = bb.utils.sha256_file(ud.localpath + "/manifest.json")
+ checksum_line = f"SRC_URI = \"{ud.url};digest=sha256:{checksum}\""
+
+ strict = d.getVar("BB_STRICT_CHECKSUM") or "0"
+
+ # If strict checking enabled and neither sum defined, raise error
+ if strict == "1":
+ raise NoChecksumError(checksum_line)
+
+ checksum_event = {"sha256sum": checksum}
+ bb.event.fire(MissingChecksumEvent(ud.url, **checksum_event), d)
+
+ if strict == "ignore":
+ return
+
+ # Log missing digest so user can more easily add it
+ logger.warning(
+ f"Missing checksum for '{ud.localpath}', consider using this " \
+ f"SRC_URI in the recipe:\n{checksum_line}")
+
+ def unpack(self, ud, rootdir, d):
+ arch = d.getVar('PACKAGE_ARCH')
+ variant_opt = ""
+ if arch == "armhf":
+ arch = "arm"
+ variant_opt = "--override-variant v7"
+ elif arch == "armel":
+ arch = "arm"
+ variant_opt = "--override-variant v6"
+ runfetchcmd(f"skopeo --override-arch {arch} {variant_opt} " \
+ f"copy dir:{ud.localpath} dir:{rootdir + '/' + ud.localname}", d)
+
+ def clean(self, ud, d):
+ bb.utils.remove(ud.localpath, recurse=True)
--
2.43.0

Jan Kiszka

unread,
Jul 15, 2024, 6:08:36 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
.../container-loader/container-loader.inc | 94 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
5 files changed, 144 insertions(+)
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..a0c2ddb3
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,94 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+
+CONTAINER_COMPRESSION ?= "zst"
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += " \
+ ${CONTAINER_ENGINE_PACKAGES} \
+ ${@', gzip' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ ', zstd' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ', xz-utils' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ ''}"
+
+CONTAINER_COMPRESSOR_CMD = "${@ \
+ 'gzip -f -9 -n --rsyncable' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -f ${XZ_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'zstd -f --rm ${ZSTD_DEFAULTS}' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+CONTAINER_DECOMPRESSOR_CMD = "${@ \
+ 'gzip -c -d -n' if d.getVar('CONTAINER_COMPRESSION') == 'gz' else \
+ 'xz -c -d -T0' if d.getVar('CONTAINER_COMPRESSION') == 'xz' else \
+ 'pzstd -c -d' if d.getVar('CONTAINER_COMPRESSION') == 'zst' else \
+ ''}"
+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \
+ CONTAINER_DECOMPRESSOR_CMD \
+ CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+ import os
+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+
+ image_name = host + (path if path != "/" else "")
+ unpacked_image = workdir + "/" + image_name.replace('/', '.')
+ dest_dir = D + "/usr/share/" + BPN + "/images"
+ tar_image = dest_dir + "/" + image_name.replace('/', '.') + ".tar"
+ docker_ref = ":" + parm["tag"] if "tag" in parm else "latest"
+
+ bb.utils.remove(tar_image)
+ cmd = f"skopeo copy dir:{unpacked_image} " \
+ f"docker-archive:{tar_image}:{image_name}{docker_ref}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ cmd = f"{d.getVar('CONTAINER_COMPRESSOR_CMD')} {tar_image}"
+ bb.note(f"running: {cmd}")
+ bb.process.run(cmd)
+
+ line = f"{os.path.basename(tar_image)}.{d.getVar('CONTAINER_COMPRESSION')} " + \
+ image_name + docker_ref
+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
--
2.43.0

Jan Kiszka

unread,
Jul 15, 2024, 6:08:37 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..733b2b30 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provides a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
+an when fetching as it also allows to validate its integrity. If tag is not
+specified, `latest` is used as tag name.
+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+
+The fetched container image are stored in a directory in the `WORKDIR` of the
+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them into an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates debian package with will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, thus ensure that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
+```
--
2.43.0

Jan Kiszka

unread,
Jul 15, 2024, 6:08:37 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

Jan Kiszka

unread,
Jul 15, 2024, 6:08:37 AM (13 days ago) Jul 15
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
.../recipes-core/images/isar-image-ci.bb | 2 ++
testsuite/citest.py | 21 +++++++++++++++++++
2 files changed, 23 insertions(+)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index e5d51e6e..9133da74 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -16,6 +16,7 @@ IMAGE_INSTALL += "sshd-regen-keys"

# qemuamd64-bookworm
WKS_FILE:qemuamd64:debian-bookworm ?= "multipart-efi.wks"
+IMAGE_INSTALL:append:qemuamd64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"

# qemuamd64-bullseye
IMAGE_FSTYPES:append:qemuamd64:debian-bullseye ?= " cpio.gz tar.gz"
@@ -51,3 +52,4 @@ IMAGER_INSTALL:append:qemuarm:debian-bookworm ?= " ${SYSTEMD_BOOTLOADER_INSTALL}
# qemuarm64-bookworm
IMAGE_FSTYPES:append:qemuarm64:debian-bookworm ?= " wic.xz"
IMAGER_INSTALL:append:qemuarm64:debian-bookworm ?= " ${GRUB_BOOTLOADER_INSTALL}"
+IMAGE_INSTALL:append:qemuarm64:debian-bookworm = " prebuilt-docker-img prebuilt-podman-img"
diff --git a/testsuite/citest.py b/testsuite/citest.py
index 8dd907d0..539c9440 100755
--- a/testsuite/citest.py
+++ b/testsuite/citest.py
@@ -522,3 +522,24 @@ class VmBootTestFull(CIBaseTest):
self.init()
self.vm_start('mipsel','bookworm', image='isar-image-ci',
script='test_kernel_module.sh example_module')
+
+
+ def test_amd64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('amd64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
+
+ def test_arm64_bookworm_prebuilt_containers(self):
+ self.init()
+ self.vm_start('arm64', 'bookworm', image='isar-image-ci',
+ cmd='echo root | su -c \'' \
+ 'PATH=\$PATH:/usr/sbin;' \
+ 'for n in \$(seq 30); do docker images | grep -q alpine && break; sleep 10; done;' \
+ 'docker run --rm quay.io/libpod/alpine:3.10.2 true && ' \
+ 'for n in \$(seq 30); do podman images | grep -q alpine && break; sleep 10; done;' \
+ 'podman run --rm quay.io/libpod/alpine:latest true\'')
--
2.43.0

Niedermayr, BENEDIKT

unread,
Jul 16, 2024, 8:56:06 AM (12 days ago) Jul 16
to isar-...@googlegroups.com, Kiszka, Jan, Cirujano Cuesta, Silvano, MOESSBAUER, Felix
On Mon, 2024-07-15 at 12:08 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
>  doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 60 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 776ae52c..733b2b30 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"
>  
>  Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
>  To install additional packages into the sbuild chroot, add them to
> `SBUILD_CHROOT_PREINSTALL_EXTRA`.
> +
> +## Pre-install container images
> +
> +If an isar-generated image shall provides a container runtime, it may also be

If an isar-generated image shall provide a container runtime...

> +desirable to pre-install container images to avoid having to download them on
> +first boot or because they may not be accessible outside of the build
> +environment. Isar supports this scenario via two services, a container fetcher
> +and a container loader.
> +
> +### Bitbake fetcher for containers
> +
> +The bitbake fetching protocol "docker://" allows to download pre-built images
> +from container registries. The URL consists of the image path, followed by
> +a recommened digests in the form `digest=sha256:<sha256sum>` and an optional
a recommended...

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not
... to indentify the ?an? when fetching...
Benedikt

Jan Kiszka

unread,
Jul 16, 2024, 8:58:12 AM (12 days ago) Jul 16
to Niedermayr, Benedikt (T CED OES-DE), isar-...@googlegroups.com, Cirujano Cuesta, Silvano (T CED OES-DE), Moessbauer, Felix (T CED OES-DE)
Thanks for actually reading it :). Update will follow.

Jan

--
Siemens AG, Technology
Linux Expert Center

Schmidt, Adriaan

unread,
Jul 16, 2024, 9:42:20 AM (12 days ago) Jul 16
to Kiszka, Jan, isar-users, Cirujano Cuesta, Silvano, Niedermayr, BENEDIKT, MOESSBAUER, Felix
'Jan Kiszka' via isar-users <isar-...@googlegroups.com>, Montag, 15. Juli 2024 12:09:
> Subject: [PATCH v2 5/5] doc: Describe how to use the container fetcher and loader
recommended *digest

> +tag in the form `tag=<tag>`. A digest is preferred over a tag to identify the
> +an when fetching as it also allows to validate its integrity. If tag is not
> +specified, `latest` is used as tag name.
> +
> +When specifying a multi-arch image, the fetcher will download the images for
> +all available architectures. If this is not desired, directly specify the
> +digest of the desired architecture manifest instead of that the manifest list.
> +
> +The fetched container image are stored in a directory in the `WORKDIR` of the

*images

> +requesting recipe. When a multi-arch image was specified, only the image
> +matching `PACKAGE_ARCH` will be stored. The name of the image directory is
> +derived from the container image name, replacing all `/` with `.`.
> +
> +### Container loader helpers
> +
> +To create a Debian package which can carry container images and load them into
> +local storage of docker or podman, there is a set of helpers available. To use
> +them into an own recipe, add

them *in an own

> +`require recipes-support/container-loader/docker-loader.inc` when using docker
> +and `require recipes-support/container-loader/podman-loader.inc` when using
> +podman. The loader will try to transfer the packaged image into the container
> +runtime storage on boot, but only if no container image of the same name and
> +tag is present already.
> +
> +Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container
> images
> +remain by default available and may be used again for loading the storage after
> +it may have been emptied later on (factory reset).
> +
> +Source container images may either be fetched as binaries from a registry, see
> +above, or built via isar as well.
> +
> +### Example
> +
> +This creates debian package with will download, package and then load the

*which will download

> +`debian:bookworm-20240701-slim` container image into the docker container
> +storage. The package will depend on `docker.io`, thus ensure that that basic

*ensuring?

Adriaan

> +runtime services are installed on the target as well. The packaged image will
> +be deleted from the target device's rootfs after successful import.
> +
> +```
> +require recipes-support/container-loader/docker-loader.inc
> +
> +CONTAINER_DELETE_AFTER_LOAD = "1"
> +
> +SRC_URI +=
> "docker://debian;digest=sha256:f528891ab1aa484bf7233dbcc84f3c806c3e427571d75
> 510a9d74bb5ec535b33;tag=bookworm-20240701-slim"
> +```
> --
> 2.43.0
>
> --
> You received this message because you are subscribed to the Google Groups "isar-
> users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-
> users/2e1537004e7ad48f33cd3d3daf2b8f7c3294d31a.1721038111.git.jan.kiszka%40s
> iemens.com.

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed, by default with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
.../container-loader/container-loader.inc | 101 ++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
5 files changed, 151 insertions(+)
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..e97e829b
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,101 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..b6abec92
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ ${CONTAINER_DECOMPRESSOR_CMD} /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v3:
- do not compress container deb packages [Benedikt]
- fix various spelling issues in the doc [Adriaan, Benedikt]

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers
ci: Add test cases for container fetching and loading
doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 60 +++++++++++
kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 12 +++
.../prebuilt-podman-img_0.1.bb | 10 ++
.../recipes-core/images/isar-image-ci.bb | 2 +
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 90 ++++++++++++++++
.../container-loader/container-loader.inc | 101 ++++++++++++++++++
.../container-loader/docker-loader.inc | 10 ++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 ++
testsuite/citest.py | 21 ++++
15 files changed, 389 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
create mode 100644 meta/lib/container_fetcher.py
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

The fetcher will try to pull all available variants of a multi-arch
image. If this is not needed, you can also directly specify the image
digest of a specific architecture.

Future versions may also introduce full unpacking of the fetched
container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 90 ++++++++++++++++++++++++++++++++++
2 files changed, 96 insertions(+)
create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..8513e246
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,90 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
doc/user_manual.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 60 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..e97a2cd5 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,63 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provide a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.
+
+When specifying a multi-arch image, the fetcher will download the images for
+all available architectures. If this is not desired, directly specify the
+digest of the desired architecture manifest instead of that the manifest list.
+
+The fetched container images are stored in a directory in the `WORKDIR` of the
+requesting recipe. When a multi-arch image was specified, only the image
+matching `PACKAGE_ARCH` will be stored. The name of the image directory is
+derived from the container image name, replacing all `/` with `.`.
+
+### Container loader helpers
+
+To create a Debian package which can carry container images and load them into
+local storage of docker or podman, there is a set of helpers available. To use
+them in an own recipe, add
+`require recipes-support/container-loader/docker-loader.inc` when using docker
+and `require recipes-support/container-loader/podman-loader.inc` when using
+podman. The loader will try to transfer the packaged image into the container
+runtime storage on boot, but only if no container image of the same name and
+tag is present already.
+
+Unless `CONTAINER_DELETE_AFTER_LOAD` is set to `1`, the source container images
+remain by default available and may be used again for loading the storage after
+it may have been emptied later on (factory reset).
+
+Source container images may either be fetched as binaries from a registry, see
+above, or built via isar as well.
+
+### Example
+
+This creates a debian package which will download, package and then load the
+`debian:bookworm-20240701-slim` container image into the docker container
+storage. The package will depend on `docker.io`, insuring that that basic
+runtime services are installed on the target as well. The packaged image will
+be deleted from the target device's rootfs after successful import.
+
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

Jan Kiszka

unread,
Jul 16, 2024, 10:18:14 AM (12 days ago) Jul 16
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

Niedermayr, BENEDIKT

unread,
Jul 16, 2024, 4:08:56 PM (11 days ago) Jul 16
to isar-...@googlegroups.com, Kiszka, Jan, Cirujano Cuesta, Silvano, MOESSBAUER, Felix
Add do_install_fetched_containers[vardeps] += "CONTAINER_COMPRESSOR_CMD" here.

By the way, for this to work the implementation in bitbake.conf needs to be fixed as well.
The vardepsexclude for ZSTD_DEFAULTS should not contain "ZSTD_LEVEL".

I could sent out a patch for this.

Benedikt

MOESSBAUER, Felix

unread,
Jul 17, 2024, 7:50:50 AM (11 days ago) Jul 17
to isar-...@googlegroups.com, Kiszka, Jan, Niedermayr, BENEDIKT, Cirujano Cuesta, Silvano
On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> This bitbake fetcher allows to pull container images from registries,
> store them in the download cache and transfer them into the workdir
> of
> recipes requesting the image. The format of the URL is
>
> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>
> Fetching without digest is supported but will cause a warning, just
> like
> downloading via wget without a checksum.

This is fine, as long as the upstream artifact is expected to be
stable.

>
> If tag is left out, "latest" is
> used.

The tag should be mandatory and it should be clear that - even without
digest - the artifact needs to be stable. Floating tags create all
kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
vote for just not allowing this.

>
> The fetcher will try to pull all available variants of a multi-arch
> image. If this is not needed, you can also directly specify the image
> digest of a specific architecture.

In most cases this does not make sense. I propose to always limit the
fetching to the current architecture. For that, we either need a
mapping between the debian architecture and the OCI architectures (what
is specified in the application/vnd.oci.image.index.v1+json manifest),
or we simply don't support index manifests at all and force people to
use image manifests (application/vnd.oci.image.manifest.v1+json).

Felix

Jan Kiszka

unread,
Jul 17, 2024, 12:02:49 PM (10 days ago) Jul 17
to Moessbauer, Felix (T CED OES-DE), isar-...@googlegroups.com, Niedermayr, Benedikt (T CED OES-DE), Cirujano Cuesta, Silvano (T CED OES-DE)
On 17.07.24 13:50, Moessbauer, Felix (T CED OES-DE) wrote:
> On Tue, 2024-07-16 at 16:18 +0200, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.k...@siemens.com>
>>
>> This bitbake fetcher allows to pull container images from registries,
>> store them in the download cache and transfer them into the workdir
>> of
>> recipes requesting the image. The format of the URL is
>>
>> docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]
>>
>> Fetching without digest is supported but will cause a warning, just
>> like
>> downloading via wget without a checksum.
>
> This is fine, as long as the upstream artifact is expected to be
> stable.
>
>>
>> If tag is left out, "latest" is
>> used.
>
> The tag should be mandatory and it should be clear that - even without
> digest - the artifact needs to be stable. Floating tags create all
> kinds of issues (w.r.t. the sstate cache and reproducible builds), so I
> vote for just not allowing this.

I cannot follow: Leaving out the tag has nothing to do with getting a
stable image. It may just lead to something being tagged "latest" on the
device that had a different tag (or none) in the registry.

>
>>
>> The fetcher will try to pull all available variants of a multi-arch
>> image. If this is not needed, you can also directly specify the image
>> digest of a specific architecture.
>
> In most cases this does not make sense. I propose to always limit the
> fetching to the current architecture. For that, we either need a
> mapping between the debian architecture and the OCI architectures (what
> is specified in the application/vnd.oci.image.index.v1+json manifest),
> or we simply don't support index manifests at all and force people to
> use image manifests (application/vnd.oci.image.manifest.v1+json).

Don't worry, I already have ideas of redesigning this into per-arch
fetches. Background is using less different formats on the build system
and, where possible, even hard-link between them to save space and time
with larger images.

Jan

MOESSBAUER, Felix

unread,
Jul 18, 2024, 3:18:36 AM (10 days ago) Jul 18
to isar-...@googlegroups.com, Kiszka, Jan, Niedermayr, BENEDIKT, Cirujano Cuesta, Silvano
Ok, that means the image is just tagged on the device. Then it should
be fine.

Felix

Jan Kiszka

unread,
Jul 19, 2024, 12:38:46 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
We noticed internally that there are at least 3 implementations for
solving the task of pre-loading container images into isar-built base
systems, may it be that the images are not publicly available or the
device is only poorly connected, if at all. This series tries to solve
the task generically - hopefully.

Changes in v4
- fetch per architecture, no longer full multi-arch
- directly archive and compress in download task
- use hard links where possible to reduce disk usage
- always use zstd, remove configurability

Changes in v3:
- do not compress container deb packages [Benedikt]
- fix various spelling issues in the doc [Adriaan, Benedikt]

Changes in v2:
- switch to open-coded unpack before loading
- switch to zstd as default compression
- use default compression settings from bitbake.conf
- add support for delete-after-load (opt-in)
- factor out fetched container installation into separate task
(should make self-built container packaging easier)
- use BPN instead of PN
- add "Requires" to systemd service
- use PACKAGE_ARCH to select container arch (rather than DISTRO_ARCH)
- add CI test
- add documentation

Jan

Jan Kiszka (5):
Introduce fetcher from container registries
container-loader: Introduce helper to load container images into local
registry
meta-isar: Add demo packages for installing prebuilt containers
ci: Add test cases for container fetching and loading
doc: Describe how to use the container fetcher and loader

doc/user_manual.md | 58 +++++++++++++
kas/package/Kconfig | 19 ++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 ++
kas/package/pkg_prebuilt-podman-img.yaml | 9 ++
.../prebuilt-docker-img_0.1.bb | 12 +++
.../prebuilt-podman-img_0.1.bb | 10 +++
.../recipes-core/images/isar-image-ci.bb | 2 +
meta/classes/dpkg-base.bbclass | 6 ++
meta/lib/container_fetcher.py | 86 +++++++++++++++++++
.../container-loader/container-loader.inc | 73 ++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 ++++
.../container-loader/podman-loader.inc | 10 +++
testsuite/citest.py | 21 +++++
15 files changed, 355 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
create mode 100644 meta/lib/container_fetcher.py
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

--
2.43.0

Jan Kiszka

unread,
Jul 19, 2024, 12:38:47 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This allows to write dpkg-raw recipes which packages archived container
images and load them into a local docker or podman registry on boot. The
scenario behind this is to pre-fill local registries in a way that still
permits live updates during runtime.

The loader script only process images which are not yet available under
the same name and tag in the local registry. Also after loading, the
archived images stay on the local file system. This allows to perform
reloading in case the local registry should be emptied (e.g. reset to
factory state). To reduce the space those original images need, they are
compressed with zstd.

Separate include files are available to cater the main container
engines, one for docker and one for podman.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
.../container-loader/container-loader.inc | 73 +++++++++++++++++++
.../container-loader/docker-loader.inc | 10 +++
.../files/container-loader.service.tmpl | 12 +++
.../files/container-loader.sh.tmpl | 18 +++++
.../container-loader/podman-loader.inc | 10 +++
5 files changed, 123 insertions(+)
create mode 100644 meta/recipes-support/container-loader/container-loader.inc
create mode 100644 meta/recipes-support/container-loader/docker-loader.inc
create mode 100644 meta/recipes-support/container-loader/files/container-loader.service.tmpl
create mode 100755 meta/recipes-support/container-loader/files/container-loader.sh.tmpl
create mode 100644 meta/recipes-support/container-loader/podman-loader.inc

diff --git a/meta/recipes-support/container-loader/container-loader.inc b/meta/recipes-support/container-loader/container-loader.inc
new file mode 100644
index 00000000..5fd8d23c
--- /dev/null
+++ b/meta/recipes-support/container-loader/container-loader.inc
@@ -0,0 +1,73 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+FILESPATH:append := ":${FILE_DIRNAME}/files"
+
+inherit dpkg-raw
+
+SRC_URI += " \
+ file://container-loader.service.tmpl \
+ file://container-loader.sh.tmpl"
+
+CONTAINER_DELETE_AFTER_LOAD ?= "0"
+
+DEBIAN_DEPENDS += "${CONTAINER_ENGINE_PACKAGES}, zstd"
+
+TEMPLATE_FILES += " \
+ container-loader.service.tmpl \
+ container-loader.sh.tmpl"
+TEMPLATE_VARS += " \
+ CONTAINER_ENGINE \
+ CONTAINER_DELETE_AFTER_LOAD"
+
+do_install() {
+ install -m 755 ${WORKDIR}/container-loader.sh ${D}/usr/share/${BPN}
+}
+do_install[cleandirs] += " \
+ ${D}/usr/share/${BPN} \
+ ${D}/usr/share/${BPN}/images"
+
+python do_install_fetched_containers() {
+ from oe.path import copyhardlink
+
+ workdir = d.getVar('WORKDIR')
+ D = d.getVar('D')
+ BPN = d.getVar('BPN')
+
+ image_list = open(D + "/usr/share/" + BPN + "/image.list", "w")
+
+ src_uri = d.getVar('SRC_URI').split()
+ for uri in src_uri:
+ scheme, host, path, _, _, parm = bb.fetch.decodeurl(uri)
+ if scheme != "docker":
+ continue
+
+ tag = parm["tag"] if "tag" in parm else "latest"
+ image_name = host + (path if path != "/" else "")
+ image_file = image_name.replace('/', '.') + \
+ ":" + tag + ".zst"
+ dest_dir = D + "/usr/share/" + BPN + "/images"
+
+ copyhardlink(workdir + "/" + image_file, dest_dir + "/" + image_file)
+
+ line = f"{image_file} {image_name}:{tag}"
+ bb.note(f"adding '{line}' to image.list")
+ image_list.write(line + "\n")
+
+ image_list.close()
+}
+
+addtask install_fetched_containers after do_install before do_prepare_build
+
+do_prepare_build:append() {
+ install -v -m 644 ${WORKDIR}/container-loader.service ${S}/debian/${BPN}.service
+
+ # Do not compress the package, most of its payload is already, and trying
+ # nevertheless will only cost time without any gain.
+ cat <<EOF >> ${S}/debian/rules
+override_dh_builddeb:
+ dh_builddeb -- -Znone
+EOF
+}
diff --git a/meta/recipes-support/container-loader/docker-loader.inc b/meta/recipes-support/container-loader/docker-loader.inc
new file mode 100644
index 00000000..b864c854
--- /dev/null
+++ b/meta/recipes-support/container-loader/docker-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "docker"
+
+CONTAINER_ENGINE_PACKAGES ?= "docker.io, apparmor"
diff --git a/meta/recipes-support/container-loader/files/container-loader.service.tmpl b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
new file mode 100644
index 00000000..1638eaf2
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.service.tmpl
@@ -0,0 +1,12 @@
+[Unit]
+Description=Load archived container images on boot
+After=${CONTAINER_ENGINE}.service
+Requires=${CONTAINER_ENGINE}.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/share/${BPN}/container-loader.sh
+RemainAfterExit=true
+
+[Install]
+WantedBy=multi-user.target
diff --git a/meta/recipes-support/container-loader/files/container-loader.sh.tmpl b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
new file mode 100755
index 00000000..2356e31c
--- /dev/null
+++ b/meta/recipes-support/container-loader/files/container-loader.sh.tmpl
@@ -0,0 +1,18 @@
+#!/bin/sh
+#
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+set -eu
+
+while read -r image ref; do
+ if [ -e /usr/share/${BPN}/images/"$image" ] && \
+ [ -z "$(${CONTAINER_ENGINE} images -q "$ref")" ]; then
+ pzstd -c -d /usr/share/${BPN}/images/"$image" | \
+ ${CONTAINER_ENGINE} load
+ if [ "${CONTAINER_DELETE_AFTER_LOAD}" = "1" ]; then
+ rm -f /usr/share/${BPN}/images/"$image"
+ fi
+ fi
+done < /usr/share/${BPN}/image.list
diff --git a/meta/recipes-support/container-loader/podman-loader.inc b/meta/recipes-support/container-loader/podman-loader.inc
new file mode 100644
index 00000000..d2c9a12d
--- /dev/null
+++ b/meta/recipes-support/container-loader/podman-loader.inc
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require container-loader.inc
+
+CONTAINER_ENGINE = "podman"
+
+CONTAINER_ENGINE_PACKAGES ?= "podman"
--
2.43.0

Jan Kiszka

unread,
Jul 19, 2024, 12:38:47 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

One recipe for docker, one for podman. Both pull from a registry that,
in contrast to infamous dockerhub, should not throttle CI jobs running
these frequently for testing purposes.

The podman variant of the recipe is intentionally leaving out the digest
to trigger the related warning of the container fetcher.

These demos also come with kas integration.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
kas/package/Kconfig | 19 +++++++++++++++++++
kas/package/pkg_prebuilt-docker-img.yaml | 9 +++++++++
kas/package/pkg_prebuilt-podman-img.yaml | 9 +++++++++
.../prebuilt-docker-img_0.1.bb | 12 ++++++++++++
.../prebuilt-podman-img_0.1.bb | 10 ++++++++++
5 files changed, 59 insertions(+)
create mode 100644 kas/package/pkg_prebuilt-docker-img.yaml
create mode 100644 kas/package/pkg_prebuilt-podman-img.yaml
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
create mode 100644 meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb

new file mode 100644
index 00000000..df96a484
--- /dev/null
+++ b/kas/package/pkg_prebuilt-docker-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-docker-img: |
+ IMAGE_INSTALL:append = " prebuilt-docker-img"
diff --git a/kas/package/pkg_prebuilt-podman-img.yaml b/kas/package/pkg_prebuilt-podman-img.yaml
new file mode 100644
index 00000000..d0b8da1c
--- /dev/null
+++ b/kas/package/pkg_prebuilt-podman-img.yaml
@@ -0,0 +1,9 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+
+header:
+ version: 14
+
+local_conf_header:
+ package-prebuilt-podman-img: |
+ IMAGE_INSTALL:append = " prebuilt-podman-img"
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
new file mode 100644
index 00000000..0dfc9b8f
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-docker-img_0.1.bb
@@ -0,0 +1,12 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+
+SRC_URI += "\
+ docker://quay.io/libpod/alpine;digest=sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f;tag=3.10.2 \
+ "
diff --git a/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
new file mode 100644
index 00000000..e671a494
--- /dev/null
+++ b/meta-isar/recipes-app/prebuilt-container/prebuilt-podman-img_0.1.bb
@@ -0,0 +1,10 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+

Jan Kiszka

unread,
Jul 19, 2024, 12:38:47 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
doc/user_manual.md | 58 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 58 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 776ae52c..2bdacbec 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -1519,3 +1519,61 @@ SBUILD_CHROOT_PREINSTALL_EXTRA += "<base packages>"

Then, in the dpkg recipe of your package, simply set `SBUILD_FLAVOR = "<your flavor>"`.
To install additional packages into the sbuild chroot, add them to `SBUILD_CHROOT_PREINSTALL_EXTRA`.
+
+## Pre-install container images
+
+If an isar-generated image shall provide a container runtime, it may also be
+desirable to pre-install container images to avoid having to download them on
+first boot or because they may not be accessible outside of the build
+environment. Isar supports this scenario via two services, a container fetcher
+and a container loader.
+
+### Bitbake fetcher for containers
+
+The bitbake fetching protocol "docker://" allows to download pre-built images
+from container registries. The URL consists of the image path, followed by
+a recommended digest in the form `digest=sha256:<sha256sum>` and an optional
+tag in the form `tag=<tag>`. A digest is preferred over a tag to identify an
+image when fetching because it also allows to validate its integrity. If a tag
+is not specified, `latest` is used as tag name.
+
+In case a multi-arch image is specified, the fetcher will only pull for the
+package architecture of the requesting recipe (`PACKAGE_ARCH`). The fetched
+images are stored as zstd-compressed in docker-archive format in the
+`WORKDIR` of the recipe. The name of the image is derived from the container
+image name, replacing all `/` with `.` and appending `:<tag>.zst`. Example:
+`docker://debian;tag=bookworm` will be saved as `debian:bookworm.zst`.
+```
+require recipes-support/container-loader/docker-loader.inc
+
+CONTAINER_DELETE_AFTER_LOAD = "1"
+

Jan Kiszka

unread,
Jul 19, 2024, 12:38:47 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This bitbake fetcher allows to pull container images from registries,
store them in the download cache and transfer them into the workdir of
recipes requesting the image. The format of the URL is

docker://[<host>/]<image>;digest=sha256:...[;tag=<tag>]

Fetching without digest is supported but will cause a warning, just like
downloading via wget without a checksum. If tag is left out, "latest" is
used.

In case a multi-arch image is specified, the fetcher will only pull for
the package architecture of the requesting recipe. The image is stored
compressed in docker-archive format and, wherever possible, hard-linked
from DL_DIR to WORKDIR. Future versions may also introduce full
unpacking of the fetched container layers in workdir if use cases come up.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 6 +++
meta/lib/container_fetcher.py | 86 ++++++++++++++++++++++++++++++++++
2 files changed, 92 insertions(+)
create mode 100644 meta/lib/container_fetcher.py

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..d90b32a9 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -98,6 +98,12 @@ python() {
if len(d.getVar('SRC_APT').strip()) > 0:
bb.build.addtask('apt_unpack', 'do_patch', '', d)
bb.build.addtask('cleanall_apt', 'do_cleanall', '', d)
+
+ # container docker fetcher
+ import container_fetcher
+ from bb.fetch2 import methods
+
+ methods.append(container_fetcher.Container())
}

do_apt_fetch() {
diff --git a/meta/lib/container_fetcher.py b/meta/lib/container_fetcher.py
new file mode 100644
index 00000000..0d659154
--- /dev/null
+++ b/meta/lib/container_fetcher.py
@@ -0,0 +1,86 @@
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2024
+#
+# SPDX-License-Identifier: MIT
+
+import oe.path
+import os
+import tempfile
+from bb.fetch2 import FetchMethod
+from bb.fetch2 import logger
+from bb.fetch2 import MissingChecksumEvent
+from bb.fetch2 import NoChecksumError
+from bb.fetch2 import runfetchcmd
+
+class Container(FetchMethod):
+ def supports(self, ud, d):
+ return ud.type in ['docker']
+
+ def urldata_init(self, ud, d):
+ ud.tag = "latest"
+ if "tag" in ud.parm:
+ ud.tag = ud.parm["tag"]
+
+ ud.digest = None
+ if "digest" in ud.parm:
+ ud.digest = ud.parm["digest"]
+
+ ud.arch = d.getVar('PACKAGE_ARCH')
+ ud.variant = None
+ if ud.arch == "armhf":
+ ud.arch = "arm"
+ ud.variant = "v7"
+ elif ud.arch == "armel":
+ ud.arch = "arm"
+ ud.variant = "v6"
+
+ ud.container_name = ud.host + (ud.path if ud.path != "/" else "")
+ ud.container_src = ud.container_name + \
+ ("@" + ud.digest if ud.digest else ":" + ud.tag)
+ ud.localname = ud.container_name.replace('/', '.')
+ ud.localfile = "container-images/" + ud.arch + "/" + \
+ (ud.variant + "/" if ud.variant else "") + ud.localname + \
+ "_" + (ud.digest.replace(":", "-") if ud.digest else ud.tag) + \
+ ".zst"
+
+ def download(self, ud, d):
+ tarball = ud.localfile[:-len('.zst')]
+ with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
+ # Take a two steps for downloading into a docker archive because
+ # not all source may have the required Docker schema 2 manifest.
+ runfetchcmd("skopeo copy --preserve-digests " + \
+ f"--override-arch {ud.arch} " + \
+ (f"--override-variant {ud.variant} " if ud.variant else "") + \
+ f"docker://{ud.container_src} dir:{tmpdir}", d)
+ runfetchcmd(f"skopeo copy dir:{tmpdir} " + \
+ f"docker-archive:{tarball}:{ud.container_name}:{ud.tag}", d)
+ zstd_defaults = d.getVar('ZSTD_DEFAULTS')
+ runfetchcmd(f"zstd -f --rm {zstd_defaults} {tarball}", d)
+ image_file = ud.localname + ":" + ud.tag + ".zst"
+ oe.path.remove(rootdir + "/" + image_file)
+ oe.path.copyhardlink(ud.localpath, rootdir + "/" + image_file)
--
2.43.0

Jan Kiszka

unread,
Jul 19, 2024, 12:38:48 PM (8 days ago) Jul 19
to isar-users, Silvano Cirujano-Cuesta, Benedikt Niedermayr, Felix Moessbauer
From: Jan Kiszka <jan.k...@siemens.com>

This plugs the two example recipes for loading container images into
VM-based testing. The test consists of running 'true' in the installed
alpine images.

Rather than enabling the ci user to do password-less sudo, this uses su
with the piped-in password. Another trick needed is to poll for the
images because loading is performed asynchronously.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Niedermayr, BENEDIKT

unread,
Jul 25, 2024, 6:48:42 AM (3 days ago) Jul 25
to isar-...@googlegroups.com, Kiszka, Jan, Cirujano Cuesta, Silvano, MOESSBAUER, Felix
do_fetch doesn't get triggered if "ZSTD_LEVEL" changes, but it should, since the
output zst file changes with a different ZSTD_LEVEL.

Even adding a do_fetch[vardeps] += "ZSTD_DEFAULTS" doesn't lead to the desired behavior:

When ZSTD_LEVEL changes, the fetch task is run but does not actually fetch anything and
immediately proceeds with do_unpack. More precisely, the download() Method of the Container()
class is not getting executed, maybe due to some logic in the upper/surrounding Fetch() class.
That would be fine if the compression task didn't implement the compression...

Another thing to mention is that he whole fetch task is run again only if the compression changes.
Skopeo seems to override already downloaded layers rather than skipping them, which means that
changing the compression also means a complete new download process.

Maybe the unpack() task for packing and hardlinking the images would be better, even though it
sounds strange to put an compression task into the unpack task.

Benedikt

Niedermayr, BENEDIKT

unread,
Jul 25, 2024, 7:10:28 AM (3 days ago) Jul 25
to isar-...@googlegroups.com, Kiszka, Jan, Cirujano Cuesta, Silvano, MOESSBAUER, Felix
Sorry wrong wording:

That would be fine if the FETCH task didn't implement the compression...

Benedikt
Reply all
Reply to author
Forward
0 new messages