[PATCH 1/6] schema: add support for isar in rootless mode

[Felix Moessbauer]

To prepare a migration path from isar to isar-rootless, we also
introduce the isar-privileged entry.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
docs/format-changelog.rst | 2 ++
kas/schema-kas.json | 4 +++-
2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/docs/format-changelog.rst b/docs/format-changelog.rst
index b43b23066..4c7a33fcc 100644
--- a/docs/format-changelog.rst
+++ b/docs/format-changelog.rst
@@ -219,3 +219,5 @@ Added

- Switch to nodistro which is the default distro setting in
openembedded-core.
+- Extend the allowed values of ``build_system`` by adding ``isar-privileged``
+ and ``isar-rootless``.
diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index f3594a247..7e3be5d34 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -83,7 +83,9 @@
"enum": [
"openembedded",
"oe",
- "isar"
+ "isar",
+ "isar-privileged",
+ "isar-rootless"
]
},
"defaults": {
--
2.53.0

[Felix Moessbauer]

Dear kas isar users,

as a preparation for our activities to implement / try out rootless isar
builds we need a test environment to execute the build (both locally and
in CI). This test environment is provided in this patch series.

Changes since RFC v3:

- rebased onto next
- improved commit message of "clean: add support for clean..."

Changes since v2:

- rebased onto next
- add support for docker and docker-rootless
- complete overhaul of the user ns subid mapping:
now supports both direct (docker) and indirect (podman)
mappings
- prohibit usage of sudo in isar-rootless mode. On rootless executors
like podman or docker-rootless, this is not strictly needed. But
on docker system this avoids accidental container breakout.
- introduce isar-privileged and map isar to isar-privileged

Note, that the various execution modes still lack CI test coverage.

Changes since v1:

- rebased onto next
- add acl tool (isar rootless host dependency)

Note, that the interfaces still have to be discussed with isar upstream.
I'm planning to send the corresponding isar series (v3) by today. With
this kas series people already have an environment for testing.

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (6):
schema: add support for isar in rootless mode
kas: add support for isar-rootless build system
extend buildsystem test to check isar-rootless configuration
kas-container: configure container for nested namespaces
kas-container: block usage of sudo in isar-rootless mode
clean: add support for cleaning isar-rootless generated data

Dockerfile | 3 +-
container-entrypoint | 55 +++++++++++++++++++
docs/format-changelog.rst | 2 +
docs/userguide/kas-container-description.inc | 5 +-
examples/isar.yml | 4 +-
kas-container | 44 ++++++++++++++-
kas/config.py | 9 ++-
kas/libcmds.py | 2 +
kas/libkas.py | 2 +-
kas/plugins/clean.py | 31 ++++++++++-
kas/plugins/menu.py | 2 +-
kas/schema-kas.json | 4 +-
tests/test_build_system.py | 12 ++++
.../test-isar-privileged.yml | 7 +++
.../test_build_system/test-isar-rootless.yml | 7 +++
15 files changed, 176 insertions(+), 13 deletions(-)
create mode 100644 tests/test_build_system/test-isar-privileged.yml
create mode 100644 tests/test_build_system/test-isar-rootless.yml

--
2.53.0

[Felix Moessbauer]

For rootless nesting of the uid_ns, we need to provide a list of sub
uids/gids which can be used in the sub namespace. As each id in the
sub namespace must be mappable into the parent namespace, we need to
compute the mapping based on the layout of the parent. The following
mapping works for rootless podman with --userns=keep-id.

Note, that the default range (65k) does NOT allow to map nobody/nogroup,
in the sub-namespace, as this usually is id 65k-1 and we loose at least
one id per nesting. If we get a larger range, we just map as much as
we can and by that make nobody/nogroup usable.

With these changes we also add the uidmap and acl binaries to the
container which are needed by isar to setup the namespaces and
permissions.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

Dockerfile | 3 +-
container-entrypoint | 44 ++++++++++++++++++++
docs/userguide/kas-container-description.inc | 5 ++-
kas-container | 43 ++++++++++++++++++-
4 files changed, 90 insertions(+), 5 deletions(-)

diff --git a/Dockerfile b/Dockerfile
index 2cc62c04a..91a520bf1 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -113,7 +113,8 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=${CACHE_SHARING} \
umoci skopeo \
python3-botocore \
bubblewrap \
- debootstrap && \
+ debootstrap \
+ uidmap acl && \
rm -f /etc/apt/apt.conf.d/use-snapshot.conf /etc/apt/apt.conf.d/keep-packages.conf && \
if [ -f "/etc/apt/sources.list.d/debian.sources~" ]; then \
mv -f /etc/apt/sources.list.d/debian.sources~ /etc/apt/sources.list.d/debian.sources; \
diff --git a/container-entrypoint b/container-entrypoint
index dee7275e8..da0c36d3a 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -34,6 +34,47 @@ enable_qemu_binfmts()
done
}

+# For rootless nesting of the uid_ns, we need to provide a list of sub
+# uids/gids which can be used in the sub namespace. As each id in the
+# sub namespace must be mappable into the parent namespace, we need to
+# compute the mapping based on the layout of the parent: Podman maps
+# to an intermediate namespace, hence giving us a user-id map we
+# actually can use for further splitting. Docker maps directly, hence
+# we need to compute the number of sub ids we can use (which have a
+# mapping in the parent ns)
+#
+# Note, that the default range (65k) does NOT allow to map nobody/nogroup,
+# in the sub-namespace, as this usually is id 65k-1 and we loose at least
+# one id per nesting. If we get a larger range, we just map as much as
+# we can and by that make nobody/nogroup usable.
+# See man user_namespaces for details.
+setup_userns_mappings()
+{
+ UID_ROW=$(sort -r /proc/self/uid_map | head -1)
+ UID_OUTER=$(printf '%s' "$UID_ROW" | awk '{print $1}')
+ UID_INNER=$(printf '%s' "$UID_ROW" | awk '{print $2}')
+ UID_COUNT=$(printf '%s' "$UID_ROW" | awk '{print $3}')
+ GID_ROW=$(sort -r /proc/self/gid_map | head -1)
+ GID_OUTER=$(printf '%s' "$GID_ROW" | awk '{print $1}')
+ GID_INNER=$(printf '%s' "$GID_ROW" | awk '{print $2}')
+ GID_COUNT=$(printf '%s' "$GID_ROW" | awk '{print $3}')
+
+ # docker (direct mapping)
+ if [ "$UID_OUTER" = "1" ]; then
+ UID_INNER="$(($(id -u builder) + 1))"
+ [ "$UID_INNER" -lt "$UID_COUNT" ] && \
+ UID_COUNT="$((UID_COUNT - UID_INNER))"
+ fi
+ if [ "$GID_OUTER" = "1" ]; then
+ GID_INNER="$(($(id -g builder) + 1))"
+ [ "$GID_INNER" -lt "$GID_COUNT" ] && \
+ GID_COUNT="$((GID_COUNT - GID_INNER))"
+ fi
+
+ echo "builder:${UID_INNER}:${UID_COUNT}" | sudo tee /etc/subuid > /dev/null
+ echo "builder:${GID_INNER}:${GID_COUNT}" | sudo tee /etc/subgid > /dev/null
+}
+
# kas-isar: enable_qemu_binfmts

chown_managed_dirs()
@@ -103,6 +144,9 @@ else

GOSU="gosu builder"
fi
+# after all uid / gid changes are done, setup the namespace mappings
+# kas-isar: setup_userns_mappings
+
# kas-container on rootless docker workaround
if [ -n "$USER_ID" ] && [ "$USER_ID" -ne 0 ] && \
[ "$KAS_DOCKER_ROOTLESS" = "1" ] && [ "$(stat -c %u /repo)" -eq 0 ]; then
diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
index 7f350138a..776251124 100644
--- a/docs/userguide/kas-container-description.inc
+++ b/docs/userguide/kas-container-description.inc
@@ -39,5 +39,6 @@ written to from the host. To completely remove all data managed by kas, use
so they can be removed from the host.

.. note::
- The ISAR build system is not compatible with rootless execution. By that,
- we fall back to the system docker or podman instance.
+ The ISAR build system is compatible with rootless execution in ``isar-rootless``
+ mode only. The ``isar`` and ``isar-privileged`` modes fall back to the system docker
+ or podman instance.
diff --git a/kas-container b/kas-container
index 8e4937d9c..33f2fcd84 100755
--- a/kas-container
+++ b/kas-container
@@ -68,6 +68,8 @@ usage()
printf "%b" "\nOptional arguments:\n"
printf "%b" "--isar\t\t\tUse kas-isar container to build Isar image. To force\n"
printf "%b" " \t\t\tthe use of run0 over sudo, set KAS_SUDO_CMD=run0.\n"
+ printf "%b" "--isar-privileged\tRun the isar build in privileged mode\n"
+ printf "%b" "--isar-rootless\t\tRun the isar build in rootless mode\n"
printf "%b" "--with-loop-dev Pass a loop device to the " \
"container. Only required if\n"
printf "%b" "\t\t\tloop-mounting is used by recipes.\n"
@@ -171,6 +173,33 @@ enable_isar_mode()
fi
}

+enable_isar_rootless_mode()
+{
+ if [ -n "${ISAR_ROOTLESS_MODE}" ]; then
+ return
+ fi
+ ISAR_ROOTLESS_MODE=1
+ KAS_CONTAINER_IMAGE_NAME_DEFAULT="kas-isar"
+
+ # Use --privileged to pass the ambient capabilities into the container.
+ # When calling from the user session (podman or docker-rootless), this
+ # is fundamentally different from the system docker run --privileged
+ if [ "${KAS_CONTAINER_ENGINE}" = "podman" ]; then
+ KAS_RUNTIME_ARGS="${KAS_RUNTIME_ARGS} --userns=keep-id --privileged"
+ elif [ "${KAS_DOCKER_ROOTLESS}" = "1" ]; then
+ KAS_ISAR_ARGS="--privileged"
+ else
+ # we don't need --privileged, but we need to run with SYS_ADMIN
+ # to be able to unshare.
+ KAS_ISAR_ARGS=" \
+ --security-opt seccomp=unconfined \
+ --security-opt apparmor=unconfined \
+ --security-opt systempaths=unconfined \
+ --cap-add=SYS_ADMIN \
+ "
+ fi
+}
+
enable_oe_mode()
{
if [ "${KAS_CONTAINER_ENGINE}" = "podman" ]; then
@@ -356,10 +385,14 @@ esac
# parse kas-container options
while [ $# -gt 0 ]; do
case "$1" in
- --isar)
+ --isar|--isar-privileged)
enable_isar_mode
shift 1
;;
+ --isar-rootless)
+ enable_isar_rootless_mode
+ shift 1
+ ;;
--with-loop-dev)
if ! KAS_LOOP_DEV=$(/sbin/losetup -f 2>/dev/null); then
if [ "$(id -u)" -eq 0 ]; then
@@ -580,8 +613,10 @@ else
sed 's/build_system:[ ]\+//')
fi

-if [ "${BUILD_SYSTEM}" = "isar" ]; then
+if [ "${BUILD_SYSTEM}" = "isar" ] || [ "${BUILD_SYSTEM}" = "isar-privileged" ]; then
enable_isar_mode
+elif [ "${BUILD_SYSTEM}" = "isar-rootless" ]; then
+ enable_isar_rootless_mode
elif [ -z "${ISAR_MODE}" ]; then
enable_oe_mode
fi
@@ -787,5 +822,9 @@ while [ $KAS_EXTRA_BITBAKE_ARGS -gt 0 ]; do
KAS_EXTRA_BITBAKE_ARGS=$((KAS_EXTRA_BITBAKE_ARGS - 1))
done

+if [ "${ISAR_MODE}" = "1" ] && [ "${ISAR_ROOTLESS_MODE}" = "1" ]; then
+ fatal_error "only one of --isar and --isar-rootless can be selected."
+fi
+
# shellcheck disable=SC2086
trace ${KAS_CONTAINER_COMMAND} run "$@"
--
2.53.0

[Felix Moessbauer]

After switching to the builder user, we prohibit using sudo. This helps
downstream layers to find locations where sudo is incorrectly used, as
well as it prevents accidential breakout on system docker.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

container-entrypoint | 11 +++++++++++
kas-container | 1 +
2 files changed, 12 insertions(+)

diff --git a/container-entrypoint b/container-entrypoint
index da0c36d3a..9d23b6248 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -168,6 +168,17 @@ if [ "$PWD" = / ]; then
cd /builder || exit 1
fi

+if [ "$KAS_BLOCK_SUDO" = "1" ]; then
+ mkdir -p /usr/local/libexec
+ cat <<'EOF' > /usr/local/libexec/kas-no-sudo
+#!/bin/sh
+printf "KAS_BLOCK_SUDO=1: sudo is prohibited\n" >&2
+exit 1
+EOF
+ chmod +x /usr/local/libexec/kas-no-sudo
+ ln -sf /usr/local/libexec/kas-no-sudo /usr/bin/sudo
+fi
+
if [ -n "$1" ]; then
case "$1" in
build|checkout|clean*|diff|dump|for-all-repos|lock|menu|purge|shell|-*)
diff --git a/kas-container b/kas-container
index 33f2fcd84..641d17714 100755
--- a/kas-container
+++ b/kas-container
@@ -180,6 +180,7 @@ enable_isar_rootless_mode()
fi
ISAR_ROOTLESS_MODE=1
KAS_CONTAINER_IMAGE_NAME_DEFAULT="kas-isar"
+ KAS_RUNTIME_ARGS="${KAS_RUNTIME_ARGS} -e KAS_BLOCK_SUDO=1"

# Use --privileged to pass the ambient capabilities into the container.

# When calling from the user session (podman or docker-rootless), this

--
2.53.0

[Felix Moessbauer]

During the rootless build, data is generated inside another user id
namespace. By that, the owner of the data from outside of the namespace
is within the the subuid / subgid range, hence does not technically
belong to the calling (kas) user.

While isar takes care to not leave any temporary artifacts with
non-caller ownership behind on successfull builds, this does not apply
to partial or interrupted runs.

To be able to delete this data as well, we enter the namespace similar
to how isar is doing it, clean from the inside (with correct uid/gid
mappings) and finally remove the remainder from the outside.

On rootfull builds, we can simply use sudo to remove everything from the
outside, however sudo is not available in rootless mode.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

kas/plugins/clean.py | 31 +++++++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index 2bffea2d4..037284020 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -95,7 +95,7 @@ class Clean():
dirs_to_remove = []
for tmpdir in tmpdirs:
logging.info(f'Removing {tmpdir}')
- if build_system == 'isar':
+ if (build_system or '').startswith('isar'):
dirs_to_remove.append(tmpdir)
else:
if not args.dry_run:
@@ -104,12 +104,39 @@ class Clean():
if len(dirs_to_remove) == 0:
return

+ # isar only
+ if build_system == 'isar-rootless':
+ self._rmtree_unshare(dirs_to_remove, args.dry_run)
+ else:
+ self._rmtree_sudo(dirs_to_remove, args.dry_run)
+
+ @staticmethod
+ def _rmtree_unshare(dirs_to_remove, dry_run):
+ uid = os.getuid()
+ for d in dirs_to_remove:
+ # find all dir entries that are not owned by the calling user
+ # and remove them by entering the user namespace first
+ clean_args = ['find', str(d), '(', '!', '-user', str(uid), '-type',
+ 'd', '-prune', ')', '-exec']
+ clean_args += ['unshare', '--map-auto', '--map-root-user',
+ '--keep-caps', 'rm', '-rf', '{}', ';']
+ logging.debug(' '.join(clean_args))
+ if not dry_run:
+ subprocess.check_call(clean_args)
+ # clean remaining files (owned by caller)
+ clean_args = ['rm', '-rf', str(d)]
+ logging.debug(' '.join(clean_args))
+ if not dry_run:
+ subprocess.check_call(clean_args)
+
+ @staticmethod
+ def _rmtree_sudo(dirs_to_remove, dry_run):
clean_args = ['sudo', '--prompt', '[sudo] enter password for %U '
'to clean ISAR artifacts']
clean_args.extend(['rm', '-rf'])
clean_args.extend([p.as_posix() for p in dirs_to_remove])
logging.debug(' '.join(clean_args))
- if not args.dry_run:
+ if not dry_run:
subprocess.check_call(clean_args)

@staticmethod
--
2.53.0

[Felix Moessbauer]

For a smooth transition from isar to isar-rootless back to isar, we
further add a warning if the build system is "isar" regarding the
semantics. For now, isar maps to isar-privileged to not break existing
layers. After some grace period, we will be able to change the default
of isar to isar-rootless.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

examples/isar.yml | 4 ++--
kas/config.py | 9 ++++++++-
kas/libcmds.py | 2 ++
kas/libkas.py | 2 +-
kas/plugins/menu.py | 2 +-
5 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/examples/isar.yml b/examples/isar.yml
index 3b9d5c7f2..4fc5750d3 100644
--- a/examples/isar.yml
+++ b/examples/isar.yml
@@ -23,9 +23,9 @@
#

header:
- version: 14
+ version: 22

-build_system: isar
+build_system: isar-privileged

machine: qemuamd64
distro: debian-trixie
diff --git a/kas/config.py b/kas/config.py
index cebc7494a..e3792efa0 100644
--- a/kas/config.py
+++ b/kas/config.py
@@ -26,6 +26,7 @@
import os
import json
import copy
+import logging
from pathlib import Path
from .repos import Repo
from .includehandler import IncludeHandler
@@ -69,7 +70,13 @@ class Config:
"""
Returns the pre-selected build system
"""
- return self._config.get('build_system', '')
+ build_system = self._config.get('build_system', '')
+ if build_system == 'isar':
+ logging.warning(
+ "The semantics of build_system: isar might change in the "
+ "future. Please use 'isar-privileged' or 'isar-rootless'.")
+ build_system = 'isar-privileged'
+ return build_system

def find_missing_repos(self, repo_paths={}):
"""
diff --git a/kas/libcmds.py b/kas/libcmds.py
index 70eb95cd1..e4d059b12 100644
--- a/kas/libcmds.py
+++ b/kas/libcmds.py
@@ -518,6 +518,8 @@ class WriteBBConfig(Command):
fds.write(f'DISTRO ??= "{ctx.config.get_distro()}"\n')
fds.write('BBMULTICONFIG ?= '
f'"{ctx.config.get_multiconfig()}"\n')
+ if ctx.config.get_build_system() == 'isar-rootless':
+ fds.write('ISAR_ROOTLESS ?= "1"\n')

_write_bblayers_conf(ctx)
_write_local_conf(ctx)
diff --git a/kas/libkas.py b/kas/libkas.py
index f5f5c45ed..4c848afcb 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -405,7 +405,7 @@ def get_build_environ(build_system):
init_repo = None
if build_system in ['openembedded', 'oe']:
scripts = ['oe-init-build-env']
- elif build_system == 'isar':
+ elif build_system.startswith('isar'):
scripts = ['isar-init-build-env']
else:
scripts = ['oe-init-build-env', 'isar-init-build-env']
diff --git a/kas/plugins/menu.py b/kas/plugins/menu.py
index c5bc725a7..9288ce73d 100644
--- a/kas/plugins/menu.py
+++ b/kas/plugins/menu.py
@@ -50,7 +50,7 @@

- The ``build_system`` that will used. The static Kconfig string variable
``KAS_BUILD_SYSTEM`` defines this value which must be ``openembedded``,
- ``oe`` or ``isar`` is set.
+ ``oe``, ``isar``, ``isar-privileged`` or ``isar-rootless`` is set.

- bitbake configuration variables that will be added to the
``local_conf_header`` section of the generated configuration. All other
--
2.53.0

[Felix Moessbauer]

With that, we also explicitly check the isar-privileged configuration.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

tests/test_build_system.py | 12 ++++++++++++
tests/test_build_system/test-isar-privileged.yml | 7 +++++++
tests/test_build_system/test-isar-rootless.yml | 7 +++++++
3 files changed, 26 insertions(+)

create mode 100644 tests/test_build_system/test-isar-privileged.yml
create mode 100644 tests/test_build_system/test-isar-rootless.yml

diff --git a/tests/test_build_system.py b/tests/test_build_system.py
index b46d559a8..abab4a351 100644
--- a/tests/test_build_system.py
+++ b/tests/test_build_system.py
@@ -22,6 +22,7 @@

import shutil
import pytest
+import re
from kas import kas


@@ -38,6 +39,17 @@ def test_build_system(monkeykas, tmpdir):
with open('build-env', 'r') as f:
assert f.readline().strip() == 'isar'

+ kas.kas(['shell', 'test-isar-privileged.yml', '-c', 'true'])
+ with open('build-env', 'r') as f:
+ assert f.readline().strip() == 'isar'
+
+ kas.kas(['shell', 'test-isar-rootless.yml', '-c', 'true'])
+ with open('build-env', 'r') as f:
+ assert f.readline().strip() == 'isar'
+ with open(monkeykas.get_kbd() / 'conf/local.conf', 'r') as f:
+ assert any(re.match(r'^ISAR_ROOTLESS.*"1"', line)
+ for line in f.readlines())
+
kas.kas(['shell', 'test-openembedded.yml', '-c', 'true'])
with open('build-env', 'r') as f:
assert f.readline().strip() == 'openembedded'
diff --git a/tests/test_build_system/test-isar-privileged.yml b/tests/test_build_system/test-isar-privileged.yml
new file mode 100644
index 000000000..4b13734f6
--- /dev/null
+++ b/tests/test_build_system/test-isar-privileged.yml
@@ -0,0 +1,7 @@
+header:
+ version: 22
+
+build_system: isar-privileged
+
+repos:
+ this:
diff --git a/tests/test_build_system/test-isar-rootless.yml b/tests/test_build_system/test-isar-rootless.yml
new file mode 100644
index 000000000..edabcb0b4
--- /dev/null
+++ b/tests/test_build_system/test-isar-rootless.yml
@@ -0,0 +1,7 @@
+header:
+ version: 22
+
+build_system: isar-rootless
+
+repos:
+ this:
--
2.53.0