[PATCH v7 0/8] kas-container: docker rootless support

10 views
Skip to first unread message

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:50 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Docker rootless is becoming more and more popular, with recent
support added in Ubuntu 24.04. It's execution model is similar to
podman, but lacks the --userns=keep-id option. By that, the user namespace
is always unshared from the host, creating uid/gid alignment issues on
the bind mounts.

For OE, this is not really an issue, as long as all data that is generated
by the container is only read by the host (and vice versa). The ISAR mode
conceptually can't work, as it requires additional privileges like the
binfmt_misc support and privileges for mknod.

These patches add limited (but probably sufficient) support for docker
rootless, as well as proper fallback paths to the system docker in case
of ISAR.

Changes since v6:

- rebased onto next
- sudo: add reason why we need it in prompt
- clean: do not run apply patches and write bb config steps
- purge: add option to preserve reference repos


Changes since v5:

- rebased onto next
- completely redesigned the bind-mounting of directories into the container:
Only if the directories are not relative to the kas work dir, they are
explicitly mounted (Patch 1).
- lessen requirements under rootless (e.g. KAS_WORK_DIR must not necessarily
be distinct)
- improved purge command handling of overlapping directories

Changes since v4:

- split feature addition and tests into separate patches
- ci: split env vars change and test addition into separate patches
- ci: test if commands (clean and purge) actually remove data
- rebased onto next

Changes since v3:

- completely redesign the clean* kas-container commands: These are now
kas commands and are always executed inside the container.
- add tests for the new clean* kas-container commands
- by having the clean* commands, we don't need the rootless clean quirks
anymore.

Changes since v2:

- avoid bash globbing in rm commands, use find instead
- fix style issues
- do not remove the whole directory, only the content if provided by the user

Changes since v1:

- restore directory ownership on rootless clean
- add kas-container prune command to remove all files created by kas
(and bitbake) inside the container, as well as restore the directory
ownerships

Best regards,
Felix

Felix Moessbauer (8):
add family of commands for cleaning
kas-container: run clean commands inside container
add kas purge command to remove all managed data
add tests for kas purge command
kas-container: add support for purge
ci: define kas container vars for whole test step
ci: run kas clean and purge after build
kas-container: add limited support for docker rootless

.github/workflows/next.yml | 14 +-
container-entrypoint | 38 ++-
docs/_man/kas-plugin-clean.rst | 23 ++
docs/_man/kas-plugin-cleanall.rst | 23 ++
docs/_man/kas-plugin-cleansstate.rst | 23 ++
docs/_man/kas-plugin-purge.rst | 23 ++
docs/conf.py | 9 +
docs/userguide/kas-container-description.inc | 12 +
docs/userguide/plugins.rst | 48 ++++
kas-container | 47 +++-
kas/context.py | 4 +
kas/plugins/__init__.py | 2 +
kas/plugins/clean.py | 229 +++++++++++++++++++
kas/plugins/shell.py | 4 +
scripts/kas-container-usage-to-rst.sh | 2 +-
tests/test_commands.py | 3 +
tests/test_menu.py | 5 +
17 files changed, 494 insertions(+), 15 deletions(-)
create mode 100644 docs/_man/kas-plugin-clean.rst
create mode 100644 docs/_man/kas-plugin-cleanall.rst
create mode 100644 docs/_man/kas-plugin-cleansstate.rst
create mode 100644 docs/_man/kas-plugin-purge.rst
create mode 100644 kas/plugins/clean.py

--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:51 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
As kas is now able to perform the various clean commands and also the
special handling outside of the container is getting too complex on
rootless container execution, we just forward the cleaning task to kas
inside the container.

By that, all clean-related special handling in the kas-container is
removed as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
kas-container | 34 +---------------------------------
1 file changed, 1 insertion(+), 33 deletions(-)

diff --git a/kas-container b/kas-container
index b202a4cd7..727936c5f 100755
--- a/kas-container
+++ b/kas-container
@@ -155,32 +155,6 @@ enable_unpriv_userns_docker()
fi
}

-run_clean()
-{
- KAS_BUILD_DIR="$(realpath "${KAS_BUILD_DIR:-${KAS_WORK_DIR}/build}")"
- trace mkdir -p "${KAS_BUILD_DIR}"
- if [ -n "${KAS_ISAR_ARGS}" ]; then
- # SC2086: Double quote to prevent globbing and word splitting.
- # shellcheck disable=2086
- trace ${KAS_CONTAINER_COMMAND} run -v "${KAS_BUILD_DIR}":/build:rw \
- --workdir=/build --rm ${KAS_ISAR_ARGS} \
- ${KAS_CONTAINER_IMAGE} \
- sudo rm -rf tmp
- else
- trace rm -rf "${KAS_BUILD_DIR}"/tmp*
- fi
-
- if [ "$1" != "clean" ]; then
- SSTATE_DIR=${SSTATE_DIR:-${KAS_BUILD_DIR}/sstate-cache}
- trace rm -rf "${SSTATE_DIR}"
-
- if [ "$1" = "cleanall" ]; then
- DL_DIR=${DL_DIR:-${KAS_BUILD_DIR}/downloads}
- trace rm -rf "${DL_DIR}"
- fi
- fi
-}
-
# Params: NAME CREATE_MODE
check_and_expand()
{
@@ -370,6 +344,7 @@ while [ $# -gt 0 ]; do
usage
;;
clean|cleansstate|cleanall)
+ KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1
break
@@ -517,13 +492,6 @@ fi

set_container_image_var

-# short circuit for clean* commands. We just need
-# the build-system information, but no repo mounts, etc...
-if echo "${KAS_CMD}" | grep -qe "^clean"; then
- run_clean "${KAS_CMD}"
- exit 0
-fi
-
KAS_REPO_MOUNT_OPT="${KAS_REPO_MOUNT_OPT:-${KAS_REPO_MOUNT_OPT_DEFAULT}}"

KAS_FILES="$(echo "${KAS_FILES}" | sed 's|'"${KAS_REPO_DIR}"'/|/repo/|g')"
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:51 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Previously we had the clean / cleansstate / cleanall commands only
implemented for the kas-container. This proved not sufficient under some
scenarios (like rootless), as the cleaning needs to happen from inside
the container. Anyways, it was an inconsitency between kas-container and
kas.

We now add support for these commands in kas as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
container-entrypoint | 2 +-
docs/_man/kas-plugin-clean.rst | 22 ++++
docs/_man/kas-plugin-cleanall.rst | 22 ++++
docs/_man/kas-plugin-cleansstate.rst | 22 ++++
docs/conf.py | 7 ++
docs/userguide/plugins.rst | 37 +++++++
kas/plugins/__init__.py | 2 +
kas/plugins/clean.py | 150 +++++++++++++++++++++++++++
8 files changed, 263 insertions(+), 1 deletion(-)
create mode 100644 docs/_man/kas-plugin-clean.rst
create mode 100644 docs/_man/kas-plugin-cleanall.rst
create mode 100644 docs/_man/kas-plugin-cleansstate.rst
create mode 100644 kas/plugins/clean.py

diff --git a/container-entrypoint b/container-entrypoint
index 41ce89d8d..aad89d2fc 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -58,7 +58,7 @@ fi

if [ -n "$1" ]; then
case "$1" in
- build|checkout|dump|for-all-repos|lock|menu|shell|-*)
+ build|checkout|clean*|dump|for-all-repos|lock|menu|shell|-*)
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=2086
exec $GOSU kas "$@"
diff --git a/docs/_man/kas-plugin-clean.rst b/docs/_man/kas-plugin-clean.rst
new file mode 100644
index 000000000..3a0d1c8ae
--- /dev/null
+++ b/docs/_man/kas-plugin-clean.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas clean command
+=================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: clean
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.Clean
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleanall.rst b/docs/_man/kas-plugin-cleanall.rst
new file mode 100644
index 000000000..e67e9b57c
--- /dev/null
+++ b/docs/_man/kas-plugin-cleanall.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas cleanall command
+====================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleanall
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.CleanAll
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleansstate.rst b/docs/_man/kas-plugin-cleansstate.rst
new file mode 100644
index 000000000..270a0d564
--- /dev/null
+++ b/docs/_man/kas-plugin-cleansstate.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas cleansstate command
+=======================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleansstate
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.CleanSstate
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleanall`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/conf.py b/docs/conf.py
index 5c87aade5..1806d3019 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -331,6 +331,13 @@ man_pages = [
[author], 1),
('_man/kas-plugin-checkout', 'kas-checkout', 'kas checkout plugin',
[author], 1),
+ ('_man/kas-plugin-clean', 'kas-clean', 'kas clean command',
+ [author], 1),
+ ('_man/kas-plugin-cleanall', 'kas-cleanall', 'kas cleanall command',
+ [author], 1),
+ ('_man/kas-plugin-cleansstate', 'kas-cleansstate',
+ 'kas cleansstate command',
+ [author], 1),
('_man/kas-plugin-dump', 'kas-dump', 'kas dump plugin',
[author], 1),
('_man/kas-plugin-for-all-repos',
diff --git a/docs/userguide/plugins.rst b/docs/userguide/plugins.rst
index 80b9e6e4f..892479d5c 100644
--- a/docs/userguide/plugins.rst
+++ b/docs/userguide/plugins.rst
@@ -26,6 +26,43 @@ typically provides a single command.
:prog: kas
:path: checkout

+``clean`` plugin
+----------------
+
+.. automodule:: kas.plugins.clean
+
+``clean`` command
+^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.Clean
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: clean
+
+``cleansstate`` command
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.CleanSstate
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleansstate
+
+``cleanall`` command
+^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.CleanAll
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleanall

``dump`` plugin
---------------
diff --git a/kas/plugins/__init__.py b/kas/plugins/__init__.py
index 8c772a262..3d4c10ad2 100644
--- a/kas/plugins/__init__.py
+++ b/kas/plugins/__init__.py
@@ -41,6 +41,7 @@ def load():
from . import build
from . import for_all_repos
from . import checkout
+ from . import clean
from . import shell
from . import menu
from . import dump
@@ -48,6 +49,7 @@ def load():

register_plugins(build)
register_plugins(checkout)
+ register_plugins(clean)
register_plugins(dump)
register_plugins(for_all_repos)
register_plugins(lock)
diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
new file mode 100644
index 000000000..e96cd1cf5
--- /dev/null
+++ b/kas/plugins/clean.py
@@ -0,0 +1,150 @@
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Siemens, 2025
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+ This plugin implements the ``kas {clean,cleansstate,cleanall}``
+ commands. In case a configuration file is provided, it will be used to
+ determine the build system and the files managed by kas.
+"""
+
+import os
+import shutil
+import logging
+import subprocess
+from pathlib import Path
+from kas.context import create_global_context, get_context
+from kas.config import Config, CONFIG_YAML_FILE
+from kas.libcmds import Macro
+
+__license__ = 'MIT'
+__copyright__ = 'Copyright (c) Siemens, 2025'
+
+
+class Clean():
+ """
+ Clean the build artifacts by removing the build artifacts
+ directory.
+ """
+
+ name = 'clean'
+ helpmsg = (
+ 'Clean build artifacts, keep sstate cache and downloads.'
+ )
+
+ @classmethod
+ def setup_parser(cls, parser):
+ parser.add_argument('--dry-run',
+ action='store_true',
+ default=False,
+ help='Do not remove anything, just print what '
+ 'would be removed')
+ parser.add_argument('--isar',
+ action='store_true',
+ default=False,
+ help='Use ISAR build directory layout')
+ parser.add_argument('config',
+ help='Config file(s), separated by colon. Using '
+ '.config.yaml in KAS_WORK_DIR if existing '
+ 'and none is specified.',
+ nargs='?')
+
+ def run(self, args):
+ ctx = create_global_context(args)
+ build_system = None
+ if args.config or (Path(ctx.kas_work_dir) / CONFIG_YAML_FILE).exists():
+ ctx.config = Config(ctx, args.config)
+ macro = Macro()
+ macro.run(ctx, ['setup_environ'])
+ build_system = ctx.config.get_build_system()
+ if args.isar:
+ build_system = 'isar'
+
+ logging.debug('Run clean in "%s" mode' % (build_system or 'default'))
+ if args.dry_run:
+ logging.warning('Dry run, not removing anything')
+ tmpdirs = Path(ctx.build_dir).glob('tmp*')
+ for tmpdir in tmpdirs:
+ logging.info(f'Removing {tmpdir}')
+ if args.dry_run:
+ continue
+ if build_system == 'isar':
+ clean_args = [
+ 'sudo', '--prompt', '[sudo] enter password for %U '
+ f'to clean ISAR artifacts in {tmpdir}',
+ 'rm', '-rf', str(tmpdir)]
+ subprocess.check_call(clean_args)
+ else:
+ shutil.rmtree(tmpdir)
+
+ @staticmethod
+ def clear_dir_content(directory):
+ """
+ Clear the contents of a directory without removing the dir itself.
+ """
+ for item in directory.iterdir():
+ if item.is_dir():
+ shutil.rmtree(item)
+ else:
+ item.unlink()
+
+
+class CleanSstate(Clean):
+ """
+ Removes the build artifacts and the empties the sstate cache.
+ """
+
+ name = 'cleansstate'
+ helpmsg = (
+ 'Clean build artifacts and sstate cache.'
+ )
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ sstate_dir = Path(os.environ.get('SSTATE_DIR',
+ Path(ctx.build_dir) / 'sstate-cache'))
+ if sstate_dir.exists():
+ logging.info(f'Removing {sstate_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(sstate_dir)
+
+
+class CleanAll(CleanSstate):
+ """
+ Removes the build artifacts, empties the sstate cache and the downloads.
+ """
+
+ name = 'cleanall'
+ helpmsg = (
+ 'Clean build artifacts, sstate-cache and downloads.'
+ )
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ downloads_dir = Path(ctx.build_dir) / 'downloads'
+ if downloads_dir.exists():
+ logging.info(f'Removing {downloads_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(downloads_dir)
+
+
+__KAS_PLUGINS__ = [Clean, CleanSstate, CleanAll]
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:52 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
The kas purge command is similar to kas cleanall, but also removes all
repos managed by kas (including the corresponding repos in
KAS_REPO_REF_DIR). Once executed, no kas or bitbake managed data remains
in any of the provided directories (whereby managed is refering to the
current configuration and environment settings).

The user specified dirs are not removed, as these are managed by the
calling user and could be mountpoints.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
container-entrypoint | 2 +-
docs/_man/kas-plugin-clean.rst | 3 +-
docs/_man/kas-plugin-cleanall.rst | 3 +-
docs/_man/kas-plugin-cleansstate.rst | 3 +-
docs/_man/kas-plugin-purge.rst | 23 ++++++++
docs/conf.py | 2 +
docs/userguide/plugins.rst | 11 ++++
kas/context.py | 4 ++
kas/plugins/clean.py | 81 +++++++++++++++++++++++++++-
kas/plugins/shell.py | 4 ++
10 files changed, 131 insertions(+), 5 deletions(-)
create mode 100644 docs/_man/kas-plugin-purge.rst

diff --git a/container-entrypoint b/container-entrypoint
index aad89d2fc..0b6a5e1ab 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -58,7 +58,7 @@ fi

if [ -n "$1" ]; then
case "$1" in
- build|checkout|clean*|dump|for-all-repos|lock|menu|shell|-*)
+ build|checkout|clean*|dump|for-all-repos|lock|menu|purge|shell|-*)
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=2086
exec $GOSU kas "$@"
diff --git a/docs/_man/kas-plugin-clean.rst b/docs/_man/kas-plugin-clean.rst
index 3a0d1c8ae..cd2269606 100644
--- a/docs/_man/kas-plugin-clean.rst
+++ b/docs/_man/kas-plugin-clean.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-cleanall`,
-:manpage:`kas-plugin-cleansstate`
+:manpage:`kas-plugin-cleansstate`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleanall.rst b/docs/_man/kas-plugin-cleanall.rst
index e67e9b57c..9149ad5d0 100644
--- a/docs/_man/kas-plugin-cleanall.rst
+++ b/docs/_man/kas-plugin-cleanall.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-clean`,
-:manpage:`kas-plugin-cleansstate`
+:manpage:`kas-plugin-cleansstate`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleansstate.rst b/docs/_man/kas-plugin-cleansstate.rst
index 270a0d564..04c14d734 100644
--- a/docs/_man/kas-plugin-cleansstate.rst
+++ b/docs/_man/kas-plugin-cleansstate.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-clean`,
-:manpage:`kas-plugin-cleanall`
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-purge.rst b/docs/_man/kas-plugin-purge.rst
new file mode 100644
index 000000000..ee214d03f
--- /dev/null
+++ b/docs/_man/kas-plugin-purge.rst
@@ -0,0 +1,23 @@
+:orphan:
+
+kas purge command
+=================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: purge
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.Purge
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/conf.py b/docs/conf.py
index 1806d3019..ef8ac6819 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -347,6 +347,8 @@ man_pages = [
[author], 1),
('_man/kas-plugin-menu', 'kas-menu', 'kas menu plugin',
[author], 1),
+ ('_man/kas-plugin-purge', 'kas-purge', 'kas purge command',
+ [author], 1),
('_man/kas-plugin-shell', 'kas-shell', 'kas shell plugin',
[author], 1),
('_man/kas-project-config',
diff --git a/docs/userguide/plugins.rst b/docs/userguide/plugins.rst
index 892479d5c..3b4521b2e 100644
--- a/docs/userguide/plugins.rst
+++ b/docs/userguide/plugins.rst
@@ -108,6 +108,17 @@ typically provides a single command.
:prog: kas
:path: menu

+``purge`` plugin
+----------------
+
+.. automodule:: kas.plugins.clean.Purge
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: purge
+
``shell`` plugin
----------------

diff --git a/kas/context.py b/kas/context.py
index bc669cbd9..ecfc0eb0d 100644
--- a/kas/context.py
+++ b/kas/context.py
@@ -91,6 +91,10 @@ class Context:
raise KasUserError('KAS_CLONE_DEPTH must be a number')
self.repo_clone_depth = max(int(clone_depth), 0)
self.setup_initial_environ()
+ # Register the paths that kas created and exclusively owns
+ self.managed_paths = set()
+ if not os.environ.get('KAS_BUILD_DIR'):
+ self.managed_paths.add(self.__kas_build_dir)
self.config = None
self.args = args

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index e96cd1cf5..8ea76ad37 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -33,6 +33,9 @@ from pathlib import Path
from kas.context import create_global_context, get_context
from kas.config import Config, CONFIG_YAML_FILE
from kas.libcmds import Macro
+from kas.kasusererror import KasUserError
+from kas import plugins
+

__license__ = 'MIT'
__copyright__ = 'Copyright (c) Siemens, 2025'
@@ -147,4 +150,80 @@ class CleanAll(CleanSstate):
self.clear_dir_content(downloads_dir)


-__KAS_PLUGINS__ = [Clean, CleanSstate, CleanAll]
+class Purge(CleanAll):
+ """
+ Clears the contents of the build directory, sstate-cache, downloads and
+ the repos managed by kas (including referenced repos in
+ ``KAS_REPO_REF_DIR``, if set). In ``KAS_WORK_DIR`` it will remove the
+ default configuration file and the ``KAS_BUILD_DIR`` (if present).
+ This command requires a configuration file to locate the managed repos.
+
+ .. note::
+ Before purging, kas needs to checkout and resolve all repos to locate
+ the repos managed by kas.
+ """
+
+ name = 'purge'
+ helpmsg = (
+ 'Purge all data managed by kas, including managed repos.'
+ )
+
+ @classmethod
+ def setup_parser(cls, parser):
+ super().setup_parser(parser)
+ parser.add_argument('--preserve-repo-refs',
+ action='store_true',
+ default=False,
+ help='Do not remove the reference repositories')
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ if not ctx.config:
+ raise KasUserError('Purge requires a config file to locate '
+ 'managed repos.')
+
+ for r in ctx.config.get_repos():
+ if r.operations_disabled:
+ logging.debug(f'Skipping {r.name} as not managed by kas')
+ continue
+ logging.info(f'Removing {r.path}')
+ if not args.dry_run:
+ shutil.rmtree(r.path)
+ if ctx.kas_repo_ref_dir and not args.preserve_repo_refs:
+ ref_repo = Path(ctx.kas_repo_ref_dir) / r.qualified_name
+ if ref_repo.exists():
+ logging.info(f'Removing {ref_repo}')
+ if not args.dry_run:
+ shutil.rmtree(ref_repo)
+
+ build_dir = Path(ctx.build_dir)
+ logging.info(f'Removing {build_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(build_dir)
+
+ work_dir = Path(ctx.kas_work_dir)
+ default_config = work_dir / CONFIG_YAML_FILE
+ if default_config.exists():
+ logging.info(f'Removing {default_config}')
+ if not args.dry_run:
+ default_config.unlink()
+
+ clean_paths = list(ctx.managed_paths)
+ # Plugins can register additional paths by providing get_managed_paths.
+ # These paths must be relative to the work_dir.
+ for plugin in plugins.all():
+ if hasattr(plugin, 'get_managed_paths'):
+ ppaths = plugin.get_managed_paths()
+ clean_paths.extend([work_dir / p for p in ppaths])
+
+ for path in [Path(p) for p in clean_paths if Path(p).exists()]:
+ logging.info(f'Removing {path}')
+ if not args.dry_run:
+ if path.is_file() or path.is_symlink():
+ path.unlink()
+ else:
+ shutil.rmtree(path)
+
+
+__KAS_PLUGINS__ = [Clean, CleanSstate, CleanAll, Purge]
diff --git a/kas/plugins/shell.py b/kas/plugins/shell.py
index 2633484f9..a0267b4dd 100644
--- a/kas/plugins/shell.py
+++ b/kas/plugins/shell.py
@@ -90,6 +90,10 @@ class Shell:
macro.add(ShellCommand(args.command))
macro.run(ctx, args.skip)

+ @classmethod
+ def get_managed_paths(cls):
+ return [SHELL_HISTORY_FILE]
+

class ShellCommand(Command):
"""
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:55 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
kas-container | 6 ++++--
scripts/kas-container-usage-to-rst.sh | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/kas-container b/kas-container
index 727936c5f..045974131 100755
--- a/kas-container
+++ b/kas-container
@@ -41,7 +41,7 @@ usage()
printf "%b" "Usage: ${SELF} [OPTIONS] { build | shell } [KASOPTIONS] [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] { checkout | dump | lock } [KASOPTIONS] [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] for-all-repos [KASOPTIONS] [KASFILE] COMMAND\n"
- printf "%b" " ${SELF} [OPTIONS] { clean | cleansstate | cleanall } [KASFILE]\n"
+ printf "%b" " ${SELF} [OPTIONS] { clean | cleansstate | cleanall | purge} [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] menu [KCONFIG]\n"
printf "%b" "\nPositional arguments:\n"
printf "%b" "build\t\t\tCheck out repositories and build target.\n"
@@ -57,6 +57,8 @@ usage()
"keep downloads.\n"
printf "%b" "cleanall\t\tClean build artifacts, sstate cache and " \
"downloads.\n"
+ printf "%b" "purge\t\t\tRemove all data managed by kas. Run with '--dry-run'\n"
+ printf "%b" " \t\t\tto check what would be removed\n"
printf "%b" "menu\t\t\tProvide configuration menu and trigger " \
"configured build.\n"
printf "%b" "\nOptional arguments:\n"
@@ -343,7 +345,7 @@ while [ $# -gt 0 ]; do
--*)
usage
;;
- clean|cleansstate|cleanall)
+ clean|cleansstate|cleanall|purge)
KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1
diff --git a/scripts/kas-container-usage-to-rst.sh b/scripts/kas-container-usage-to-rst.sh
index 48b74117c..6bee90d7c 100755
--- a/scripts/kas-container-usage-to-rst.sh
+++ b/scripts/kas-container-usage-to-rst.sh
@@ -35,7 +35,7 @@ cat - | \
perl -0pe 's/\n\s\s+/ /g' | \
sed 's/^Positional arguments:/|KAS-COMMANDS|\n--------------/g' | \
# each commands starts with a new line
- sed -r 's/^(build|checkout|dump|lock|shell|for-all-repos|clean|cleansstate|cleanall|menu)\t\t*(.*)$/:\1: \2/g' | \
+ sed -r 's/^(build|checkout|dump|lock|shell|for-all-repos|clean|cleansstate|cleanall|purge|menu)\t\t*(.*)$/:\1: \2/g' | \
sed 's/^Optional arguments:/|OPTIONS|\n---------/g' | \
sed '/^You can force/d' | \
cat
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:57 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer, Felix Moessbauer
From: Felix Moessbauer <felix.mo...@gmail.com>

By that we cover the more complex processing of purge in the container
as well (e.g. purging the overlapping mountpoints /work and /build).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
.github/workflows/next.yml | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 1c3e5d3b4..77c667efa 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -88,6 +88,15 @@ jobs:
run: |
cd image-tests/${{ matrix.image-name }}
../../kas-container build kas.yml
+ [ -d build/tmp ]
+ echo "Test kas clean"
+ ../../kas-container clean kas.yml
+ ! [ -d build/tmp/deploy ]
+ [ -d build/sstate-cache ] && [ -d build/downloads ]
+ [ -d poky ] || [ -d isar ]
+ echo "Test kas purge"
+ ../../kas-container purge kas.yml
+ ! [ -d poky ] && ! [ -d isar ] && ! [ -d build ]
- name: Complete build and deploy ${{ matrix.image-name }} image
if: github.ref == 'refs/heads/next'
uses: docker/build-push-action@v6
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:55:58 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
From: Felix Moessbauer <felix.mo...@gmail.com>

Docker rootless mode is similar to podman rootless mode, except that it
does not support to share the userid namespace. By that, the bind
mounted directories (like /repo, /work, /build) which are owned by the
calling user, are mapped with uid==gid==0 inside the container.
While we could align this by running as root inside the container, this
is not an option as bitbake does not allow this.

This comes with the following limitations:

- /repo must be mounted ro to not destroy the uid mappings on the host
- /work, /build shall be exclusive dirs on the host that is only
written to by the tooling inside kas-container
- a git safe.dirs exception is needed as git operates as builder on
repos owned by root
- only operations that do not strictly require /repo:rw are supported.
- ISAR mode is not supported in rootless mode, fallback to system docker

Closes: #124

Signed-off-by: Felix Moessbauer <felix.mo...@gmail.com>
---
container-entrypoint | 36 ++++++++++++++++++--
docs/userguide/kas-container-description.inc | 12 +++++++
kas-container | 35 +++++++++++++++++++
3 files changed, 80 insertions(+), 3 deletions(-)

diff --git a/container-entrypoint b/container-entrypoint
index 0b6a5e1ab..915a747b1 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -24,6 +24,20 @@

# kas-isar: sudo update-binfmts --enable && [ -f /proc/sys/fs/binfmt_misc/status ]

+chown_managed_dirs()
+{
+ for DIR in /build /work /sstate /downloads /repo-ref; do
+ if [ -d "$DIR" ]; then
+ chown "$1":"$2" "$DIR"
+ fi
+ done
+}
+
+restore_managed_dirs_owner()
+{
+ chown_managed_dirs 0 0
+}
+
if mount | grep -q "on / type aufs"; then
cat <<EOF >&2
WARNING: Generation of wic images will fail!
@@ -51,6 +65,15 @@ else

GOSU="gosu builder"
fi
+# kas-container on rootless docker workaround
+if [ -n "$USER_ID" ] && [ "$USER_ID" -ne 0 ] && \
+ [ "$KAS_DOCKER_ROOTLESS" = "1" ] && [ "$(stat -c %u /repo)" -eq 0 ]; then
+ # Docker rootless does not support keeping the user namespace
+ # (podman option --userns=keep-id). By that, the bind mounts
+ # are owned by root.
+ sudo git config --system safe.directory /repo
+ chown_managed_dirs "$USER_ID" "$GROUP_ID"
+fi

if [ "$PWD" = / ]; then
cd /builder || exit 1
@@ -59,9 +82,16 @@ fi
if [ -n "$1" ]; then
case "$1" in
build|checkout|clean*|dump|for-all-repos|lock|menu|purge|shell|-*)
- # SC2086: Double quote to prevent globbing and word splitting.
- # shellcheck disable=2086
- exec $GOSU kas "$@"
+ # We must restore the dir owner after every kas invocation.
+ # This is cheap as only the top-level dirs are changed (non recursive).
+ if [ "$KAS_DOCKER_ROOTLESS" = "1" ]; then
+ trap restore_managed_dirs_owner EXIT INT TERM
+ $GOSU kas "$@"
+ else
+ # SC2086: Double quote to prevent globbing and word splitting.
+ # shellcheck disable=2086
+ exec $GOSU kas "$@"
+ fi
;;
*)
# SC2086: Double quote to prevent globbing and word splitting.
diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
index ac8397a9e..ad8e5030e 100644
--- a/docs/userguide/kas-container-description.inc
+++ b/docs/userguide/kas-container-description.inc
@@ -14,3 +14,15 @@ By default ``kas-container`` uses the official images provided by the kas projec
``KAS_CONTAINER_IMAGE`` environment variable. As container backends, Docker and
Podman are supported. To force the use of podman over docker, set
``KAS_CONTAINER_ENGINE=podman``. For details, see :ref:`env-vars-label`.
+
+Running under docker in `rootless mode <https://docs.docker.com/engine/security/rootless/>`_
+is partially supported. It is recommended to use a distinct ``KAS_WORK_DIR`` outside of the
+calling directory (repo-dir), as kas temporarily changes the ownership of the working
+directory during its operation. All files managed by kas (including the repos) must not be
+written to from the host. To completely remove all data managed by kas, use
+``kas-container purge``. This also restores the directory owners of the dirs passed to kas,
+so they can be removed from the host.
+
+.. note::
+ The ISAR build system is not compatible with rootless execution. By that,
+ we fall back to the system docker or podman instance.
diff --git a/kas-container b/kas-container
index 045974131..e440e3421 100755
--- a/kas-container
+++ b/kas-container
@@ -130,6 +130,12 @@ enable_isar_mode()
KAS_CONTAINER_COMMAND="sudo --preserve-env ${KAS_CONTAINER_COMMAND}"
# preserved user PATH may lack sbin needed by privileged podman
export PATH="${PATH}:/usr/sbin"
+ elif [ "${KAS_DOCKER_ROOTLESS}" = "1" ]; then
+ export DOCKER_HOST="${DOCKER_HOST:-unix:///var/run/docker.sock}"
+ debug "kas-isar does not support rootless docker. Using system docker"
+ # force use of well-known system docker socket
+ KAS_CONTAINER_COMMAND="sudo --preserve-env ${KAS_CONTAINER_COMMAND}"
+ KAS_DOCKER_ROOTLESS=0
fi
}

@@ -190,6 +196,27 @@ forward_dir()
fi
}

+check_docker_rootless()
+{
+ KAS_DOCKER_ROOTLESS=0
+ if [ "$(docker context show)" = "rootless" ]; then
+ KAS_DOCKER_ROOTLESS=1
+ fi
+}
+
+enable_docker_rootless()
+{
+ warning "Rootless docker used, only limited functionality available."
+ if [ "${KAS_WORK_DIR}" = "${KAS_REPO_DIR}" ]; then
+ warning "On docker rootless a exclusive KAS_WORK_DIR should be used" \
+ "as kas temporarily changes the ownership of this directory."
+ fi
+ if [ "${KAS_REPO_MOUNT_OPT}" = "rw" ]; then
+ fatal_error "Docker rootless requires read-only repo."
+ fi
+ KAS_RUNTIME_ARGS="${KAS_RUNTIME_ARGS} -e KAS_DOCKER_ROOTLESS=1"
+}
+
KAS_GIT_OVERLAY_FILE=""
kas_container_cleanup()
{
@@ -251,6 +278,7 @@ case "${KAS_CONTAINER_ENGINE}" in
docker)
KAS_CONTAINER_COMMAND="docker"
enable_unpriv_userns_docker
+ check_docker_rootless
;;
podman)
KAS_CONTAINER_COMMAND="podman"
@@ -494,6 +522,9 @@ fi

set_container_image_var

+if [ "${KAS_DOCKER_ROOTLESS}" = "1" ]; then
+ KAS_REPO_MOUNT_OPT_DEFAULT="ro"
+fi
KAS_REPO_MOUNT_OPT="${KAS_REPO_MOUNT_OPT:-${KAS_REPO_MOUNT_OPT_DEFAULT}}"

KAS_FILES="$(echo "${KAS_FILES}" | sed 's|'"${KAS_REPO_DIR}"'/|/repo/|g')"
@@ -504,6 +535,10 @@ if [ "$(id -u)" -eq 0 ] && [ "${KAS_ALLOW_ROOT}" != "yes" ] ; then
"KAS_ALLOW_ROOT=yes to override."
fi

+if [ "${KAS_DOCKER_ROOTLESS}" = "1" ]; then
+ enable_docker_rootless
+fi
+
set -- "$@" -v "${KAS_REPO_DIR}:/repo:${KAS_REPO_MOUNT_OPT}" \
-v "${KAS_WORK_DIR}":/work:rw -e KAS_WORK_DIR=/work \
--workdir=/repo \
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:56:02 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Previously the kas container environment variables (e.g. to control
which image is used) were only defined for a single kas-container
invocation. As a preparation to add more tests to that step, we define
the variables for the whole step.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
.github/workflows/next.yml | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 403e75a8a..1c3e5d3b4 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -82,10 +82,11 @@ jobs:
outputs: type=docker,rewrite-timestamp=true
tags: ghcr.io/${{ github.repository }}/${{ matrix.image-name }}:next
- name: Test ${{ matrix.image-name }} image
+ env:
+ KAS_CONTAINER_IMAGE: ghcr.io/${{ github.repository }}/${{ matrix.image-name }}:next
+ KAS_CLONE_DEPTH: 1
run: |
cd image-tests/${{ matrix.image-name }}
- KAS_CONTAINER_IMAGE=ghcr.io/${{ github.repository }}/${{ matrix.image-name }}:next \
- KAS_CLONE_DEPTH=1 \
../../kas-container build kas.yml
- name: Complete build and deploy ${{ matrix.image-name }} image
if: github.ref == 'refs/heads/next'
--
2.47.2

Felix Moessbauer

unread,
Mar 5, 2025, 2:56:22 PM3/5/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
We add a test both for checkout with repo-ref-dir, as well as one for a
kas menu build that generates a config. As the purge command calls the
clean, cleansstate and cleanall commands internally, we transitively
test them as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
tests/test_commands.py | 3 +++
tests/test_menu.py | 5 +++++
2 files changed, 8 insertions(+)

diff --git a/tests/test_commands.py b/tests/test_commands.py
index a9448f39a..b108adef5 100644
--- a/tests/test_commands.py
+++ b/tests/test_commands.py
@@ -112,6 +112,9 @@ def test_checkout_create_refs(monkeykas, tmpdir):
kas.kas(['checkout', 'test.yml'])
assert os.path.exists(str(repo_cache / 'github.com.siemens.kas.git'))
assert os.path.exists('kas_1.0/.git/objects/info/alternates')
+ # check if refs are removed on purge
+ kas.kas(['purge', 'test.yml'])
+ assert not os.path.exists(repo_cache / 'github.com.siemens.kas.git')


@pytest.mark.online
diff --git a/tests/test_menu.py b/tests/test_menu.py
index 9c73e32e6..6b4891911 100644
--- a/tests/test_menu.py
+++ b/tests/test_menu.py
@@ -90,6 +90,11 @@ def test_menu_inc_workdir(monkeykas, tmpdir):
os.mkdir(kas_workdir)
monkeykas.setenv('KAS_WORK_DIR', kas_workdir)
kas.kas(['menu'])
+ default_config = os.path.join(kas_workdir, '.config.yaml')
+ assert os.path.exists(default_config)
+ # check if purge removes all files (including the generated .config.yml)
+ kas.kas(['purge'])
+ assert not os.path.exists(default_config)


def test_menu_implicit_workdir(monkeykas, tmpdir):
--
2.47.2

MOESSBAUER, Felix

unread,
Mar 6, 2025, 5:36:47 AM3/6/25
to kas-...@googlegroups.com, Kiszka, Jan
> diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
> new file mode 100644
> index 000000000..e96cd1cf5
> --- /dev/null
> +++ b/kas/plugins/clean.py
> @@ -0,0 +1,150 @@
>

This was actually a bad change (functionally it is correct, but now we
do more than needed). The last parameter describes the steps that
should be skipped. Probably we don't even need the use_common_setup for
normal "clean". That's just needed for the purge to track where things
come from.

Felix

> +            build_system = ctx.config.get_build_system()
> +        if args.isar:
> +            build_system = 'isar'
> +
>

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

Jan Kiszka

unread,
Mar 6, 2025, 6:49:51 AM3/6/25
to Moessbauer, Felix (FT RPD CED OES-DE), kas-...@googlegroups.com
Ouch. Positional parameters are evil...

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:32 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Docker rootless is becoming more and more popular, with recent
support added in Ubuntu 24.04. It's execution model is similar to
podman, but lacks the --userns=keep-id option. By that, the user namespace
is always unshared from the host, creating uid/gid alignment issues on
the bind mounts.

For OE, this is not really an issue, as long as all data that is generated
by the container is only read by the host (and vice versa). The ISAR mode
conceptually can't work, as it requires additional privileges like the
binfmt_misc support and privileges for mknod.

These patches add limited (but probably sufficient) support for docker
rootless, as well as proper fallback paths to the system docker in case
of ISAR.

Changes since v7:

- revert incorrect "clean: do not run apply patches" change from v6
As the use of positional arguments was misleading, we now use the
named argument "skip=" do denote that the passed commands are not executed.
- improve documentation of purge command
- add note, that we only can access the config once it has been fully
imported. @Jan: We should really consider adding an assertion if
config.get_config() is called without running the IncludeHandler upfront.
kas-container | 73 +++---
kas/context.py | 4 +
kas/plugins/__init__.py | 2 +
kas/plugins/clean.py | 232 +++++++++++++++++++
kas/plugins/shell.py | 4 +
scripts/kas-container-usage-to-rst.sh | 2 +-
tests/test_commands.py | 3 +
tests/test_menu.py | 5 +
17 files changed, 497 insertions(+), 41 deletions(-)
create mode 100644 docs/_man/kas-plugin-clean.rst
create mode 100644 docs/_man/kas-plugin-cleanall.rst
create mode 100644 docs/_man/kas-plugin-cleansstate.rst

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:32 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
As kas is now able to perform the various clean commands and also the
special handling outside of the container is getting too complex on
rootless container execution, we just forward the cleaning task to kas
inside the container.

By that, all clean-related special handling in the kas-container is
removed as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
kas-container | 34 +---------------------------------
1 file changed, 1 insertion(+), 33 deletions(-)

diff --git a/kas-container b/kas-container
index b202a4cd7..727936c5f 100755
--- a/kas-container
+++ b/kas-container
@@ -155,32 +155,6 @@ enable_unpriv_userns_docker()
fi
}

-run_clean()
-{
- KAS_BUILD_DIR="$(realpath "${KAS_BUILD_DIR:-${KAS_WORK_DIR}/build}")"
- trace mkdir -p "${KAS_BUILD_DIR}"
- if [ -n "${KAS_ISAR_ARGS}" ]; then
- # SC2086: Double quote to prevent globbing and word splitting.
- # shellcheck disable=2086
- trace ${KAS_CONTAINER_COMMAND} run -v "${KAS_BUILD_DIR}":/build:rw \
- --workdir=/build --rm ${KAS_ISAR_ARGS} \
- ${KAS_CONTAINER_IMAGE} \
- sudo rm -rf tmp
- else
- trace rm -rf "${KAS_BUILD_DIR}"/tmp*
- fi
-
- if [ "$1" != "clean" ]; then
- SSTATE_DIR=${SSTATE_DIR:-${KAS_BUILD_DIR}/sstate-cache}
- trace rm -rf "${SSTATE_DIR}"
-
- if [ "$1" = "cleanall" ]; then
- DL_DIR=${DL_DIR:-${KAS_BUILD_DIR}/downloads}
- trace rm -rf "${DL_DIR}"
- fi
- fi
-}
-
# Params: NAME CREATE_MODE
check_and_expand()
{
@@ -370,6 +344,7 @@ while [ $# -gt 0 ]; do
usage
;;
clean|cleansstate|cleanall)
+ KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1
break
@@ -517,13 +492,6 @@ fi

set_container_image_var

-# short circuit for clean* commands. We just need
-# the build-system information, but no repo mounts, etc...
-if echo "${KAS_CMD}" | grep -qe "^clean"; then
- run_clean "${KAS_CMD}"
- exit 0
-fi
-
KAS_REPO_MOUNT_OPT="${KAS_REPO_MOUNT_OPT:-${KAS_REPO_MOUNT_OPT_DEFAULT}}"

KAS_FILES="$(echo "${KAS_FILES}" | sed 's|'"${KAS_REPO_DIR}"'/|/repo/|g')"
--
2.47.2

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:32 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Previously we had the clean / cleansstate / cleanall commands only
implemented for the kas-container. This proved not sufficient under some
scenarios (like rootless), as the cleaning needs to happen from inside
the container. Anyways, it was an inconsitency between kas-container and
kas.

We now add support for these commands in kas as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
container-entrypoint | 2 +-
docs/_man/kas-plugin-clean.rst | 22 ++++
docs/_man/kas-plugin-cleanall.rst | 22 ++++
docs/_man/kas-plugin-cleansstate.rst | 22 ++++
docs/conf.py | 7 ++
docs/userguide/plugins.rst | 37 +++++++
kas/plugins/__init__.py | 2 +
kas/plugins/clean.py | 152 +++++++++++++++++++++++++++
8 files changed, 265 insertions(+), 1 deletion(-)
create mode 100644 docs/_man/kas-plugin-clean.rst
create mode 100644 docs/_man/kas-plugin-cleanall.rst
create mode 100644 docs/_man/kas-plugin-cleansstate.rst
create mode 100644 kas/plugins/clean.py

diff --git a/container-entrypoint b/container-entrypoint
index 41ce89d8d..aad89d2fc 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -58,7 +58,7 @@ fi

if [ -n "$1" ]; then
case "$1" in
- build|checkout|dump|for-all-repos|lock|menu|shell|-*)
+ build|checkout|clean*|dump|for-all-repos|lock|menu|shell|-*)
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=2086
exec $GOSU kas "$@"
diff --git a/docs/_man/kas-plugin-clean.rst b/docs/_man/kas-plugin-clean.rst
new file mode 100644
index 000000000..3a0d1c8ae
--- /dev/null
+++ b/docs/_man/kas-plugin-clean.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas clean command
+=================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: clean
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.Clean
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleanall.rst b/docs/_man/kas-plugin-cleanall.rst
new file mode 100644
index 000000000..e67e9b57c
--- /dev/null
+++ b/docs/_man/kas-plugin-cleanall.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas cleanall command
+====================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleanall
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.CleanAll
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleansstate.rst b/docs/_man/kas-plugin-cleansstate.rst
new file mode 100644
index 000000000..270a0d564
--- /dev/null
+++ b/docs/_man/kas-plugin-cleansstate.rst
@@ -0,0 +1,22 @@
+:orphan:
+
+kas cleansstate command
+=======================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleansstate
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.CleanSstate
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleanall`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/conf.py b/docs/conf.py
index 5c87aade5..1806d3019 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -331,6 +331,13 @@ man_pages = [
[author], 1),
('_man/kas-plugin-checkout', 'kas-checkout', 'kas checkout plugin',
[author], 1),
+ ('_man/kas-plugin-clean', 'kas-clean', 'kas clean command',
+ [author], 1),
+ ('_man/kas-plugin-cleanall', 'kas-cleanall', 'kas cleanall command',
+ [author], 1),
+ ('_man/kas-plugin-cleansstate', 'kas-cleansstate',
+ 'kas cleansstate command',
+ [author], 1),
('_man/kas-plugin-dump', 'kas-dump', 'kas dump plugin',
[author], 1),
('_man/kas-plugin-for-all-repos',
diff --git a/docs/userguide/plugins.rst b/docs/userguide/plugins.rst
index 80b9e6e4f..892479d5c 100644
--- a/docs/userguide/plugins.rst
+++ b/docs/userguide/plugins.rst
@@ -26,6 +26,43 @@ typically provides a single command.
:prog: kas
:path: checkout

+``clean`` plugin
+----------------
+
+.. automodule:: kas.plugins.clean
+
+``clean`` command
+^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.Clean
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: clean
+
+``cleansstate`` command
+^^^^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.CleanSstate
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleansstate
+
+``cleanall`` command
+^^^^^^^^^^^^^^^^^^^^
+
+.. automodule:: kas.plugins.clean.CleanAll
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: cleanall

``dump`` plugin
---------------
diff --git a/kas/plugins/__init__.py b/kas/plugins/__init__.py
index 8c772a262..3d4c10ad2 100644
--- a/kas/plugins/__init__.py
+++ b/kas/plugins/__init__.py
@@ -41,6 +41,7 @@ def load():
from . import build
from . import for_all_repos
from . import checkout
+ from . import clean
from . import shell
from . import menu
from . import dump
@@ -48,6 +49,7 @@ def load():

register_plugins(build)
register_plugins(checkout)
+ register_plugins(clean)
register_plugins(dump)
register_plugins(for_all_repos)
register_plugins(lock)
diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
new file mode 100644
index 000000000..b120f09ff
--- /dev/null
+++ b/kas/plugins/clean.py
@@ -0,0 +1,152 @@
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Siemens, 2025
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+"""
+ This plugin implements the ``kas {clean,cleansstate,cleanall}``
+ commands. In case a configuration file is provided, it will be used to
+ determine the build system and the files managed by kas.
+"""
+
+import os
+import shutil
+import logging
+import subprocess
+from pathlib import Path
+from kas.context import create_global_context, get_context
+from kas.config import Config, CONFIG_YAML_FILE
+from kas.libcmds import Macro
+
+__license__ = 'MIT'
+__copyright__ = 'Copyright (c) Siemens, 2025'
+
+
+ # to read the config, we need all repos (but no build env),
+ macro = Macro()
+ macro.run(ctx, skip=['repos_apply_patches', 'write_bb_config',
+ 'setup_environ'])
+ build_system = ctx.config.get_build_system()
+ if args.isar:
+ build_system = 'isar'
+
+ """
+
+ name = 'cleansstate'
+ helpmsg = (
+ 'Clean build artifacts and sstate cache.'
+ )
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ sstate_dir = Path(os.environ.get('SSTATE_DIR',
+ Path(ctx.build_dir) / 'sstate-cache'))
+ if sstate_dir.exists():
+ logging.info(f'Removing {sstate_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(sstate_dir)
+
+
+class CleanAll(CleanSstate):
+ """
+ Removes the build artifacts, empties the sstate cache and the downloads.
+ """
+
+ name = 'cleanall'
+ helpmsg = (
+ 'Clean build artifacts, sstate-cache and downloads.'
+ )
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ downloads_dir = Path(ctx.build_dir) / 'downloads'
+ if downloads_dir.exists():
+ logging.info(f'Removing {downloads_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(downloads_dir)
+
+
+__KAS_PLUGINS__ = [Clean, CleanSstate, CleanAll]
--
2.47.2

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:36 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
We add a test both for checkout with repo-ref-dir, as well as one for a
kas menu build that generates a config. As the purge command calls the
clean, cleansstate and cleanall commands internally, we transitively
test them as well.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
kas/plugins/clean.py | 1 +
tests/test_commands.py | 3 +++
tests/test_menu.py | 5 +++++
3 files changed, 9 insertions(+)

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index 2616d0124..0b4a79533 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -158,6 +158,7 @@ class Purge(CleanAll):
the repos managed by kas (including referenced repos in
``KAS_REPO_REF_DIR``, if set). In ``KAS_WORK_DIR`` it will remove the
default configuration file and the ``KAS_BUILD_DIR`` (if present).
+ To preserve the reference repositories, run with ``--preserve-repo-refs``.
This command requires a configuration file to locate the managed repos.

.. note::

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:37 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
The kas purge command is similar to kas cleanall, but also removes all
repos managed by kas (including the corresponding repos in
KAS_REPO_REF_DIR). Once executed, no kas or bitbake managed data remains
in any of the provided directories (whereby managed is refering to the
current configuration and environment settings).

The user specified dirs are not removed, as these are managed by the
calling user and could be mountpoints.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
container-entrypoint | 2 +-
docs/_man/kas-plugin-clean.rst | 3 +-
docs/_man/kas-plugin-cleanall.rst | 3 +-
docs/_man/kas-plugin-cleansstate.rst | 3 +-
docs/_man/kas-plugin-purge.rst | 23 ++++++++
docs/conf.py | 2 +
docs/userguide/plugins.rst | 11 ++++
kas/context.py | 4 ++
kas/plugins/clean.py | 81 +++++++++++++++++++++++++++-
kas/plugins/shell.py | 4 ++
10 files changed, 131 insertions(+), 5 deletions(-)
create mode 100644 docs/_man/kas-plugin-purge.rst

diff --git a/container-entrypoint b/container-entrypoint
index aad89d2fc..0b6a5e1ab 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -58,7 +58,7 @@ fi

if [ -n "$1" ]; then
case "$1" in
- build|checkout|clean*|dump|for-all-repos|lock|menu|shell|-*)
+ build|checkout|clean*|dump|for-all-repos|lock|menu|purge|shell|-*)
# SC2086: Double quote to prevent globbing and word splitting.
# shellcheck disable=2086
exec $GOSU kas "$@"
diff --git a/docs/_man/kas-plugin-clean.rst b/docs/_man/kas-plugin-clean.rst
index 3a0d1c8ae..cd2269606 100644
--- a/docs/_man/kas-plugin-clean.rst
+++ b/docs/_man/kas-plugin-clean.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-cleanall`,
-:manpage:`kas-plugin-cleansstate`
+:manpage:`kas-plugin-cleansstate`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleanall.rst b/docs/_man/kas-plugin-cleanall.rst
index e67e9b57c..9149ad5d0 100644
--- a/docs/_man/kas-plugin-cleanall.rst
+++ b/docs/_man/kas-plugin-cleanall.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-clean`,
-:manpage:`kas-plugin-cleansstate`
+:manpage:`kas-plugin-cleansstate`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-cleansstate.rst b/docs/_man/kas-plugin-cleansstate.rst
index 270a0d564..04c14d734 100644
--- a/docs/_man/kas-plugin-cleansstate.rst
+++ b/docs/_man/kas-plugin-cleansstate.rst
@@ -17,6 +17,7 @@ SEE ALSO
--------

:manpage:`kas-plugin-clean`,
-:manpage:`kas-plugin-cleanall`
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-purge`

.. include:: _kas-man-footer.inc
diff --git a/docs/_man/kas-plugin-purge.rst b/docs/_man/kas-plugin-purge.rst
new file mode 100644
index 000000000..ee214d03f
--- /dev/null
+++ b/docs/_man/kas-plugin-purge.rst
@@ -0,0 +1,23 @@
+:orphan:
+
+kas purge command
+=================
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: purge
+ :manpage:
+
+ .. automodule:: kas.plugins.clean.Purge
+ :noindex:
+
+SEE ALSO
+--------
+
+:manpage:`kas-plugin-clean`,
+:manpage:`kas-plugin-cleanall`,
+:manpage:`kas-plugin-cleansstate`
+
+.. include:: _kas-man-footer.inc
diff --git a/docs/conf.py b/docs/conf.py
index 1806d3019..ef8ac6819 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -347,6 +347,8 @@ man_pages = [
[author], 1),
('_man/kas-plugin-menu', 'kas-menu', 'kas menu plugin',
[author], 1),
+ ('_man/kas-plugin-purge', 'kas-purge', 'kas purge command',
+ [author], 1),
('_man/kas-plugin-shell', 'kas-shell', 'kas shell plugin',
[author], 1),
('_man/kas-project-config',
diff --git a/docs/userguide/plugins.rst b/docs/userguide/plugins.rst
index 892479d5c..3b4521b2e 100644
--- a/docs/userguide/plugins.rst
+++ b/docs/userguide/plugins.rst
@@ -108,6 +108,17 @@ typically provides a single command.
:prog: kas
:path: menu

+``purge`` plugin
+----------------
+
+.. automodule:: kas.plugins.clean.Purge
+
+.. argparse::
+ :module: kas.kas
+ :func: kas_get_argparser
+ :prog: kas
+ :path: purge
+
``shell`` plugin
----------------

diff --git a/kas/context.py b/kas/context.py
index bc669cbd9..ecfc0eb0d 100644
--- a/kas/context.py
+++ b/kas/context.py
@@ -91,6 +91,10 @@ class Context:
raise KasUserError('KAS_CLONE_DEPTH must be a number')
self.repo_clone_depth = max(int(clone_depth), 0)
self.setup_initial_environ()
+ # Register the paths that kas created and exclusively owns
+ self.managed_paths = set()
+ if not os.environ.get('KAS_BUILD_DIR'):
+ self.managed_paths.add(self.__kas_build_dir)
self.config = None
self.args = args

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index b120f09ff..2616d0124 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -33,6 +33,9 @@ from pathlib import Path
from kas.context import create_global_context, get_context
from kas.config import Config, CONFIG_YAML_FILE
from kas.libcmds import Macro
+from kas.kasusererror import KasUserError
+from kas import plugins
+

__license__ = 'MIT'
__copyright__ = 'Copyright (c) Siemens, 2025'
@@ -149,4 +152,80 @@ class CleanAll(CleanSstate):
self.clear_dir_content(downloads_dir)


-__KAS_PLUGINS__ = [Clean, CleanSstate, CleanAll]
+class Purge(CleanAll):
+ """
+ Clears the contents of the build directory, sstate-cache, downloads and
+ the repos managed by kas (including referenced repos in
+ ``KAS_REPO_REF_DIR``, if set). In ``KAS_WORK_DIR`` it will remove the
+ default configuration file and the ``KAS_BUILD_DIR`` (if present).
+ This command requires a configuration file to locate the managed repos.
+
+ .. note::
+ Before purging, kas needs to checkout and resolve all repos to locate
+ the repos managed by kas.
+ """
+
+ name = 'purge'
+ helpmsg = (
+ 'Purge all data managed by kas, including managed repos.'
+ )
+
+ @classmethod
+ def setup_parser(cls, parser):
+ super().setup_parser(parser)
+ parser.add_argument('--preserve-repo-refs',
+ action='store_true',
+ default=False,
+ help='Do not remove the reference repositories')
+
+ def run(self, args):
+ super().run(args)
+ ctx = get_context()
+ if not ctx.config:
+ raise KasUserError('Purge requires a config file to locate '
+ 'managed repos.')
+
+ for r in ctx.config.get_repos():
+ if r.operations_disabled:
+ logging.debug(f'Skipping {r.name} as not managed by kas')
+ continue
+ logging.info(f'Removing {r.path}')
+ if not args.dry_run:
+ shutil.rmtree(r.path)
+ if ctx.kas_repo_ref_dir and not args.preserve_repo_refs:
+ ref_repo = Path(ctx.kas_repo_ref_dir) / r.qualified_name
+ if ref_repo.exists():
+ logging.info(f'Removing {ref_repo}')
+ if not args.dry_run:
+ shutil.rmtree(ref_repo)
+
+ build_dir = Path(ctx.build_dir)
+ logging.info(f'Removing {build_dir}/*')
+ if not args.dry_run:
+ self.clear_dir_content(build_dir)
+
+ work_dir = Path(ctx.kas_work_dir)
+ default_config = work_dir / CONFIG_YAML_FILE
+ if default_config.exists():
+ logging.info(f'Removing {default_config}')
+ if not args.dry_run:
+ default_config.unlink()
+
+ clean_paths = list(ctx.managed_paths)
+ # Plugins can register additional paths by providing get_managed_paths.
+ # These paths must be relative to the work_dir.
+ for plugin in plugins.all():
+ if hasattr(plugin, 'get_managed_paths'):
+ ppaths = plugin.get_managed_paths()
+ clean_paths.extend([work_dir / p for p in ppaths])
+
+ for path in [Path(p) for p in clean_paths if Path(p).exists()]:
+ logging.info(f'Removing {path}')
+ if not args.dry_run:
+ if path.is_file() or path.is_symlink():
+ path.unlink()
+ else:
+ shutil.rmtree(path)
+
+

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:38 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Previously the kas container environment variables (e.g. to control
which image is used) were only defined for a single kas-container
invocation. As a preparation to add more tests to that step, we define
the variables for the whole step.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:38 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer, Felix Moessbauer
From: Felix Moessbauer <felix.mo...@gmail.com>

By that we cover the more complex processing of purge in the container
as well (e.g. purging the overlapping mountpoints /work and /build).

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
.github/workflows/next.yml | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 1c3e5d3b4..77c667efa 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -88,6 +88,15 @@ jobs:
run: |
cd image-tests/${{ matrix.image-name }}
../../kas-container build kas.yml
+ [ -d build/tmp ]
+ echo "Test kas clean"
+ ../../kas-container clean kas.yml
+ ! [ -d build/tmp/deploy ]
+ [ -d build/sstate-cache ] && [ -d build/downloads ]
+ [ -d poky ] || [ -d isar ]
+ echo "Test kas purge"
+ ../../kas-container purge kas.yml
+ ! [ -d poky ] && ! [ -d isar ] && ! [ -d build ]
- name: Complete build and deploy ${{ matrix.image-name }} image
if: github.ref == 'refs/heads/next'
uses: docker/build-push-action@v6
--
2.47.2

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:39 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
kas-container | 6 ++++--
scripts/kas-container-usage-to-rst.sh | 2 +-
2 files changed, 5 insertions(+), 3 deletions(-)

diff --git a/kas-container b/kas-container
index 727936c5f..045974131 100755
--- a/kas-container
+++ b/kas-container
@@ -41,7 +41,7 @@ usage()
printf "%b" "Usage: ${SELF} [OPTIONS] { build | shell } [KASOPTIONS] [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] { checkout | dump | lock } [KASOPTIONS] [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] for-all-repos [KASOPTIONS] [KASFILE] COMMAND\n"
- printf "%b" " ${SELF} [OPTIONS] { clean | cleansstate | cleanall } [KASFILE]\n"
+ printf "%b" " ${SELF} [OPTIONS] { clean | cleansstate | cleanall | purge} [KASFILE]\n"
printf "%b" " ${SELF} [OPTIONS] menu [KCONFIG]\n"
printf "%b" "\nPositional arguments:\n"
printf "%b" "build\t\t\tCheck out repositories and build target.\n"
@@ -57,6 +57,8 @@ usage()
"keep downloads.\n"
printf "%b" "cleanall\t\tClean build artifacts, sstate cache and " \
"downloads.\n"
+ printf "%b" "purge\t\t\tRemove all data managed by kas. Run with '--dry-run'\n"
+ printf "%b" " \t\t\tto check what would be removed\n"
printf "%b" "menu\t\t\tProvide configuration menu and trigger " \
"configured build.\n"
printf "%b" "\nOptional arguments:\n"
@@ -343,7 +345,7 @@ while [ $# -gt 0 ]; do
--*)
usage
;;
- clean|cleansstate|cleanall)
+ clean|cleansstate|cleanall|purge)
KAS_REPO_MOUNT_OPT_DEFAULT="ro"
KAS_CMD=$1
shift 1

Felix Moessbauer

unread,
Mar 6, 2025, 9:39:43 AM3/6/25
to kas-...@googlegroups.com, jan.k...@siemens.com, Felix Moessbauer
From: Felix Moessbauer <felix.mo...@gmail.com>

Docker rootless mode is similar to podman rootless mode, except that it
does not support to share the userid namespace. By that, the bind
mounted directories (like /repo, /work, /build) which are owned by the
calling user, are mapped with uid==gid==0 inside the container.
While we could align this by running as root inside the container, this
is not an option as bitbake does not allow this.

This comes with the following limitations:

- /repo must be mounted ro to not destroy the uid mappings on the host
- /work, /build shall be exclusive dirs on the host that is only
written to by the tooling inside kas-container
- a git safe.dirs exception is needed as git operates as builder on
repos owned by root
- only operations that do not strictly require /repo:rw are supported.
- ISAR mode is not supported in rootless mode, fallback to system docker

Closes: #124

Signed-off-by: Felix Moessbauer <felix.mo...@gmail.com>
---
container-entrypoint | 36 ++++++++++++++++++--
docs/userguide/kas-container-description.inc | 12 +++++++
kas-container | 35 +++++++++++++++++++
3 files changed, 80 insertions(+), 3 deletions(-)

diff --git a/container-entrypoint b/container-entrypoint
index 0b6a5e1ab..915a747b1 100755
--- a/container-entrypoint
+++ b/container-entrypoint
@@ -59,9 +82,16 @@ fi
if [ -n "$1" ]; then
case "$1" in
build|checkout|clean*|dump|for-all-repos|lock|menu|purge|shell|-*)
- # SC2086: Double quote to prevent globbing and word splitting.
- # shellcheck disable=2086
- exec $GOSU kas "$@"
+ # We must restore the dir owner after every kas invocation.
+ # This is cheap as only the top-level dirs are changed (non recursive).
+ if [ "$KAS_DOCKER_ROOTLESS" = "1" ]; then
+ trap restore_managed_dirs_owner EXIT INT TERM
+ $GOSU kas "$@"
+ else
+ # SC2086: Double quote to prevent globbing and word splitting.
+ # shellcheck disable=2086
+ exec $GOSU kas "$@"
+ fi
;;
*)
# SC2086: Double quote to prevent globbing and word splitting.
diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
index ac8397a9e..ad8e5030e 100644
--- a/docs/userguide/kas-container-description.inc
+++ b/docs/userguide/kas-container-description.inc
@@ -14,3 +14,15 @@ By default ``kas-container`` uses the official images provided by the kas projec
``KAS_CONTAINER_IMAGE`` environment variable. As container backends, Docker and
Podman are supported. To force the use of podman over docker, set
``KAS_CONTAINER_ENGINE=podman``. For details, see :ref:`env-vars-label`.
+
+Running under docker in `rootless mode <https://docs.docker.com/engine/security/rootless/>`_
+is partially supported. It is recommended to use a distinct ``KAS_WORK_DIR`` outside of the
+calling directory (repo-dir), as kas temporarily changes the ownership of the working
+directory during its operation. All files managed by kas (including the repos) must not be
+written to from the host. To completely remove all data managed by kas, use
+``kas-container purge``. This also restores the directory owners of the dirs passed to kas,
+so they can be removed from the host.
+
+.. note::
+ The ISAR build system is not compatible with rootless execution. By that,
+ we fall back to the system docker or podman instance.
diff --git a/kas-container b/kas-container
index 045974131..e440e3421 100755
--- a/kas-container
+++ b/kas-container
KAS_REPO_MOUNT_OPT="${KAS_REPO_MOUNT_OPT:-${KAS_REPO_MOUNT_OPT_DEFAULT}}"

KAS_FILES="$(echo "${KAS_FILES}" | sed 's|'"${KAS_REPO_DIR}"'/|/repo/|g')"

Jan Kiszka

unread,
Mar 27, 2025, 1:41:29 AM3/27/25
to Felix Moessbauer, kas-...@googlegroups.com
On 06.03.25 15:39, 'Felix Moessbauer' via kas-devel wrote:
> We add a test both for checkout with repo-ref-dir, as well as one for a
> kas menu build that generates a config. As the purge command calls the
> clean, cleansstate and cleanall commands internally, we transitively
> test them as well.
>
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
> ---
> kas/plugins/clean.py | 1 +
> tests/test_commands.py | 3 +++
> tests/test_menu.py | 5 +++++
> 3 files changed, 9 insertions(+)
>
> diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
> index 2616d0124..0b4a79533 100644
> --- a/kas/plugins/clean.py
> +++ b/kas/plugins/clean.py
> @@ -158,6 +158,7 @@ class Purge(CleanAll):
> the repos managed by kas (including referenced repos in
> ``KAS_REPO_REF_DIR``, if set). In ``KAS_WORK_DIR`` it will remove the
> default configuration file and the ``KAS_BUILD_DIR`` (if present).
> + To preserve the reference repositories, run with ``--preserve-repo-refs``.

This is unrelated. I've moved it into patch 3.

Jan

Jan Kiszka

unread,
Mar 27, 2025, 1:45:25 AM3/27/25
to Felix Moessbauer, kas-...@googlegroups.com
On 06.03.25 15:39, 'Felix Moessbauer' via kas-devel wrote:
Thanks, applied.

Jan
Reply all
Reply to author
Forward
0 new messages