[PATCH v7 0/8] Add buildtools support

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,

---

Changes in v7:
- add support for purging in 'clean' plugin
- fix code style issues
- add kas-buildtools to next.yml


Joao Marcos Costa (8):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR
image-tests: add kas-buildtools.yml
plugins/clean: handle buildtools in purge
ci: Add kas-buildtools to next.yml

.github/workflows/next.yml | 3 +
docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 70 +++++++++
examples/openembedded-buildtools.yml | 46 ++++++
image-tests/kas/kas-buildtools.yml | 1 +
kas-container | 2 +
kas/config.py | 9 ++
kas/libkas.py | 157 ++++++++++++++++++++
kas/plugins/clean.py | 10 ++
kas/schema-kas.json | 26 ++++
10 files changed, 328 insertions(+)
create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Add buildtools configuration scheme:
- version: sets the buildtools version to be fetched
- sha256sum: buildtools installer checksum, to verify integrity
- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---
kas/schema-kas.json | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index e1bff6d..db28b3d 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -406,6 +406,32 @@
"_source_dir_host": {
"description": "Source directory of the config file on the host (auto-generated by kas menu plugin, when using kas-container).",
"type": "string"
+ },
+ "buildtools": {
+ "type": "object",
+ "required": [
+ "version",
+ "sha256sum"
+ ],
+ "properties": {
+ "version": {
+ "description": "Yocto Project version, as 5.0 (or even 5.0.8) for Scarthgap.",
+ "type": "string"
+ },
+ "sha256sum": {
+ "description": "The installer's checksum, to perform integrity validation of the fetched artifact.",
+ "type": "string"
+ },
+ "base_url": {
+ "default": "https://downloads.yoctoproject.org/releases/yocto",
+ "description": "Base URL to fetch downloads from.",
+ "type": "string"
+ },
+ "filename": {
+ "description": "Alternative name for the buildtools archive (.sh) to be downloaded.",
+ "type": "string"
+ }
+ }
}
}
}
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/kas/config.py b/kas/config.py
index d868a5c..cebc749 100644
--- a/kas/config.py
+++ b/kas/config.py
@@ -264,3 +264,12 @@ class Config:
else:
return {k: v for k, v in signers.items()
if v.get('type', 'gpg') == keytype}
+
+ def get_buildtools(self):
+ """
+ Returns the buildtools keys: version, download URL and
+ archive filename. These are provided so kas knows which
+ buildtools archive to fetch and from what source.
+ """
+
+ return self._config.get('buildtools')
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add the core of buildtools support: download the installer, check its
integrity, extract it, and set up the environment.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 157 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 5327efb..7f7e240 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -31,11 +31,18 @@ import logging
import tempfile
import asyncio
import errno
+import hashlib
+import hmac
import pathlib
+import platform
+import shutil
import signal
+import stat
from subprocess import Popen, PIPE, run as subprocess_run
+from urllib.parse import quote
from .context import get_context
from .kasusererror import KasUserError, CommandExecError
+from .configschema import CONFIGSCHEMA

__license__ = 'MIT'
__copyright__ = 'Copyright (c) Siemens AG, 2017-2018'
@@ -260,6 +267,122 @@ def repos_apply_patches(repos):
raise TaskExecError('apply patches', e.ret_code)


+def get_buildtools_dir():
+ # Set the dest. directory for buildtools's setup
+ env_path = os.environ.get("KAS_BUILDTOOLS_DIR")
+ if env_path:
+ return pathlib.Path(env_path).resolve()
+
+ # defaults to KAS_BUILD_DIR/buildtools
+ return (pathlib.Path(get_context().build_dir) / 'buildtools').resolve()
+
+
+def get_buildtools_filename():
+ arch = platform.machine()
+ ctx = get_context()
+
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ if 'filename' in conf_buildtools:
+ filename = conf_buildtools['filename']
+ else:
+ filename = (
+ f"{arch}-buildtools-extended-"
+ f"nativesdk-standalone-{version}.sh"
+ )
+
+ return filename
+
+
+def get_buildtools_path():
+ return get_buildtools_dir() / get_buildtools_filename()
+
+
+def get_buildtools_url():
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ filename = get_buildtools_filename()
+ version = conf_buildtools['version']
+
+ if 'base_url' in conf_buildtools:
+ base_url = conf_buildtools['base_url']
+ else:
+ default = (
+ CONFIGSCHEMA['properties']['buildtools']['properties']
+ ['base_url']['default']
+ )
+ base_url = f"{default}/yocto-{version}/buildtools/"
+
+ return f"{base_url}/{quote(filename)}"
+
+
+def check_sha256sum(filename, expected_checksum):
+ hash_sha256 = hashlib.sha256()
+ with open(filename, "rb") as f:
+ for chunk in iter(lambda: f.read(8192), b""):
+ hash_sha256.update(chunk)
+
+ actual_checksum = hash_sha256.hexdigest()
+ logging.info(
+ f"Buildtools installer's checksum (sha256) is: "
+ f"{actual_checksum}"
+ )
+
+ return hmac.compare_digest(actual_checksum, expected_checksum)
+
+
+def download_buildtools():
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ buildtools_dir = get_buildtools_dir()
+
+ # Enable extended buildtools tarball
+ buildtools_url = get_buildtools_url()
+ tmpbuildtools = get_buildtools_path()
+
+ logging.info(f"Downloading Buildtools {version}")
+ # Download installer
+ fetch_cmd = ['wget', '-q', '-O', str(tmpbuildtools), buildtools_url]
+ (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
+ if ret != 0:
+ raise InitBuildEnvError("Could not download buildtools installer")
+
+ # Check if the installer's sha256sum matches
+ if not check_sha256sum(tmpbuildtools, conf_buildtools['sha256sum']):
+ raise InitBuildEnvError(
+ "sha256sum mismatch: installer may be corrupted"
+ )
+
+ # Make installer executable
+ st = tmpbuildtools.stat()
+ tmpbuildtools.chmod(st.st_mode | stat.S_IEXEC)
+
+ # Run installer (in an isolated environment)
+ installer_cmd = [str(tmpbuildtools), '-d', str(buildtools_dir), '-y']
+ env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
+ (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)
+ if ret != 0:
+ raise InitBuildEnvError("Could not run buildtools installer")
+
+
+def get_buildtools_version():
+ try:
+ version_file = list(get_buildtools_dir().glob("version-*"))
+ if len(version_file) != 1:
+ raise ValueError("Invalid number of version files")
+
+ with version_file[0].resolve().open('r') as f:
+ lines = f.readlines()
+ for line in lines:
+ if line.startswith("Distro Version"):
+ return lines[1].split(':', 1)[1].strip()
+ except Exception as e:
+ logging.warning(f"Unable to read buildtools version: {e}")
+
+ return -1
+
+
def get_build_environ(build_system):
"""
Creates the build environment variables.
@@ -290,6 +413,39 @@ def get_build_environ(build_system):
if not init_repo:
raise InitBuildEnvError('Did not find any init-build-env script')

+ conf_buildtools = get_context().config.get_buildtools()
+ buildtools_env = ""
+
+ if conf_buildtools:
+ # Create the dest. directory if it doesn't exist
+ buildtools_dir = get_buildtools_dir()
+ buildtools_dir.mkdir(parents=True, exist_ok=True)
+
+ if not any(buildtools_dir.iterdir()):
+ # Directory is empty, try to fetch from upstream
+ logging.info(f"Buildtools ({buildtools_dir}): directory is empty")
+ download_buildtools()
+ else:
+ # Fetch buildtools when versions differ in non-empty dir
+ found_version = get_buildtools_version()
+ if found_version != conf_buildtools['version']:
+ logging.warning("Buildtools: version mismatch")
+ logging.info(f"Required version: {conf_buildtools['version']}")
+ logging.info(f"Found version: {found_version}")
+ shutil.rmtree(os.path.realpath(buildtools_dir))
+ os.makedirs(os.path.realpath(buildtools_dir))
+ download_buildtools()
+
+ envfiles = list(get_buildtools_dir().glob("environment-setup-*"))
+ if len(envfiles) == 1:
+ buildtools_env = "source {}\n".format(envfiles[0].resolve())
+ else:
+ logging.error(
+ f"Expected 1 environment setup file, found {len(envfiles)}."
+ "Invalid or misconfigured buildtools package."
+ )
+ return -1
+
with tempfile.TemporaryDirectory() as temp_dir:
if logging.getLogger().getEffectiveLevel() == logging.DEBUG:
init_script_log = pathlib.Path(temp_dir) / '.init_script.log'
@@ -297,6 +453,7 @@ def get_build_environ(build_system):
init_script_log = '/dev/null'
script = f"""#!/bin/bash
set -e
+ {buildtools_env}
source {init_script} $1 > {init_script_log}
env
"""
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/command-line/environment-variables.inc | 4 ++
docs/userguide/project-configuration.rst | 70 +++++++++++++++++++++
2 files changed, 74 insertions(+)

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index b4ea240..1800348 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -207,6 +207,10 @@ overwritten using the ``env`` section of the config file.
| | not preserve the environment and cannot setup |
| | loopback devices. |
+--------------------------+--------------------------------------------------+
+| ``KAS_BUILDTOOLS_DIR`` | Explicitly set the path where kas will download |
+| (C,K) | and install buildtools. If not set, kas will use |
+| | ``KAS_BUILD_DIR/buildtools`` as the default path.|
++--------------------------+--------------------------------------------------+

.. |aws_cred| replace:: ``AWS_ROLE_ARN``
``AWS_SHARED_CREDENTIALS_FILE``
diff --git a/docs/userguide/project-configuration.rst b/docs/userguide/project-configuration.rst
index 9bdce1f..a7e17f5 100644
--- a/docs/userguide/project-configuration.rst
+++ b/docs/userguide/project-configuration.rst
@@ -503,6 +503,76 @@ Configuration reference
``kas-container`` script. It must not be set manually and might only be
defined in the top-level ``.config.yaml`` file.

+``buildtools``: dict [optional]
+ Provides variables to define which buildtools version should be fetched and
+ where it is (or will be) installed. Both ``version`` and ``sha256sum`` should
+ be set. The environment variable ``KAS_BUILDTOOLS_DIR`` can be used to set the
+ directory where buildtools will be installed, otherwise the default path
+ (i.e., ``KAS_BUILD_DIR/buildtools``) will be used. If such directory already
+ has buildtools installed, kas will check the ``Distro Version`` line in the
+ version file, and if it doesn't match with ``version``, the directory will
+ be cleaned and kas will download buildtools according to ``version``. After
+ the download, kas will perform integrity validation by calculating the
+ artifact's checksum and comparing it with ``sha256sum``. As for the optional
+ variables, they are meant to be used to support cases as: mirrors, changes in
+ the installer's file name, and fetching unofficial (i.e., custom) buildtools.
+ Finally, the environment-setup script will run before bitbake, so the whole
+ buildtools environment will be available. ``wget`` is the host tool required
+ for this feature. More information on how to install or generate buildtools
+ can be found at: |yp_doc_buildtools|
+
+ ``version``: string
+ :kasschemadesc:`buildtools.properties.version`
+
+ ``sha256sum``: string
+ :kasschemadesc:`buildtools.properties.sha256sum`
+
+ ``base_url``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+
+ ``filename``: string [optional]
+ :kasschemadesc:`buildtools.properties.filename`
+ It will be combined with to ``base_url`` to form the whole download URL, if
+ set. If not set, kas will combine the platform architecture and ``version``
+ to form the standard script filename:
+ ``{arch}-buildtools-extended-nativesdk-standalone-{version}.sh``
+
+ Example:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "5.0.5"
+
+ And for unofficial (custom) sources:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "1.0.0"
+ base_url: "https://downloads.mysources.com/yocto/buildtools/"
+ filename: "x86_64-buildtools-beta-testing-1.0.0.sh"
+
+.. |yp_doc_buildtools| replace:: https://docs.yoctoproject.org/dev/ref-manual/system-requirements.html#downloading-a-pre-built-buildtools-tarball
+
+Buildtools archive
+------------------
+
+kas expects the buildtools installer to be a shell script (i.e., as a standard
+Yocto SDK). Once executed, the resulting directory should contain the elements
+below:
+
+- ``sysroots``: the native and target sysroots, containing (among libraries and
+ headers) the build system's requirements: Git, tar, Python and make.
+- ``environment-setup-*``: the environment setup script, sourced by kas, to
+ setup variables such as ``PATH`` in such a way that it points to
+ the directories in ``sysroots``.
+- ``version-*``: the version file. Its second line contains a string as
+ ``Distro Version: X.Y.Z``, parsed to retrieve the version number.
+
+The archive can contain other files, such as ``buildinfo``, but they are not
+relevant for kas.
+
.. _example-configurations-label:

Example project configurations
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas-container | 2 ++
1 file changed, 2 insertions(+)

diff --git a/kas-container b/kas-container
index 041a455..884bb75 100755
--- a/kas-container
+++ b/kas-container
@@ -306,6 +306,7 @@ setup_kas_dirs()
KAS_REPO_REF_DIR="$(check_and_expand KAS_REPO_REF_DIR required)"
DL_DIR="$(check_and_expand DL_DIR createrec)"
SSTATE_DIR="$(check_and_expand SSTATE_DIR createrec)"
+ KAS_BUILDTOOLS_DIR="$(check_and_expand KAS_BUILDTOOLS_DIR createrec)"
}
setup_kas_dirs

@@ -620,6 +621,7 @@ forward_dir KAS_BUILD_DIR "/build" "rw"
forward_dir DL_DIR "/downloads" "rw"
forward_dir KAS_REPO_REF_DIR "/repo-ref" "rw"
forward_dir SSTATE_DIR "/sstate" "rw"
+forward_dir KAS_BUILDTOOLS_DIR "/buildtools" "rw"

if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
&& [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is a test case based on kas.yml, using Yocto 5.2 (Walnascar).

Please note that kas-buildtools.yml is merely a symbolic link to
examples/openembedded-buildtools.yml.

openembedded-buildtools.yml is based on examples/openembedded.yml, only
setting a different revision (i.e., Walnascar) for Poky repository.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

examples/openembedded-buildtools.yml | 46 ++++++++++++++++++++++++++++
image-tests/kas/kas-buildtools.yml | 1 +
2 files changed, 47 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

diff --git a/examples/openembedded-buildtools.yml b/examples/openembedded-buildtools.yml
new file mode 100644
index 0000000..9d09232
--- /dev/null
+++ b/examples/openembedded-buildtools.yml
@@ -0,0 +1,46 @@
+#
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Siemens AG, 2022-2025
+# Copyright (c) Bootlin, 2025
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+header:
+ version: 19
+ includes:
+ - examples/openembedded.yml
+
+repos:
+ poky:
+ url: https://git.yoctoproject.org/poky.git
+ # when specifying a tag, optionally provide a commit hash
+ tag: yocto-5.2
+ commit: 9b96fdbb0cab02f4a6180e812b02bc9d4c41b1a5
+ signed: true
+ allowed_signers:
+ - YoctoBuildandRelease
+ layers:
+ meta:
+ meta-poky:
+
+buildtools:
+ version: "5.2"
+ sha256sum: "6f69fba75c8f3142bb49558afa3ed5dd0723a3beda169b057a5238013623462d"
diff --git a/image-tests/kas/kas-buildtools.yml b/image-tests/kas/kas-buildtools.yml
new file mode 120000
index 0000000..7b17a91
--- /dev/null
+++ b/image-tests/kas/kas-buildtools.yml
@@ -0,0 +1 @@
+../../examples/openembedded-buildtools.yml
\ No newline at end of file
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

By default, KAS_BUILDTOOLS_DIR points to 'build/buildtools', which is
already covered by purge's default cleanup of the 'build/' directory.

If KAS_BUILDTOOLS_DIR is set to a location outside of 'build/', purge
needs to remove it explicitly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/plugins/clean.py | 9 +++++++++

1 file changed, 9 insertions(+)

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index 6eb39c6..af6a77a 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -225,6 +225,15 @@ class Purge(CleanAll):
if not args.dry_run:
self.clear_dir_content(build_dir)

+ if os.environ.get("KAS_BUILDTOOLS_DIR"):
+ buildtools_dir = (
+ Path(os.environ.get("KAS_BUILDTOOLS_DIR")).resolve()
+ )
+ if buildtools_dir.exists():
+ logging.info(f'Removing {buildtools_dir}')
+ if not args.dry_run:
+ shutil.rmtree(buildtools_dir)
+
work_dir = Path(ctx.kas_work_dir)
default_config = work_dir / CONFIG_YAML_FILE
if default_config.exists():
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Make sure that buildtools is tested, such as its purging.

Add some tracing to clear_dir_content() routine so we can clearly see
what directories are being cleaned.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

.github/workflows/next.yml | 3 +++
kas/plugins/clean.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 77c667e..b3761c6 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -88,6 +88,9 @@ jobs:
run: |
cd image-tests/${{ matrix.image-name }}
../../kas-container build kas.yml
+ if [ "${{ matrix.image-name }}" = "kas" ]; then
+ ../../kas-container build kas-buildtools.yml
+ fi
[ -d build/tmp ]
echo "Test kas clean"
../../kas-container clean kas.yml
diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index af6a77a..ea56b5e 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py
@@ -118,6 +118,7 @@ class Clean():
Clear the contents of a directory without removing the dir itself.
"""
for item in directory.iterdir():
+ logging.info(f'Removing {item}')
if item.is_dir():
shutil.rmtree(item)
else:
--
2.47.0

[João Marcos Costa]

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,
---

Resending:
- there is a minor change to handle/ignore missing pkg-config:
call "set -e" after {buildtools_env} in get_build_environ() (@kas/libkas.py)

Changes in v7:
- add support for purging in 'clean' plugin
- fix code style issues
- add kas-buildtools to next.yml

Joao Marcos Costa (8):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR
image-tests: add kas-buildtools.yml
plugins/clean: handle buildtools in purge
ci: Add kas-buildtools to next.yml

.github/workflows/next.yml | 3 +
docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 70 +++++++++
examples/openembedded-buildtools.yml | 46 ++++++
image-tests/kas/kas-buildtools.yml | 1 +
kas-container | 2 +
kas/config.py | 9 ++
kas/libkas.py | 157 ++++++++++++++++++++
kas/plugins/clean.py | 10 ++
kas/schema-kas.json | 26 ++++

10 files changed, 328 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

--
2.47.0

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Add buildtools configuration scheme:
- version: sets the buildtools version to be fetched
- sha256sum: buildtools installer checksum, to verify integrity
- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 9 +++++++++

1 file changed, 9 insertions(+)

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add the core of buildtools support: download the installer, check its
integrity, extract it, and set up the environment.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 157 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py

index 5327efb..c785178 100644

@@ -290,12 +413,46 @@ def get_build_environ(build_system):

else:

init_script_log = '/dev/null'
script = f"""#!/bin/bash

+ {buildtools_env}
set -e

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is a test case based on kas.yml, using Yocto 5.2 (Walnascar).

Please note that kas-buildtools.yml is merely a symbolic link to
examples/openembedded-buildtools.yml.

openembedded-buildtools.yml is based on examples/openembedded.yml, only
setting a different revision (i.e., Walnascar) for Poky repository.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

examples/openembedded-buildtools.yml | 46 ++++++++++++++++++++++++++++
image-tests/kas/kas-buildtools.yml | 1 +
2 files changed, 47 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

By default, KAS_BUILDTOOLS_DIR points to 'build/buildtools', which is
already covered by purge's default cleanup of the 'build/' directory.

If KAS_BUILDTOOLS_DIR is set to a location outside of 'build/', purge
needs to remove it explicitly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/plugins/clean.py | 9 +++++++++

1 file changed, 9 insertions(+)

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index 6eb39c6..af6a77a 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Make sure that buildtools is tested, such as its purging.

Add some tracing to clear_dir_content() routine so we can clearly see
what directories are being cleaned.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

.github/workflows/next.yml | 3 +++
kas/plugins/clean.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 77c667e..b3761c6 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -88,6 +88,9 @@ jobs:
run: |
cd image-tests/${{ matrix.image-name }}
../../kas-container build kas.yml
+ if [ "${{ matrix.image-name }}" = "kas" ]; then
+ ../../kas-container build kas-buildtools.yml
+ fi
[ -d build/tmp ]
echo "Test kas clean"
../../kas-container clean kas.yml

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index af6a77a..ea56b5e 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py

[Jan Kiszka]

On 09.09.25 11:00, João Marcos Costa wrote:
> Hello,
>
> This patch series allows Kas to fetch and install Buildtools, and subsequently
> run Bitbake with the Buildtools environment fully set-up. The use case addresses
> limitations in development environments, such as outdated distributions,
> significantly older Python versions, and, most importantly, the absence of a
> containerization solution.
>
> Best regards,
> ---
>
> Resending:
> - there is a minor change to handle/ignore missing pkg-config:
> call "set -e" after {buildtools_env} in get_build_environ() (@kas/libkas.py)
>

So, it's v8 - please avoid confusions.

Jan

Siemens AG, Foundational Technologies
Linux Expert Center

[Jan Kiszka]

On 09.09.25 11:00, João Marcos Costa wrote:

This needs a comment why we can (and have to?) swallow errors. Or we
need to limit this to known broken version of buildtools, moving it down
for modern ones.

Jan

> set -e
> source {init_script} $1 > {init_script_log}
> env


--

[Joao Marcos Costa]

Hello,

On 9/9/25 12:04, Jan Kiszka wrote:
> On 09.09.25 11:00, João Marcos Costa wrote:
>> Hello,
>>
>> This patch series allows Kas to fetch and install Buildtools, and subsequently
>> run Bitbake with the Buildtools environment fully set-up. The use case addresses
>> limitations in development environments, such as outdated distributions,
>> significantly older Python versions, and, most importantly, the absence of a
>> containerization solution.
>>
>> Best regards,
>> ---
>>
>> Resending:
>> - there is a minor change to handle/ignore missing pkg-config:
>> call "set -e" after {buildtools_env} in get_build_environ() (@kas/libkas.py)
>>
>
> So, it's v8 - please avoid confusions.

Sure. Let's call this version v7/resend, and will name it v8 for the
next series, to avoid a gap in the mail thread (e.g. v7, v7 resend, v9).

Best regards,
João Marcos Costa

[Joao Marcos Costa]

Hello,

AFAIK, the fix Daniel sent (currently in v2) is not even merged yet, so
technically all the buildtools version are "broken"? Otherwise, we could
do something as:

For version < 5.2 (just in case):
buildtools_env = "source {} || true \n".format(envfiles[0].resolve())
Else (keep it as it is):

buildtools_env = "source {}\n".format(envfiles[0].resolve())

And then we'd keep the same order as before:

script = f"""#!/bin/bash
set -e

{buildtools_env}

source {init_script} $1 > {init_script_log}
env
"""

What do you think?

> Jan
>
>> set -e
>> source {init_script} $1 > {init_script_log}
>> env
>
>

--

[Jan Kiszka]

If there is no official version without this undesirable behavior yet,
we should not speculate about which one might be fixed. Then just leave
a comment that explains the ordering or some "|| true".

Jan

[João Marcos Costa]

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,
---

Changes in v8:
- Add " || true" to ignore the error(s) from {buildtools_env} in kas/libkas.py,
and a comment explaining why we are ignoring it
- Rephrase the third commit’s title for clarity and grammar

Joao Marcos Costa (8):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties

kas: introduce buildtools support

docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR
image-tests: add kas-buildtools.yml
plugins/clean: handle buildtools in purge
ci: Add kas-buildtools to next.yml

.github/workflows/next.yml | 3 +
docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 70 +++++++++
examples/openembedded-buildtools.yml | 46 ++++++
image-tests/kas/kas-buildtools.yml | 1 +
kas-container | 2 +
kas/config.py | 9 ++

kas/libkas.py | 160 ++++++++++++++++++++

kas/plugins/clean.py | 10 ++
kas/schema-kas.json | 26 ++++

10 files changed, 331 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

--

2.47.0

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Add buildtools configuration scheme:
- version: sets the buildtools version to be fetched
- sha256sum: buildtools installer checksum, to verify integrity
- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 9 +++++++++

1 file changed, 9 insertions(+)

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add the core of buildtools support: download the installer, check its
integrity, extract it, and set up the environment.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 160 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 160 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 5327efb..ee587eb 100644

@@ -290,6 +413,42 @@ def get_build_environ(build_system):

+ envfiles = list(get_buildtools_dir().glob("environment-setup-*"))

+ if len(envfiles) == 1:

+ # Ignore missing pkg-config error until oe-core fix is merged
+ buildtools_env = (
+ "source {} || true\n".format(envfiles[0].resolve())
+ )

+ else:
+ logging.error(
+ f"Expected 1 environment setup file, found {len(envfiles)}."
+ "Invalid or misconfigured buildtools package."
+ )
+ return -1
+
with tempfile.TemporaryDirectory() as temp_dir:
if logging.getLogger().getEffectiveLevel() == logging.DEBUG:
init_script_log = pathlib.Path(temp_dir) / '.init_script.log'

@@ -297,6 +456,7 @@ def get_build_environ(build_system):

init_script_log = '/dev/null'
script = f"""#!/bin/bash

set -e
+ {buildtools_env}

source {init_script} $1 > {init_script_log}
env
"""

--
2.47.0

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is a test case based on kas.yml, using Yocto 5.2 (Walnascar).

Please note that kas-buildtools.yml is merely a symbolic link to
examples/openembedded-buildtools.yml.

openembedded-buildtools.yml is based on examples/openembedded.yml, only
setting a different revision (i.e., Walnascar) for Poky repository.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

examples/openembedded-buildtools.yml | 46 ++++++++++++++++++++++++++++
image-tests/kas/kas-buildtools.yml | 1 +
2 files changed, 47 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

By default, KAS_BUILDTOOLS_DIR points to 'build/buildtools', which is
already covered by purge's default cleanup of the 'build/' directory.

If KAS_BUILDTOOLS_DIR is set to a location outside of 'build/', purge
needs to remove it explicitly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/plugins/clean.py | 9 +++++++++

1 file changed, 9 insertions(+)

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index 6eb39c6..af6a77a 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py

[João Marcos Costa]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Make sure that buildtools is tested, such as its purging.

Add some tracing to clear_dir_content() routine so we can clearly see
what directories are being cleaned.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

.github/workflows/next.yml | 3 +++
kas/plugins/clean.py | 1 +
2 files changed, 4 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 77c667e..b3761c6 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -88,6 +88,9 @@ jobs:
run: |
cd image-tests/${{ matrix.image-name }}
../../kas-container build kas.yml
+ if [ "${{ matrix.image-name }}" = "kas" ]; then
+ ../../kas-container build kas-buildtools.yml
+ fi
[ -d build/tmp ]
echo "Test kas clean"
../../kas-container clean kas.yml

diff --git a/kas/plugins/clean.py b/kas/plugins/clean.py
index af6a77a..ea56b5e 100644
--- a/kas/plugins/clean.py
+++ b/kas/plugins/clean.py

[Jan Kiszka]

Applied - finally :)

Thanks a lot for your patience and commitment to polish this feature for
mainline!

[Joao Marcos Costa]

Hello, Jan

Sure, my pleasure. I always do my fair share of kas propaganda during my
training sessions so I'm glad to actually contribute to the project :)

[Jan Kiszka]

Ah, that's why we have so many users! ;) Appreciated!

[Jan Kiszka]

On 12.09.25 14:57, João Marcos Costa wrote:

This line is a debugging left-over, right?

$ kas-container cleansstate
2025-09-15 14:05:38 - INFO - kas 4.8.2 started
2025-09-15 14:05:38 - INFO - Removing /work/build/tmp
2025-09-15 14:06:02 - INFO - Removing /work/build/sstate-cache/*
2025-09-15 14:06:02 - INFO - Removing /work/build/sstate-cache/15
2025-09-15 14:06:02 - INFO - Removing /work/build/sstate-cache/53
2025-09-15 14:06:02 - INFO - Removing /work/build/sstate-cache/f4
[and further 200+ of these]

I would drop that again from this patch in next.

[Joao Marcos Costa]

Hello,

At first, it was meant for debugging but then I thought it could be
productive to leave it there, but I didn't realize this would raise
spurious logs specially with sstate-cache. Please feel free to remove it.