[PATCH v4 0/5] Add buildtools support

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,

---

Changes in v4:
- remove unreachable branches in the code, since version is mandatory
- add integrity validation with sha256
- simplified get_buildtools
- scan for 'Distro Version' in get_buildtools_version
- fail if multiple (or none) environment setup files are found
- add support for KAS_BUILDTOOLS_DIR in kas-container

Joao Marcos Costa (5):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR

docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 65 ++++++++++
kas-container | 1 +
kas/config.py | 9 ++
kas/libkas.py | 136 ++++++++++++++++++++
kas/schema-kas.json | 20 +++
6 files changed, 235 insertions(+)

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---
kas/schema-kas.json | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index e1bff6d..1c3122d 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -406,6 +406,26 @@
"_source_dir_host": {
"description": "Source directory of the config file on the host (auto-generated by kas menu plugin, when using kas-container).",
"type": "string"
+ },
+ "buildtools": {
+ "type": "object",
+ "required": [
+ "version"
+ ],
+ "properties": {
+ "version": {
+ "description": "Yocto Project version, as 5.0 (or even 5.0.8) for Scarthgap.",
+ "type": "string"
+ },
+ "base_url": {
+ "description": "Base URL to fetch downloads from, used instead of the default Yocto source (downloads.yoctoproject.org/releases/yocto).",
+ "type": "string"
+ },
+ "filename": {
+ "description": "Alternative name for the buildtools archive (.sh) to be downloaded.",
+ "type": "string"
+ }
+ }
}
}
}
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/kas/config.py b/kas/config.py
index d868a5c..af1c1ed 100644
--- a/kas/config.py
+++ b/kas/config.py
@@ -264,3 +264,12 @@ class Config:
else:
return {k: v for k, v in signers.items()
if v.get('type', 'gpg') == keytype}
+
+ def get_buildtools(self):
+ """
+ Returns the buildtools keys: version, download URL and
+ archive filename. These are provided so kas knows which
+ buildtools archive to fetch and from what source.
+ """
+
+ return self._config.get('buildtools', {})
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Introduce buildtools configuration scheme:

- version: sets the buildtools version to be fetched
- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 136 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 136 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 89113fa..8207035 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -31,9 +31,15 @@ import logging
import tempfile
import asyncio
import errno
+import glob
+import hashlib
import pathlib
+import platform
+import shutil
import signal
+import stat
from subprocess import Popen, PIPE, run as subprocess_run
+from urllib.parse import quote
from .context import get_context
from .kasusererror import KasUserError, CommandExecError

@@ -258,6 +264,102 @@ def repos_apply_patches(repos):
raise TaskExecError('apply patches', e.ret_code)


+def get_buildtools_path():
+ # Set the dest. directory for buildtools's setup
+ env = os.environ.get("KAS_BUILDTOOLS_DIR")
+ if env:
+ buildtools_dir = os.path.realpath(env)
+ else:
+ buildtools_dir = get_context().build_dir + "/buildtools"
+
+ return buildtools_dir
+
+
+def check_sha256sum(installer_path, sha256sum_url):
+ hash_sha256 = hashlib.sha256()
+ with open(installer_path, "rb") as f:
+ for chunk in iter(lambda: f.read(8192), b""):
+ hash_sha256.update(chunk)
+
+ calc = hash_sha256.hexdigest()
+
+ # Download sha256sum file
+ sha256sum_path = installer_path + ".sha256sum"
+ fetch_cmd = ['wget', '-q', '-O', sha256sum_path, sha256sum_url]
+ (ret, _) = run_cmd(fetch_cmd, cwd=get_context().kas_work_dir)
+ if ret != 0:
+ raise InitBuildEnvError("Could not download buildtools sha256sum file")
+
+ with open(sha256sum_path, "r") as f:
+ lines = f.readlines()
+
+ reference = lines[0].split()[0]
+
+ return calc == reference
+
+
+def download_buildtools():
+ arch = platform.machine()
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ buildtools_dir = get_buildtools_path()
+
+ if 'base_url' in conf_buildtools:
+ base_url = conf_buildtools['base_url']
+ else:
+ yocto_releases = "https://downloads.yoctoproject.org/releases/yocto"
+ base_url = f"{yocto_releases}/yocto-{version}/buildtools/"
+
+ if 'filename' in conf_buildtools:
+ filename = conf_buildtools['filename']
+ else:
+ filename = f"{arch}-buildtools-extended-nativesdk-standalone-{version}.sh"
+
+ # Enable extended buildtools tarball
+ buildtools_url = f"{base_url}/{quote(filename)}"
+ tmpbuildtools = os.path.join(buildtools_dir, filename)
+
+ logging.info(f"Downloading Buildtools {version}")
+ # Download installer
+ fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]
+ (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
+ if ret != 0:
+ raise InitBuildEnvError("Could not download buildtools installer")
+
+ # Check if the installer's sha256sum matches
+ if not check_sha256sum(tmpbuildtools, buildtools_url + ".sha256sum"):
+ raise InitBuildEnvError("sha256sum mismatch: installer may be corrupted")
+
+ # Make installer executable
+ st = os.stat(tmpbuildtools)
+ os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
+
+ # Run installer
+ installer_cmd = [tmpbuildtools, '-d', buildtools_dir, '-y']
+ env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
+ (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)
+ if ret != 0:
+ raise InitBuildEnvError("Could not run buildtools installer")
+
+
+def get_buildtools_version():
+ try:
+ version_file = glob.glob(os.path.join(get_buildtools_path(), "version-*"))
+ if len(version_file) != 1:
+ raise ValueError("Invalid number of version files")
+
+ with open(os.path.realpath(version_file[0]), 'r') as f:
+ lines = f.readlines()
+
+ if len(lines) > 1 and "Distro Version" in lines[1]:
+ return lines[1].split(':', 1)[1].strip()
+ except Exception as e:
+ logging.warning(f"Unable to read buildtools version: {e}")
+
+ return -1
+
+
def get_build_environ(build_system):
"""
Creates the build environment variables.
@@ -288,9 +390,43 @@ def get_build_environ(build_system):
if not init_repo:
raise InitBuildEnvError('Did not find any init-build-env script')

+ conf_buildtools = get_context().config.get_buildtools()
+ buildtools_env = ""
+
+ if conf_buildtools:
+ # Create the dest. directory if it doesn't exist
+ buildtools_dir = get_buildtools_path()
+ pathlib.Path(buildtools_dir).mkdir(parents=True, exist_ok=True)
+
+ if not os.listdir(buildtools_dir):
+ # Directory is empty, try to fetch from upstream
+ logging.info(f"Buildtools ({buildtools_dir}): directory is empty")
+ download_buildtools()
+ else:
+ # Directory is not empty: fetch buildtools if the versions don't match
+ found_version = get_buildtools_version()
+ if found_version != conf_buildtools['version']:
+ logging.warning("Buildtools: version mismatch")
+ logging.info(f"Required version: {conf_buildtools['version']}")
+ logging.info(f"Found version: {found_version}")
+ shutil.rmtree(os.path.realpath(buildtools_dir))
+ os.makedirs(os.path.realpath(buildtools_dir))
+ download_buildtools()
+
+ envfiles = glob.glob(os.path.join(buildtools_dir, "environment-setup-*"))
+ if len(envfiles) == 1:
+ buildtools_env = "source {}\n".format(os.path.realpath(envfiles[0]))
+ else:
+ logging.error(
+ f"Expected 1 environment setup file, but found {len(envfiles)}. "
+ "Invalid or misconfigured buildtools package."
+ )
+ return -1
+
with tempfile.TemporaryDirectory() as temp_dir:
script = f"""#!/bin/bash
set -e
+ {buildtools_env}
source {init_script} $1 > /dev/null
env
"""
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas-container | 1 +
1 file changed, 1 insertion(+)

diff --git a/kas-container b/kas-container
index 0e99898..046c5be 100755
--- a/kas-container
+++ b/kas-container
@@ -283,6 +283,7 @@ setup_kas_dirs()
KAS_REPO_REF_DIR="$(check_and_expand KAS_REPO_REF_DIR required)"
DL_DIR="$(check_and_expand DL_DIR createrec)"
SSTATE_DIR="$(check_and_expand SSTATE_DIR createrec)"
+ KAS_BUILDTOOLS_DIR="$(check_and_expand KAS_BUILDTOOLS_DIR createrec)"
}
setup_kas_dirs

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/command-line/environment-variables.inc | 4 ++
docs/userguide/project-configuration.rst | 65 +++++++++++++++++++++
2 files changed, 69 insertions(+)

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index 875f4b3..5e69544 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -200,6 +200,10 @@ overwritten using the ``env`` section of the config file.
| (C) | ``docker`` or ``podman``). If not set, this is |
| | auto-detected (preference: docker). |
+--------------------------+--------------------------------------------------+
+| ``KAS_BUILDTOOLS_DIR`` | Explicitly set the path where kas will download |
+| (K) | and install buildtools. If not set, kas will use |
+| | ``KAS_BUILD_DIR/buildtools`` as the default path.|
++--------------------------+--------------------------------------------------+

.. |aws_cred| replace:: ``AWS_ROLE_ARN``
``AWS_SHARED_CREDENTIALS_FILE``
diff --git a/docs/userguide/project-configuration.rst b/docs/userguide/project-configuration.rst
index df9fe58..b97719d 100644
--- a/docs/userguide/project-configuration.rst
+++ b/docs/userguide/project-configuration.rst
@@ -503,6 +503,71 @@ Configuration reference
``kas-container`` script. It must not be set manually and might only be
defined in the top-level ``.config.yaml`` file.

+``buildtools``: dict [optional]
+ Provides variables to define which buildtools version should be fetched and
+ where it is (or will be) installed. At least ``version`` should be set. The
+ environment variable ``KAS_BUILDTOOLS_DIR`` can be used to set the directory
+ where buildtools will be installed, otherwise the default path (i.e.,
+ ``KAS_BUILD_DIR/buildtools``) will be used. If such directory already has
+ buildtools installed, kas will check the ``Distro Version`` line in the
+ version file, and if it doesn't match with ``version``, the directory will
+ be cleaned and kas will download buildtools according to ``version``. As for
+ the optional variables, they are meant to be used to support cases as:
+ mirrors, changes in the installer's file name, and fetching unofficial (i.e.,
+ custom) buildtools. Finally, the environment-setup script will run before
+ bitbake, so the whole buildtools environment will be available. ``wget`` is
+ the host tool required for this feature. More information on how to install
+ or generate buildtools can be found at: |yp_doc_buildtools|
+
+ ``version``: string
+ :kasschemadesc:`buildtools.properties.version`
+
+ ``base_url``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+
+ ``filename``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+ It will be combined with to ``base_url`` to form the whole URL to be passed
+ to ``wget``, if set. If not set, kas will combine the platform architecture
+ and ``version`` to form the standard script filename:
+ ``{arch}-buildtools-extended-nativesdk-standalone-{version}.sh``
+
+ Example:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "5.0.5"
+
+ And for unofficial (custom) sources:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "1.0.0"
+ base_url: "https://downloads.mysources.com/yocto/buildtools/"
+ filename: "x86_64-buildtools-beta-testing-1.0.0.sh"
+
+.. |yp_doc_buildtools| replace:: https://docs.yoctoproject.org/dev/ref-manual/system-requirements.html#downloading-a-pre-built-buildtools-tarball
+
+Buildtools archive
+------------------
+
+kas expects the buildtools installer to be a shell script (i.e., as a standard
+Yocto SDK). Once executed, the resulting directory should contain the elements
+below:
+
+- ``sysroots``: the native and target sysroots, containing (among libraries and
+ headers) the build system's requirements: Git, tar, Python and make.
+- ``environment-setup-*``: the environment setup script, sourced by kas, to
+ setup variables such as ``PATH`` in such a way that it points to
+ the directories in ``sysroots``.
+- ``version-*``: the version file. Its second line contains a string as
+ ``Distro Version: X.Y.Z``, parsed to retrieve the version number.
+
+The archive can contain other files, such as ``buildinfo``, but they are not
+relevant for kas.
+
.. _example-configurations-label:

Example project configurations
--
2.47.0

[Jan Kiszka]

Sorry, but this adds no security: If upstream changes both the payload
and sha file, we will not detect that. That's why bitbake manages the
checksums in recipes.

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[Joao Marcos Costa]

Hello Jan,

Thanks for the reply. Indeed, at first I only considered checking the
integrity of the installer, nothing else. My take then is to add another
entry in the scheme, like 'sha256sum'. It would be mandatory, such as
'version'. Does it look coherent to you?

--
Best regards,
João Marcos Costa

[Jan Kiszka]

Yes, this is what came to my mind as well.

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,

---

Changes in v5:
- add another (mandatory) entry in the scheme to provide the checksum for
integrity validation, instead of retrieving this checksum from a downloaded
.sha256sum file
- update the docs accordingly
- add some helper functions to simplify download_buildtools(), namely
get_buildtools_url(), where the KAS_PREMIRRORS could be handled in a future
separate patch series

Joao Marcos Costa (5):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR

docs/command-line/environment-variables.inc | 4 +

docs/userguide/project-configuration.rst | 70 ++++++++++

kas-container | 1 +
kas/config.py | 9 ++

kas/libkas.py | 147 ++++++++++++++++++++
kas/schema-kas.json | 25 ++++
6 files changed, 256 insertions(+)

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Add buildtools configuration scheme:

- version: sets the buildtools version to be fetched

- sha256sum: buildtools installer checksum, to verify integrity

- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/schema-kas.json | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index e1bff6d..658a74b 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -406,6 +406,31 @@

"_source_dir_host": {
"description": "Source directory of the config file on the host (auto-generated by kas menu plugin, when using kas-container).",
"type": "string"
+ },
+ "buildtools": {
+ "type": "object",
+ "required": [

+ "version",
+ "sha256sum"

+ ],
+ "properties": {
+ "version": {
+ "description": "Yocto Project version, as 5.0 (or even 5.0.8) for Scarthgap.",
+ "type": "string"
+ },

+ "sha256sum": {
+ "description": "The installer's checksum, to perform integrity validation of the fetched artifact.",

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add the core of buildtools support: download the installer, check its
integrity, extract it, and set up the environment.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 147 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 147 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 89113fa..0f691b0 100644

--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -31,9 +31,15 @@ import logging
import tempfile
import asyncio
import errno
+import glob
+import hashlib
import pathlib
+import platform
+import shutil
import signal
+import stat
from subprocess import Popen, PIPE, run as subprocess_run
+from urllib.parse import quote
from .context import get_context
from .kasusererror import KasUserError, CommandExecError

@@ -258,6 +264,113 @@ def repos_apply_patches(repos):

raise TaskExecError('apply patches', e.ret_code)

+def get_buildtools_dir():

+ # Set the dest. directory for buildtools's setup
+ env = os.environ.get("KAS_BUILDTOOLS_DIR")
+ if env:
+ buildtools_dir = os.path.realpath(env)
+ else:
+ buildtools_dir = get_context().build_dir + "/buildtools"
+
+ return buildtools_dir
+

+def get_buildtools_filename():

+ arch = platform.machine()
+ ctx = get_context()
+
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']

+ if 'filename' in conf_buildtools:
+ filename = conf_buildtools['filename']
+ else:

+ filename = f"{arch}-buildtools-extended-nativesdk-standalone-{version}.sh"
+
+ return filename
+
+
+def get_buildtools_path():
+ return os.path.join(get_buildtools_dir(), get_buildtools_filename())
+
+
+def get_buildtools_url():

+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()

+ filename = get_buildtools_filename()

+ version = conf_buildtools['version']
+

+ if 'base_url' in conf_buildtools:
+ base_url = conf_buildtools['base_url']
+ else:
+ yocto_releases = "https://downloads.yoctoproject.org/releases/yocto"
+ base_url = f"{yocto_releases}/yocto-{version}/buildtools/"
+

+ return f"{base_url}/{quote(filename)}"
+
+
+def check_sha256sum(filename):
+ hash_sha256 = hashlib.sha256()
+ with open(filename, "rb") as f:

+ for chunk in iter(lambda: f.read(8192), b""):
+ hash_sha256.update(chunk)
+
+ calc = hash_sha256.hexdigest()
+

+ # Get user-defined sha256sum
+ conf_sha256sum = get_context().config.get_buildtools()['sha256sum']
+
+ return (conf_sha256sum == calc)
+
+
+def download_buildtools():

+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']

+ buildtools_dir = get_buildtools_dir()

+
+ # Enable extended buildtools tarball

+ buildtools_url = get_buildtools_url()
+ tmpbuildtools = get_buildtools_path()

+
+ logging.info(f"Downloading Buildtools {version}")
+ # Download installer

+ fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]
+ (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)

+ if ret != 0:

+ raise InitBuildEnvError("Could not download buildtools installer")
+
+ # Check if the installer's sha256sum matches

+ if not check_sha256sum(tmpbuildtools):

+ raise InitBuildEnvError("sha256sum mismatch: installer may be corrupted")
+
+ # Make installer executable
+ st = os.stat(tmpbuildtools)
+ os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
+
+ # Run installer
+ installer_cmd = [tmpbuildtools, '-d', buildtools_dir, '-y']
+ env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
+ (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)

+ if ret != 0:

+ raise InitBuildEnvError("Could not run buildtools installer")
+
+
+def get_buildtools_version():
+ try:

+ version_file = glob.glob(os.path.join(get_buildtools_dir(), "version-*"))

+ if len(version_file) != 1:
+ raise ValueError("Invalid number of version files")
+

+ with open(os.path.realpath(version_file[0]), 'r') as f:

+ lines = f.readlines()
+

+ if len(lines) > 1 and "Distro Version" in lines[1]:
+ return lines[1].split(':', 1)[1].strip()
+ except Exception as e:
+ logging.warning(f"Unable to read buildtools version: {e}")
+
+ return -1
+
+
def get_build_environ(build_system):
"""
Creates the build environment variables.

@@ -288,9 +401,43 @@ def get_build_environ(build_system):

if not init_repo:
raise InitBuildEnvError('Did not find any init-build-env script')

+ conf_buildtools = get_context().config.get_buildtools()
+ buildtools_env = ""
+
+ if conf_buildtools:
+ # Create the dest. directory if it doesn't exist

+ buildtools_dir = get_buildtools_dir()

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/command-line/environment-variables.inc | 4 ++
docs/userguide/project-configuration.rst | 70 +++++++++++++++++++++
2 files changed, 74 insertions(+)

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index 875f4b3..5e69544 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -200,6 +200,10 @@ overwritten using the ``env`` section of the config file.
| (C) | ``docker`` or ``podman``). If not set, this is |
| | auto-detected (preference: docker). |
+--------------------------+--------------------------------------------------+
+| ``KAS_BUILDTOOLS_DIR`` | Explicitly set the path where kas will download |
+| (K) | and install buildtools. If not set, kas will use |
+| | ``KAS_BUILD_DIR/buildtools`` as the default path.|
++--------------------------+--------------------------------------------------+

.. |aws_cred| replace:: ``AWS_ROLE_ARN``
``AWS_SHARED_CREDENTIALS_FILE``
diff --git a/docs/userguide/project-configuration.rst b/docs/userguide/project-configuration.rst

index df9fe58..de323c8 100644
--- a/docs/userguide/project-configuration.rst
+++ b/docs/userguide/project-configuration.rst
@@ -503,6 +503,76 @@ Configuration reference

``kas-container`` script. It must not be set manually and might only be
defined in the top-level ``.config.yaml`` file.

+``buildtools``: dict [optional]
+ Provides variables to define which buildtools version should be fetched and

+ where it is (or will be) installed. Both ``version`` and ``sha256sum`` should
+ be set. The environment variable ``KAS_BUILDTOOLS_DIR`` can be used to set the
+ directory where buildtools will be installed, otherwise the default path
+ (i.e., ``KAS_BUILD_DIR/buildtools``) will be used. If such directory already
+ has buildtools installed, kas will check the ``Distro Version`` line in the

+ version file, and if it doesn't match with ``version``, the directory will

+ be cleaned and kas will download buildtools according to ``version``. After
+ the download, kas will perform integrity validation by calculating the
+ artifact's checksum and comparing it with ``sha256sum``. As for the optional
+ variables, they are meant to be used to support cases as: mirrors, changes in
+ the installer's file name, and fetching unofficial (i.e., custom) buildtools.
+ Finally, the environment-setup script will run before bitbake, so the whole
+ buildtools environment will be available. ``wget`` is the host tool required
+ for this feature. More information on how to install or generate buildtools
+ can be found at: |yp_doc_buildtools|

+
+ ``version``: string
+ :kasschemadesc:`buildtools.properties.version`
+

+ ``sha256sum``: string
+ :kasschemadesc:`buildtools.properties.sha256sum`

+
+ ``base_url``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+
+ ``filename``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+ It will be combined with to ``base_url`` to form the whole URL to be passed
+ to ``wget``, if set. If not set, kas will combine the platform architecture
+ and ``version`` to form the standard script filename:

+ ``{arch}-buildtools-extended-nativesdk-standalone-{version}.sh``
+

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

[Jan Kiszka]

Why do we need to (re?)set PATH this way here? And why is it OK to not
set all the other typical env vars? We need at least some explaining
comment.

> + (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)
> + if ret != 0:
> + raise InitBuildEnvError("Could not run buildtools installer")
> +
> +
> +def get_buildtools_version():
> + try:
> + version_file = glob.glob(os.path.join(get_buildtools_dir(), "version-*"))
> + if len(version_file) != 1:
> + raise ValueError("Invalid number of version files")
> +
> + with open(os.path.realpath(version_file[0]), 'r') as f:
> + lines = f.readlines()
> +
> + if len(lines) > 1 and "Distro Version" in lines[1]:
> + return lines[1].split(':', 1)[1].strip()

This is hard-coding to look for "Distro Version" only in line 2. Why not
parsing each line as it is read in the loop above and even stop reading
the file once you found the match (startswith())?

[Jan Kiszka]

On 27.06.25 15:23, joaomarc...@bootlin.com wrote:

That alone will not be sufficient. You also need to forward that dir
into the container (forward_dir).

And please update the documentation that KAS_BUILDTOOLS_DIR is now also
supported by the container (C marker).

[Jan Kiszka]

On 27.06.25 15:23, joaomarc...@bootlin.com wrote:

Looks we are almost there. But it would still be good to have a test
case as well. Can't we extends the existing container test for yocto
with buildtools (-> image-tests/kas/kas.yml)?

[MOESSBAUER, Felix]

Hi Joao Marcos Costa,

I'll also have a closer look this week and report back.
In general, it already looks quite promising.

Best regards,
Felix

>
> Jan
>
> --
> Siemens AG, Foundational Technologies
> Linux Expert Center

--
Siemens AG
Linux Expert Center
Friedrich-Ludwig-Bauer-Str. 3
85748 Garching, Germany

[MOESSBAUER, Felix]

On Fri, 2025-06-27 at 15:23 +0200, joaomarcos.costa via kas-devel
wrote:

Hi,

instead of mentioning the default in the description, please directly
use the "default" property [1]. You can then also use the default value
in the python code [2].

Please also add the default via a reference to the project-config docs
(in the docs commit).

[1]
https://github.com/siemens/kas/blob/32f59530e210fd34e56852da56d26e51431b9ad4/kas/schema-kas.json#L148

[2] https://github.com/siemens/kas/blob/master/kas/config.py#L169

Best regards,
Felix

> +                    "type": "string"
> +                },
> +                "filename": {
> +                    "description": "Alternative name for the
> buildtools archive (.sh) to be downloaded.",
> +                    "type": "string"
> +                }
> +            }
>          }
>      }
>  }
> --
> 2.47.0

--

[MOESSBAUER, Felix]

On Fri, 2025-06-27 at 15:23 +0200, joaomarcos.costa via kas-devel
wrote:

Why return an empty dict? IIRC, either the buildtools dict is fully
specified, enforced via the schema (i.e. all keys are populated), or no
buildtools are set. In this case, we better return None.

Felix

[Joao Marcos Costa]

Hello Felix and Jan,

On 7/17/25 14:18, MOESSBAUER, Felix wrote:
> On Tue, 2025-07-01 at 00:01 +0200, 'Jan Kiszka' via kas-devel wrote:
>> On 27.06.25 15:23, joaomarc...@bootlin.com wrote:
>>> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>>>

(...)

>>
>> Looks we are almost there. But it would still be good to have a test
>> case as well. Can't we extends the existing container test for yocto
>> with buildtools (-> image-tests/kas/kas.yml)?
>
> Hi Joao Marcos Costa,
>
> I'll also have a closer look this week and report back.
> In general, it already looks quite promising.
>
> Best regards,
> Felix
>
>>
>> Jan
>>
>> --
>> Siemens AG, Foundational Technologies
>> Linux Expert Center
>

I wasn't able to reply the latest review as the last couple weeks were a
bit busy, but I already started working on the V6. @Felix, should I wait
for more comments on your side?

Thanks!

[MOESSBAUER, Felix]

On Fri, 2025-06-27 at 15:23 +0200, joaomarcos.costa via kas-devel
wrote:

We slowly try to switch over to pathlib (and already have it imported).
This also helps to avoid string concatenations of paths. Further,
please give the variables are more meaningful name (e.g. rawpath
instead of env).

The code above can probably just be replaced by:

Path(rawpath).resolve()

> +    else:
> +        buildtools_dir = get_context().build_dir + "/buildtools"

and here you then can use buildtools_dir / 'buildtools'.

Can we read this from the to-be-introduced default property in the
schema?

> +
> +    return f"{base_url}/{quote(filename)}"
> +
> +
> +def check_sha256sum(filename):

Please make the expected checksum a parameter of this function, then we
don't need to get it from the context->config and avoid this global
access. It also helps unit testing.

> +    hash_sha256 = hashlib.sha256()
> +    with open(filename, "rb") as f:
> +        for chunk in iter(lambda: f.read(8192), b""):
> +            hash_sha256.update(chunk)
> +
> +    calc = hash_sha256.hexdigest()
> +
> +    # Get user-defined sha256sum
> +    conf_sha256sum =
> get_context().config.get_buildtools()['sha256sum']
> +
> +    return (conf_sha256sum == calc)

Instead of manually comparing, please consider using:
hmac.compare_digest(good, actual). Example [1].

[1] https://docs.python.org/3/library/hashlib.html

> +
> +
> +def download_buildtools():
> +    ctx = get_context()
> +    conf_buildtools = ctx.config.get_buildtools()
> +    version = conf_buildtools['version']
> +    buildtools_dir = get_buildtools_dir()
> +
> +    # Enable extended buildtools tarball
> +    buildtools_url = get_buildtools_url()
> +    tmpbuildtools = get_buildtools_path()
> +
> +    logging.info(f"Downloading Buildtools {version}")
> +    # Download installer
> +    fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]

Is there a particular reason why we use wget instead of the python
requests library?

> +    (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
> +    if ret != 0:
> +        raise InitBuildEnvError("Could not download buildtools
> installer")
> +
> +    # Check if the installer's sha256sum matches
> +    if not check_sha256sum(tmpbuildtools):
> +        raise InitBuildEnvError("sha256sum mismatch: installer may
> be corrupted")
> +
> +    # Make installer executable
> +    st = os.stat(tmpbuildtools)
> +    os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
> +
> +    # Run installer
> +    installer_cmd = [tmpbuildtools, '-d', buildtools_dir, '-y']
> +    env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
> +    (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)

By that, the installer is executed in a completely isolated
environment. If the installer needs internet access (I hope no
installer does this!), this would fail if a proxy is needed.

Is that by-design? If so, please add a comment.

This probably can be simplified if we already return a pathlib.Path
from get_buildtools_dir.

Felix

[MOESSBAUER, Felix]

On Fri, 2025-06-27 at 15:23 +0200, joaomarcos.costa via kas-devel
wrote:

Hi,

please add the default as well (similar to how other defaults are
documented).

> +
> +  ``filename``: string [optional]
> +  :kasschemadesc:`buildtools.properties.base_url`
> +  It will be combined with to ``base_url`` to form the whole URL to
> be passed
> +  to ``wget``, if set. If not set, kas will combine the platform

Is there a particular reason why wget must be used for downloading?
Otherwise I prefer to not be specific here to later allow changing the
underlying implementation (e.g. use curl or python requests).

Felix

[MOESSBAUER, Felix]

Hi, many thanks for not giving up. I'm sure this is a valuable feature
for our Yocto users and I would like to see it in kas :)

>
> @Felix, should I wait
> for more comments on your side?

I'm done with my comments. Please go ahead with your implementation.

BTW: Maybe I missed it, but do you haven an OSS example of a layer that
needs the buildtools? Then I would give it a try as well.

It would also be great to either extend the existing image test (as
pointed out by Jan), or add a new one for this feature.

Cheers!
Felix

>
> Thanks!

[Joao Marcos Costa]

Hello Jan,

On 6/30/25 23:57, Jan Kiszka wrote:

(...)

>> +
>> + # Run installer
>> + installer_cmd = [tmpbuildtools, '-d', buildtools_dir, '-y']
>> + env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
>
> Why do we need to (re?)set PATH this way here? And why is it OK to not
> set all the other typical env vars? We need at least some explaining
> comment.
>

If I don't set the PATH as such, I get the following error when run_cmd
runs the installer script: "Error: The SDK needs a python installed"

As per run_cmd's definition, if 'env' is not defined, it will be set to
get_context().environ. If I print such variable, I get the following value:

{'LC_ALL': 'en_US.utf8', 'LANG': 'en_US.utf8', 'LANGUAGE': 'en_US',
'SSH_AUTH_SOCK': '/run/user/1000/keyring/ssh', 'HOME': '/tmp/tmpq6mvmj28'}

So it seems PATH is not being set at all, unless I'm missing something.
I'd expect to get my whole set of environment variables, of course, but
this is not the case. Also, PATH seems to be enough to run the installer
correctly.

[MOESSBAUER, Felix]

On Thu, 2025-07-17 at 16:23 +0200, 'Joao Marcos Costa' via kas-devel
wrote:

I'll have a look into this. We need to check when exactly the script is
executed and which kas setup parts have been run before.

I just added the following to the examples/openembedded.yml to test the
buildtools support:

buildtools:
version: 5.1.1
sha256sum:
dcbf03e4c87e33b41a1745415f1d6d024cde8c57825b384c3447e38a0c5d0936

Best regards,
Felix

>
> --
> Best regards,
> João Marcos Costa

[Joao Marcos Costa]

Hello,

On 7/17/25 16:49, MOESSBAUER, Felix wrote:
(...)

>
> I'll have a look into this. We need to check when exactly the script is
> executed and which kas setup parts have been run before.
>
> I just added the following to the examples/openembedded.yml to test the
> buildtools support:
>
> buildtools:
> version: 5.1.1
> sha256sum:
> dcbf03e4c87e33b41a1745415f1d6d024cde8c57825b384c3447e38a0c5d0936

I'm testing the same way, with a fairly simple yml. You asked in another
email about a layer that needs the buildtools support, but the actual
context is a set of private (customer's) layers. This YP environment is
used in a server where we can't really update python/git/tar/wget/etc,
hence the buildtools support.

[MOESSBAUER, Felix]

On Thu, 2025-07-17 at 16:23 +0200, 'Joao Marcos Costa' via kas-devel
wrote:

As of now, we don't set PATH in the environment we use to call most of
our commands. The get_bb_env in get_build_environ is called with a
specific path of "/usr/sbin:/usr/bin:/sbin:/bin" and later bitbake
itself is called with the environment that was setup in get_bb_env.

By that, the path used to call tmpbuildtools should probably also be
"/usr/sbin:/usr/bin:/sbin:/bin" to keep things consistent.

However!
I don't in particular like the idea of running relative commands via
subprocess.run without a PATH variable, as this relies on subprocess
internals and in the end on the behavior of execvpe. That again
resolves the path to whatever confstr(_CS_PATH) returns, which usually
is /bin:/usr/bin.

Independent of this series, I propose to call all kas commands with a
fixed path of "/usr/sbin:/usr/bin:/sbin:/bin", similar to how we pin
LC_ALL and alike variables in Context.setup_initial_environ. I'm also
considering to inherit the PATH variable from the calling environment
(only in kas, not in kas-container).

Are there any opinions on this topic? @Jan

Best regards,
Felix

>
> --
> Best regards,
> João Marcos Costa

[Joao Marcos Costa]

Hello, Felix

On 7/17/25 17:43, MOESSBAUER, Felix wrote:
(...)

> Independent of this series, I propose to call all kas commands with a
> fixed path of "/usr/sbin:/usr/bin:/sbin:/bin", similar to how we pin
> LC_ALL and alike variables in Context.setup_initial_environ. I'm also
> considering to inherit the PATH variable from the calling environment
> (only in kas, not in kas-container).
>
> Are there any opinions on this topic? @Jan
>
> Best regards,
> Felix
>

Looks out of the scope of this series, then. Unless you have greater
concerns, I will leave that piece of code as it is.

[MOESSBAUER, Felix]

That's fine, just leave it as it is. We will sort this out in a cleanup
series.

Felix

[Joao Marcos Costa]

Hello Felix,

On 7/17/25 16:16, MOESSBAUER, Felix wrote:
(...)

>> +    logging.info(f"Downloading Buildtools {version}")
>> +    # Download installer
>> +    fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]
>
> Is there a particular reason why we use wget instead of the python
> requests library?

I never used requests, but simply calling wget looks cleaner to me than
writing more code to stream the file's (~70Mb) content through chunks. I
would prefer to leave it as it is for now, keeping this download helper
simple, and maybe in the future to make the downloading tool (so wget,
python requests, curl and whatnot) configurable, if someone actually
needs such level of customization.

>> +    (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)
>
> By that, the installer is executed in a completely isolated
> environment. If the installer needs internet access (I hope no
> installer does this!), this would fail if a proxy is needed.
>
> Is that by-design? If so, please add a comment.
>

It doesn't need internet access, and I don't feel like this will ever
change, since the installer is fairly straightforward: it just extracts
the compressed blob shipped in the installer's file itself, hence it's
completely fine IMHO to run it in an isolated environment.

Thanks!

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,

---

Changes in v6:
- rework some functions to use pathlib
- pass the expected installer's checksum as a parameter to check_sha256sum()
- add a default value for base_url configuration entry
- fix the hard-coded way to look for "Distro Version"
- add missing call to forward_dir in kas-container
- fix some formatting issues with the previous version of the docs
- add a test image (i.e., kas-buildtools.yml)
- address some other minor points from last review

Joao Marcos Costa (7):

schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry
kas-container: add support for KAS_BUILDTOOLS_DIR

sphinx_kas_schema.py: support numbers in regex
image-tests: add kas-buildtools.yml

docs/_ext/sphinx_kas_schema.py | 2 +-

docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 70 ++++++++++

examples/openembedded-buildtools.yml | 45 ++++++
image-tests/kas/kas-buildtools.yml | 1 +
kas-container | 2 +
kas/config.py | 9 ++
kas/libkas.py | 145 ++++++++++++++++++++
kas/schema-kas.json | 26 ++++
9 files changed, 303 insertions(+), 1 deletion(-)
create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Add buildtools configuration scheme:
- version: sets the buildtools version to be fetched
- sha256sum: buildtools installer checksum, to verify integrity

- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org

- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/schema-kas.json | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index e1bff6d..db28b3d 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -406,6 +406,32 @@
"_source_dir_host": {
"description": "Source directory of the config file on the host (auto-generated by kas menu plugin, when using kas-container).",

"type": "string"
+ },
+ "buildtools": {
+ "type": "object",
+ "required": [
+ "version",
+ "sha256sum"
+ ],
+ "properties": {
+ "version": {
+ "description": "Yocto Project version, as 5.0 (or even 5.0.8) for Scarthgap.",
+ "type": "string"
+ },
+ "sha256sum": {
+ "description": "The installer's checksum, to perform integrity validation of the fetched artifact.",
+ "type": "string"
+ },
+ "base_url": {

+ "default": "https://downloads.yoctoproject.org/releases/yocto",
+ "description": "Base URL to fetch downloads from.",

+ "type": "string"
+ },

+ "filename": {

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 9 +++++++++
1 file changed, 9 insertions(+)

diff --git a/kas/config.py b/kas/config.py

index d868a5c..cebc749 100644

--- a/kas/config.py
+++ b/kas/config.py
@@ -264,3 +264,12 @@ class Config:
else:
return {k: v for k, v in signers.items()
if v.get('type', 'gpg') == keytype}
+
+ def get_buildtools(self):
+ """
+ Returns the buildtools keys: version, download URL and
+ archive filename. These are provided so kas knows which
+ buildtools archive to fetch and from what source.
+ """
+

+ return self._config.get('buildtools')
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add the core of buildtools support: download the installer, check its
integrity, extract it, and set up the environment.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 145 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 145 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 89113fa..4be894f 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -31,11 +31,18 @@ import logging

import tempfile
import asyncio
import errno

+import hashlib
+import hmac

import pathlib
+import platform
+import shutil
import signal
+import stat
from subprocess import Popen, PIPE, run as subprocess_run
+from urllib.parse import quote
from .context import get_context
from .kasusererror import KasUserError, CommandExecError

+from .configschema import CONFIGSCHEMA

__license__ = 'MIT'
__copyright__ = 'Copyright (c) Siemens AG, 2017-2018'
@@ -258,6 +265,110 @@ def repos_apply_patches(repos):

raise TaskExecError('apply patches', e.ret_code)


+def get_buildtools_dir():
+ # Set the dest. directory for buildtools's setup

+ env_path = os.environ.get("KAS_BUILDTOOLS_DIR")
+ if env_path:
+ return pathlib.Path(env_path).resolve()
+
+ # defaults to KAS_BUILD_DIR/buildtools
+ return (pathlib.Path(get_context().build_dir) / 'buildtools').resolve()

+
+def get_buildtools_filename():
+ arch = platform.machine()
+ ctx = get_context()
+
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ if 'filename' in conf_buildtools:
+ filename = conf_buildtools['filename']
+ else:

+ filename = f"{arch}-buildtools-extended-nativesdk-standalone-{version}.sh"
+
+ return filename
+
+
+def get_buildtools_path():
+ return get_buildtools_dir() / get_buildtools_filename()

+
+
+def get_buildtools_url():
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ filename = get_buildtools_filename()
+ version = conf_buildtools['version']
+
+ if 'base_url' in conf_buildtools:
+ base_url = conf_buildtools['base_url']
+ else:

+ default = CONFIGSCHEMA['properties']['buildtools']['properties']['base_url']['default']
+ base_url = f"{default}/yocto-{version}/buildtools/"

+
+ return f"{base_url}/{quote(filename)}"
+
+

+def check_sha256sum(filename, expected_checksum):

+ hash_sha256 = hashlib.sha256()
+ with open(filename, "rb") as f:
+ for chunk in iter(lambda: f.read(8192), b""):
+ hash_sha256.update(chunk)
+

+ actual_checksum = hash_sha256.hexdigest()
+ logging.info(f"Buildtools installer's checksum (sha256) is: {actual_checksum}")
+
+ return hmac.compare_digest(actual_checksum, expected_checksum)

+
+
+def download_buildtools():
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ buildtools_dir = get_buildtools_dir()
+
+ # Enable extended buildtools tarball
+ buildtools_url = get_buildtools_url()
+ tmpbuildtools = get_buildtools_path()
+

+ logging.info(f"Downloading Buildtools {version}")
+ # Download installer

+ fetch_cmd = ['wget', '-q', '-O', str(tmpbuildtools), buildtools_url]

+ (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
+ if ret != 0:
+ raise InitBuildEnvError("Could not download buildtools installer")
+
+ # Check if the installer's sha256sum matches

+ if not check_sha256sum(tmpbuildtools, conf_buildtools['sha256sum']):

+ raise InitBuildEnvError("sha256sum mismatch: installer may be corrupted")
+
+ # Make installer executable

+ st = tmpbuildtools.stat()
+ tmpbuildtools.chmod(st.st_mode | stat.S_IEXEC)
+
+ # Run installer (in an isolated environment)
+ installer_cmd = [str(tmpbuildtools), '-d', str(buildtools_dir), '-y']

+ env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}

+ (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)

+ if ret != 0:
+ raise InitBuildEnvError("Could not run buildtools installer")
+
+
+def get_buildtools_version():
+ try:

+ version_file = list(get_buildtools_dir().glob("version-*"))

+ if len(version_file) != 1:
+ raise ValueError("Invalid number of version files")
+

+ with version_file[0].resolve().open('r') as f:
+ lines = f.readlines()
+ for line in lines:
+ if line.startswith("Distro Version"):

+ return lines[1].split(':', 1)[1].strip()
+ except Exception as e:
+ logging.warning(f"Unable to read buildtools version: {e}")
+
+ return -1
+
+
def get_build_environ(build_system):
"""
Creates the build environment variables.

@@ -288,9 +399,43 @@ def get_build_environ(build_system):

if not init_repo:
raise InitBuildEnvError('Did not find any init-build-env script')

+ conf_buildtools = get_context().config.get_buildtools()
+ buildtools_env = ""
+
+ if conf_buildtools:
+ # Create the dest. directory if it doesn't exist
+ buildtools_dir = get_buildtools_dir()

+ buildtools_dir.mkdir(parents=True, exist_ok=True)
+
+ if not any(buildtools_dir.iterdir()):

+ # Directory is empty, try to fetch from upstream
+ logging.info(f"Buildtools ({buildtools_dir}): directory is empty")
+ download_buildtools()
+ else:
+ # Directory is not empty: fetch buildtools if the versions don't match
+ found_version = get_buildtools_version()
+ if found_version != conf_buildtools['version']:
+ logging.warning("Buildtools: version mismatch")
+ logging.info(f"Required version: {conf_buildtools['version']}")
+ logging.info(f"Found version: {found_version}")
+ shutil.rmtree(os.path.realpath(buildtools_dir))
+ os.makedirs(os.path.realpath(buildtools_dir))
+ download_buildtools()
+

+ envfiles = list(get_buildtools_dir().glob("environment-setup-*"))

+ if len(envfiles) == 1:

+ buildtools_env = "source {}\n".format(envfiles[0].resolve())

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/command-line/environment-variables.inc | 4 ++
docs/userguide/project-configuration.rst | 70 +++++++++++++++++++++
2 files changed, 74 insertions(+)

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc

index 875f4b3..a3d0051 100644

--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -200,6 +200,10 @@ overwritten using the ``env`` section of the config file.
| (C) | ``docker`` or ``podman``). If not set, this is |
| | auto-detected (preference: docker). |
+--------------------------+--------------------------------------------------+
+| ``KAS_BUILDTOOLS_DIR`` | Explicitly set the path where kas will download |

+| (C,K) | and install buildtools. If not set, kas will use |

+| | ``KAS_BUILD_DIR/buildtools`` as the default path.|
++--------------------------+--------------------------------------------------+

.. |aws_cred| replace:: ``AWS_ROLE_ARN``
``AWS_SHARED_CREDENTIALS_FILE``
diff --git a/docs/userguide/project-configuration.rst b/docs/userguide/project-configuration.rst

index 9bdce1f..a7e17f5 100644

+
+ ``filename``: string [optional]

+ :kasschemadesc:`buildtools.properties.filename`
+ It will be combined with to ``base_url`` to form the whole download URL, if
+ set. If not set, kas will combine the platform architecture and ``version``
+ to form the standard script filename:
+ ``{arch}-buildtools-extended-nativesdk-standalone-{version}.sh``
+
+ Example:

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

The original expression didn't support numbers, so 'sha256sum' from
'buildtools' was raising the error below:

kas/docs/userguide/project-configuration.rst:527: ERROR: Invalid path: buildtools.properties.sha256sum

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/_ext/sphinx_kas_schema.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/_ext/sphinx_kas_schema.py b/docs/_ext/sphinx_kas_schema.py
index 5ea7f54..40e35c0 100644
--- a/docs/_ext/sphinx_kas_schema.py
+++ b/docs/_ext/sphinx_kas_schema.py
@@ -45,7 +45,7 @@ class KasSchemaDescRole(SphinxRole):

def __init__(self):
super().__init__()
- self.key_regex = re.compile(r'([a-zA-Z_]+)(?:\[(\d+)\])?')
+ self.key_regex = re.compile(r'([a-zA-Z0-9_]+)(?:\[(\d+)\])?')

def run(self) -> tuple[list[nodes.Node], list[nodes.system_message]]:
messages = []
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas-container | 2 ++
1 file changed, 2 insertions(+)

diff --git a/kas-container b/kas-container
index e873588..1f850dc 100755

--- a/kas-container
+++ b/kas-container
@@ -283,6 +283,7 @@ setup_kas_dirs()
KAS_REPO_REF_DIR="$(check_and_expand KAS_REPO_REF_DIR required)"
DL_DIR="$(check_and_expand DL_DIR createrec)"
SSTATE_DIR="$(check_and_expand SSTATE_DIR createrec)"
+ KAS_BUILDTOOLS_DIR="$(check_and_expand KAS_BUILDTOOLS_DIR createrec)"
}
setup_kas_dirs

@@ -593,6 +594,7 @@ forward_dir KAS_BUILD_DIR "/build" "rw"
forward_dir DL_DIR "/downloads" "rw"
forward_dir KAS_REPO_REF_DIR "/repo-ref" "rw"
forward_dir SSTATE_DIR "/sstate" "rw"
+forward_dir KAS_BUILDTOOLS_DIR "/buildtools" "rw"

if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
&& [ "$git_com_dir" != "$(git -C "${KAS_REPO_DIR}" rev-parse --git-dir)" ]; then
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is a test case based on kas.yml, using Yocto 5.2 (Walnascar).

Please note that kas-buildtools.yml is merely a symbolic link to
examples/openembedded-buildtools.yml.

openembedded-buildtools.yml is based on examples/openembedded.yml, only
setting a different revision (i.e., Walnascar) for Poky repository.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

examples/openembedded-buildtools.yml | 45 ++++++++++++++++++++++++++++
image-tests/kas/kas-buildtools.yml | 1 +
2 files changed, 46 insertions(+)

create mode 100644 examples/openembedded-buildtools.yml
create mode 120000 image-tests/kas/kas-buildtools.yml

diff --git a/examples/openembedded-buildtools.yml b/examples/openembedded-buildtools.yml
new file mode 100644
index 0000000..b0a68a1
--- /dev/null
+++ b/examples/openembedded-buildtools.yml
@@ -0,0 +1,45 @@
+#
+# kas - setup tool for bitbake based projects
+#
+# Copyright (c) Siemens AG, 2022-2025
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be
+# included in all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+header:
+ version: 19
+ includes:
+ - openembedded.yml
+
+repos:
+ poky:
+ url: https://git.yoctoproject.org/poky.git
+ # when specifying a tag, optionally provide a commit hash
+ tag: yocto-5.2
+ commit: 9b96fdbb0cab02f4a6180e812b02bc9d4c41b1a5
+ signed: true
+ allowed_signers:
+ - YoctoBuildandRelease
+ layers:
+ meta:
+ meta-poky:
+
+buildtools:
+ version: "5.2"
+ sha256sum: "6f69fba75c8f3142bb49558afa3ed5dd0723a3beda169b057a5238013623462d"
diff --git a/image-tests/kas/kas-buildtools.yml b/image-tests/kas/kas-buildtools.yml
new file mode 120000
index 0000000..7b17a91
--- /dev/null
+++ b/image-tests/kas/kas-buildtools.yml
@@ -0,0 +1 @@
+../../examples/openembedded-buildtools.yml
\ No newline at end of file
--
2.47.0

[MOESSBAUER, Felix]

On Fri, 2025-07-18 at 15:29 +0200, joaomarcos.costa via kas-devel
wrote:

> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>
> The original expression didn't support numbers, so 'sha256sum' from
> 'buildtools' was raising the error below:
>
> kas/docs/userguide/project-configuration.rst:527: ERROR: Invalid
> path: buildtools.properties.sha256sum

Good catch!

Acked-by: Felix Moessbauer <felix.mo...@siemens.com>

@Jan: This can be applied individually, if needed.

>
> Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
> ---
>  docs/_ext/sphinx_kas_schema.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/docs/_ext/sphinx_kas_schema.py
> b/docs/_ext/sphinx_kas_schema.py
> index 5ea7f54..40e35c0 100644
> --- a/docs/_ext/sphinx_kas_schema.py
> +++ b/docs/_ext/sphinx_kas_schema.py
> @@ -45,7 +45,7 @@ class KasSchemaDescRole(SphinxRole):
>  
>      def __init__(self):
>          super().__init__()
> -        self.key_regex = re.compile(r'([a-zA-Z_]+)(?:\[(\d+)\])?')
> +        self.key_regex = re.compile(r'([a-zA-Z0-
> 9_]+)(?:\[(\d+)\])?')
>  
>      def run(self) -> tuple[list[nodes.Node],
> list[nodes.system_message]]:
>          messages = []
> --
> 2.47.0

[Joao Marcos Costa]

Hello Felix,

On 7/23/25 11:10, MOESSBAUER, Felix wrote:
> On Fri, 2025-07-18 at 15:29 +0200, joaomarcos.costa via kas-devel
> wrote:
>> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>>
>> The original expression didn't support numbers, so 'sha256sum' from
>> 'buildtools' was raising the error below:
>>
>> kas/docs/userguide/project-configuration.rst:527: ERROR: Invalid
>> path: buildtools.properties.sha256sum
>
> Good catch! >
> Acked-by: Felix Moessbauer <felix.mo...@siemens.com>
>
> @Jan: This can be applied individually, if needed.
>
>>
>> Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>

To be fair, I should credit my colleague Antonin Godard
<antonin...@bootlin.com> for this one! :)

Could you please fix this if you commit this patch? Thanks!

[Joao Marcos Costa]

Hello,

I hope this email finds you well.

Would there be any updates on this patch series?

On 7/18/25 15:29, joaomarc...@bootlin.com wrote:
> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>
> Hello,
>
> This patch series allows Kas to fetch and install Buildtools, and subsequently
> run Bitbake with the Buildtools environment fully set-up. The use case addresses
> limitations in development environments, such as outdated distributions,
> significantly older Python versions, and, most importantly, the absence of a
> containerization solution.
>
> Best regards,
>
> ---
>

[Jan Kiszka]

On 30.07.25 09:41, 'Joao Marcos Costa' via kas-devel wrote:
> Hello,
>
> I hope this email finds you well.
>
> Would there be any updates on this patch series?

In my backlog. Will review it this week.

Jan

>
> On 7/18/25 15:29, joaomarc...@bootlin.com wrote:
>> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>>
>> Hello,
>>
>> This patch series allows Kas to fetch and install Buildtools, and
>> subsequently
>> run Bitbake with the Buildtools environment fully set-up. The use case
>> addresses
>> limitations in development environments, such as outdated distributions,
>> significantly older Python versions, and, most importantly, the
>> absence of a
>> containerization solution.
>>
>> Best regards,
>>
>> ---
>>
>
> Thanks!

--

[Jan Kiszka]

Reported-by? This is what I did now in next while applying this patch
(ahead of the others).

Thanks,
Jan

[Jan Kiszka]

Err, but this test is not auto-magically picked up by our CI. The "Test
image" step in .github/workflows/next.yml is only looking for kas.yml
configs. You would have to extend that workflow with something that does
that, but only for the "kas" image-name.

How much time will that run take, also in comparison to our existing
image smoke test? Maybe it makes sense and is still sufficiently
covering the feature if we build kas-buildtools.yml right after the
regular kas.yml build, to reuse some fetches e.g. Please try that out in
your own pipeline.

[Jan Kiszka]

On 05.08.25 12:38, 'Jan Kiszka' via kas-devel wrote:
> On 30.07.25 09:41, 'Joao Marcos Costa' via kas-devel wrote:
>> Hello,
>>
>> I hope this email finds you well.
>>
>> Would there be any updates on this patch series?
>
> In my backlog. Will review it this week.
>

Small remark on the new last patch only, rest looks good. I'm holding
the rest back, though, until after some 4.8.2 which may come out soon.

Jan

[Joao Marcos Costa]

Hello, Jan

On 8/7/25 19:13, Jan Kiszka wrote:
> On 18.07.25 15:29, joaomarc...@bootlin.com wrote:
>> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>>
>> This is a test case based on kas.yml, using Yocto 5.2 (Walnascar).
>>
>> Please note that kas-buildtools.yml is merely a symbolic link to
>> examples/openembedded-buildtools.yml.
>>
>> openembedded-buildtools.yml is based on examples/openembedded.yml, only
>> setting a different revision (i.e., Walnascar) for Poky repository.
>>
>> Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
>> ---
>> examples/openembedded-buildtools.yml | 45 ++++++++++++++++++++++++++++
>> image-tests/kas/kas-buildtools.yml | 1 +
>> 2 files changed, 46 insertions(+)
>> create mode 100644 examples/openembedded-buildtools.yml
>> create mode 120000 image-tests/kas/kas-buildtools.yml
>>

(...)

>
> Err, but this test is not auto-magically picked up by our CI. The "Test
> image" step in .github/workflows/next.yml is only looking for kas.yml
> configs. You would have to extend that workflow with something that does
> that, but only for the "kas" image-name.
>
> How much time will that run take, also in comparison to our existing
> image smoke test? Maybe it makes sense and is still sufficiently
> covering the feature if we build kas-buildtools.yml right after the
> regular kas.yml build, to reuse some fetches e.g. Please try that out in
> your own pipeline.

I came up with this:
https://github.com/jmarcoscosta/kas/commit/9f4a1e2d8515a26c4237340418171e462e44b266

I'm not sure it is the most idiomatic way, though, as I'm not really
used to these Github workflows. Please let me know what you think.
Here's the corresponding result:
https://github.com/jmarcoscosta/kas/actions/runs/16826415898

Thanks!

>
> Thanks,
> Jan
>

p.s.: I skipped the code checking step for now, and needless to say I
will bring it back once we agree upon this image-tests commit.

[Jan Kiszka]

Cleaning/purging does not need to happen against both configs. They are
identical /wrt that task (only build_system matters).

[Jan Kiszka]

Actually, purging needs more than build_system. And that makes me wonder
if the artifacts of the buildtools fetching are properly purged already,
namely KAS_BUILDTOOLS_DIR. Please ensure that this is covered.

[Joao Marcos Costa]

Hello Jan,

On 8/8/25 11:42, Jan Kiszka wrote:

(...)

>>>> Err, but this test is not auto-magically picked up by our CI. The "Test
>>>> image" step in .github/workflows/next.yml is only looking for kas.yml
>>>> configs. You would have to extend that workflow with something that does
>>>> that, but only for the "kas" image-name.
>>>>
>>>> How much time will that run take, also in comparison to our existing
>>>> image smoke test? Maybe it makes sense and is still sufficiently
>>>> covering the feature if we build kas-buildtools.yml right after the
>>>> regular kas.yml build, to reuse some fetches e.g. Please try that out in
>>>> your own pipeline.
>>>
>>> I came up with this:
>>> https://github.com/jmarcoscosta/kas/
>>> commit/9f4a1e2d8515a26c4237340418171e462e44b266
>>>
>>> I'm not sure it is the most idiomatic way, though, as I'm not really
>>> used to these Github workflows. Please let me know what you think.
>>> Here's the corresponding result:
>>> https://github.com/jmarcoscosta/kas/actions/runs/16826415898
>>>
>>
>> Cleaning/purging does not need to happen against both configs. They are
>> identical /wrt that task (only build_system matters).
>
> Actually, purging needs more than build_system. And that makes me wonder
> if the artifacts of the buildtools fetching are properly purged already,
> namely KAS_BUILDTOOLS_DIR. Please ensure that this is covered.

Could you please clarify what you meant with "purging needs more than
build_system"?

I was thinking of adding a bit more code to Purge (@
kas/plugins/clean.py) to remove whatever is the content of
KAS_BUILDTOOLS_DIR, and leave 'clean' itself unchanged. Thus, the only
change in next.yml would be:

+ if [ "${{ matrix.image-name }}" = "kas" ]; then
+ ../../kas-container build kas-buildtools.yml
+ fi

Does this seem appropriate to you?
Thanks.

[Jan Kiszka]

Yeah, that should work if just all content of KAS_BUILDTOOLS_DIR needs
to be deleted. Maybe Felix can help cross-checking that we are not
missing a detail.

[Joao Marcos Costa]

Hello,

On 9/1/25 16:47, Jan Kiszka wrote:
> On 01.09.25 15:47, Joao Marcos Costa wrote:
>> Hello Jan,
>>
>> On 8/8/25 11:42, Jan Kiszka wrote:
>>
>> (...)
>>

(...)

>>
>> Could you please clarify what you meant with "purging needs more than
>> build_system"?
>>
>> I was thinking of adding a bit more code to Purge (@ kas/plugins/
>> clean.py) to remove whatever is the content of KAS_BUILDTOOLS_DIR, and
>> leave 'clean' itself unchanged. Thus, the only change in next.yml would be:
>>
>> +          if  [ "${{ matrix.image-name }}" = "kas" ]; then
>> +            ../../kas-container build kas-buildtools.yml
>> +          fi
>>
>> Does this seem appropriate to you?
>> Thanks.
>
> Yeah, that should work if just all content of KAS_BUILDTOOLS_DIR needs
> to be deleted. Maybe Felix can help cross-checking that we are not
> missing a detail.
>
> Jan
>

Just in case, here's the job:

https://github.com/jmarcoscosta/kas/actions/runs/17381051677

The corresponding changes are in the last couple commits in my 'next'
branch: https://github.com/jmarcoscosta/kas/commits/refs/heads/next/

FYI, I fixed the code style issues, so whenever we agree upon the CI
side of the series I will send the v7.

[Joao Marcos Costa]

Hello,

On 9/1/25 16:47, Jan Kiszka wrote:

Just in case, here's the last run:
https://github.com/jmarcoscosta/kas/actions/runs/17397304181/job/49382398080#step:5:297

If you could please take a look on that, I'd appreciate it. Then, I
would proceed to send the v7 of the series.

Thanks!

[Jan Kiszka]

2025-09-02 08:15:59 - WARNING - Falling back to file-relative addressing of local include "openembedded.yml"
2025-09-02 08:15:59 - WARNING - Update your layer to repo-relative addressing to avoid this warning

Please address this, and then send v7.

Thanks,