[PATCH v3 0/4] Add buildtools support

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Hello,

This patch series allows Kas to fetch and install Buildtools, and subsequently
run Bitbake with the Buildtools environment fully set-up. The use case addresses
limitations in development environments, such as outdated distributions,
significantly older Python versions, and, most importantly, the absence of a
containerization solution.

Best regards,

---

Changes in v3:
- added the "required" marker for "version" in the schema
- added documentation about the expected internal structure of the buildtools
archive
- mention 'wget' as a dependency, since it is needed to fetch buildtools
installer
- added more error handling to get_buildtools_upstream()
- changed the patch series title

Joao Marcos Costa (4):
schema-kas: introduce buildtools configuration entry
config.py: add get_buildtools to read buildtools properties
kas: introduce support to buildtools
docs: introduce buildtools configuration entry

docs/command-line/environment-variables.inc | 4 +
docs/userguide/project-configuration.rst | 65 ++++++++++++
kas/config.py | 23 +++++
kas/libkas.py | 106 ++++++++++++++++++++
kas/schema-kas.json | 20 ++++
5 files changed, 218 insertions(+)

--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

This is the first step towards adding the support for buildtools in kas.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---
kas/schema-kas.json | 20 ++++++++++++++++++++
1 file changed, 20 insertions(+)

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index 700b48f..96f949f 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -413,6 +413,26 @@
"_source_dir_host": {
"description": "Source directory of the config file on the host (auto-generated by kas menu plugin, when using kas-container).",
"type": "string"
+ },
+ "buildtools": {
+ "type": "object",
+ "required": [
+ "version"
+ ],
+ "properties": {
+ "version": {
+ "description": "Yocto Project version, as 5.0 (or even 5.0.8) for Scarthgap.",
+ "type": "string"
+ },
+ "base_url": {
+ "description": "Base URL to fetch downloads from, used instead of the default Yocto source (downloads.yoctoproject.org/releases/yocto).",
+ "type": "string"
+ },
+ "filename": {
+ "description": "Alternative name for the buildtools archive (.sh) to be downloaded.",
+ "type": "string"
+ }
+ }
}
}
}
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

These variables are used as parameters for buildtools support in kas, so
they should be documented accordingly.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

docs/command-line/environment-variables.inc | 4 ++
docs/userguide/project-configuration.rst | 65 +++++++++++++++++++++
2 files changed, 69 insertions(+)

diff --git a/docs/command-line/environment-variables.inc b/docs/command-line/environment-variables.inc
index fb3dded..2551aef 100644
--- a/docs/command-line/environment-variables.inc
+++ b/docs/command-line/environment-variables.inc
@@ -195,6 +195,10 @@ overwritten using the ``env`` section of the config file.
| (C) | ``docker`` or ``podman``). If not set, this is |
| | auto-detected (preference: docker). |
+--------------------------+--------------------------------------------------+
+| ``KAS_BUILDTOOLS_DIR`` | Explicitly set the path where kas will download |
+| (K) | and install buildtools. If not set, kas will use |
+| | ``KAS_BUILD_DIR/buildtools`` as the default path.|
++--------------------------+--------------------------------------------------+

.. |aws_cred| replace:: ``AWS_ROLE_ARN``
``AWS_SHARED_CREDENTIALS_FILE``
diff --git a/docs/userguide/project-configuration.rst b/docs/userguide/project-configuration.rst
index dc37dba..5cb1fc2 100644
--- a/docs/userguide/project-configuration.rst
+++ b/docs/userguide/project-configuration.rst
@@ -481,6 +481,71 @@ Configuration reference
``kas-container`` script. It must not be set manually and might only be
defined in the top-level ``.config.yaml`` file.

+``buildtools``: dict [optional]
+ Provides variables to define which buildtools version should be fetched and
+ where it is (or will be) installed. At least ``version`` should be set. The
+ environment variable ``KAS_BUILDTOOLS_DIR`` can be used to set the directory
+ where buildtools will be installed, otherwise the default path (i.e.,
+ ``KAS_BUILD_DIR/buildtools``) will be used. If such directory already has
+ buildtools installed, kas will check the ``Distro Version`` line in the
+ version file, and if it doesn't match with ``version``, the directory will
+ be cleaned and kas will download buildtools according to ``version``. As for
+ the optional variables, they are meant to be used to support cases as:
+ mirrors, changes in the installer's file name, and fetching unofficial (i.e.,
+ custom) buildtools. Finally, the environment-setup script will run before
+ bitbake, so the whole buildtools environment will be available. ``wget`` is
+ the host tool required for this feature. More information on how to install
+ or generate buildtools can be found at: |yp_doc_buildtools|
+
+ ``version``: string
+ :kasschemadesc:`buildtools.properties.version`
+
+ ``base_url``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+
+ ``filename``: string [optional]
+ :kasschemadesc:`buildtools.properties.base_url`
+ It will be combined with to ``base_url`` to form the whole URL to be passed
+ to ``wget``, if set. If not set, kas will combine the platform architecture
+ and ``version`` to form the standard script filename:
+ ``{arch}-buildtools-extended-nativesdk-standalone-{version}.sh``
+
+ Example:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "5.0.5"
+
+ And for unofficial (custom) sources:
+
+ .. code-block:: yaml
+
+ buildtools:
+ version: "1.0.0"
+ base_url: "https://downloads.mysources.com/yocto/buildtools/"
+ filename: "x86_64-buildtools-beta-testing-1.0.0.sh"
+
+.. |yp_doc_buildtools| replace:: https://docs.yoctoproject.org/dev/ref-manual/system-requirements.html#downloading-a-pre-built-buildtools-tarball
+
+Buildtools archive
+------------------
+
+kas expects the buildtools installer to be a shell script (i.e., as a standard
+Yocto SDK). Once executed, the resulting directory should contain the elements
+below:
+
+- ``sysroots``: the native and target sysroots, containing (among libraries and
+ headers) the build system's requirements: Git, tar, Python and make.
+- ``environment-setup-*``: the environment setup script, sourced by kas, to
+ setup variables such as ``PATH`` in such a way that it points to
+ the directories in ``sysroots``.
+- ``version-*``: the version file. Its second line contains a string as
+ ``Distro Version: X.Y.Z``, parsed to retrieve the version number.
+
+The archive can contain other files, such as ``buildinfo``, but they are not
+relevant for kas.
+
.. _example-configurations-label:

Example project configurations
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Introduce buildtools configuration scheme:

- version: sets the buildtools version to be fetched
- base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
- filename: optional archive filename.

All of the properties above are further documented in a follow-up commit.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/libkas.py | 106 ++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 106 insertions(+)

diff --git a/kas/libkas.py b/kas/libkas.py
index 2c353d2..9d26b7c 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -31,9 +31,14 @@ import logging
import tempfile
import asyncio
import errno
+import glob
import pathlib
+import platform
+import shutil
import signal
+import stat
from subprocess import Popen, PIPE, run as subprocess_run
+from urllib.parse import quote
from .context import get_context
from .kasusererror import KasUserError, CommandExecError

@@ -258,6 +263,74 @@ def repos_apply_patches(repos):
raise TaskExecError('apply patches', e.ret_code)


+def get_buildtools_path():
+ # Set the dest. directory for buildtools's setup
+ env = os.environ.get("KAS_BUILDTOOLS_DIR")
+ if env:
+ buildtools_dir = os.path.realpath(env)
+ else:
+ buildtools_dir = get_context().build_dir + "/buildtools"
+
+ return buildtools_dir
+
+
+def get_buildtools_upstream():
+ arch = platform.machine()
+ ctx = get_context()
+ conf_buildtools = ctx.config.get_buildtools()
+ version = conf_buildtools['version']
+ buildtools_dir = get_buildtools_path()
+
+ if conf_buildtools['base_url']:
+ base_url = conf_buildtools['base_url']
+ else:
+ yocto_releases = "https://downloads.yoctoproject.org/releases/yocto"
+ base_url = f"{yocto_releases}/yocto-{version}/buildtools/"
+
+ if conf_buildtools['filename']:
+ filename = conf_buildtools['filename']
+ else:
+ filename = f"{arch}-buildtools-extended-nativesdk-standalone-{version}.sh"
+
+ # Enable extended buildtools tarball
+ buildtools_url = f"{base_url}/{quote(filename)}"
+ tmpbuildtools = os.path.join(buildtools_dir, filename)
+
+ logging.info(f"Downloading Buildtools {version}")
+ # Download installer
+ fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]
+ (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
+ if ret != 0:
+ raise InitBuildEnvError("Could not download buildtools installer")
+
+ # Make installer executable
+ st = os.stat(tmpbuildtools)
+ os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
+
+ # Run installer
+ installer_cmd = [tmpbuildtools, '-d', buildtools_dir, '-y']
+ env = {'PATH': '/usr/sbin:/usr/bin:/sbin:/bin'}
+ (ret, _) = run_cmd(installer_cmd, cwd=ctx.kas_work_dir, env=env)
+ if ret != 0:
+ raise InitBuildEnvError("Could not run buildtools installer")
+
+
+def get_buildtools_version():
+ buildtools_dir = get_buildtools_path()
+ version_file = glob.glob(os.path.join(buildtools_dir, "version-*"))
+
+ if len(version_file) == 1:
+ version_file = os.path.realpath(version_file[0])
+ else:
+ logging.warning("Unable to read buildtools version.")
+ return -1
+
+ with open(version_file, 'r') as f:
+ lines = f.readlines()
+
+ return lines[1].split(':')[1].strip()
+
+
def get_build_environ(build_system):
"""
Creates the build environment variables.
@@ -288,9 +361,42 @@ def get_build_environ(build_system):
if not init_repo:
raise InitBuildEnvError('Did not find any init-build-env script')

+ conf_buildtools = get_context().config.get_buildtools()
+ buildtools_env = ""
+
+ if conf_buildtools:
+ # Create the dest. directory if it doesn't exist
+ buildtools_dir = get_buildtools_path()
+ pathlib.Path(buildtools_dir).mkdir(parents=True, exist_ok=True)
+
+ if not os.listdir(buildtools_dir):
+ # Directory is empty, try to fetch from upstream
+ logging.info(f"Buildtools ({buildtools_dir}): directory is empty")
+ if "version" in conf_buildtools:
+ get_buildtools_upstream()
+ else:
+ raise InitBuildEnvError('Unable to fetch buildtools from upstream:'
+ 'version was not set')
+ else:
+ # Directory is not empty: fetch buildtools if the versions don't match
+ if "version" in conf_buildtools:
+ found_version = get_buildtools_version()
+ if found_version != conf_buildtools['version']:
+ logging.warning("Buildtools: version mismatch")
+ logging.info(f"Required version: {conf_buildtools['version']}")
+ logging.info(f"Found version: {found_version}")
+ shutil.rmtree(os.path.realpath(buildtools_dir))
+ os.makedirs(os.path.realpath(buildtools_dir))
+ get_buildtools_upstream()
+
+ envfiles = glob.glob(os.path.join(buildtools_dir, "environment-setup-*"))
+ if len(envfiles) == 1:
+ buildtools_env = "source {}\n".format(os.path.realpath(envfiles[0]))
+
with tempfile.TemporaryDirectory() as temp_dir:
script = f"""#!/bin/bash
set -e
+ {buildtools_env}
source {init_script} $1 > /dev/null
env
"""
--
2.47.0

[joaomarc...@bootlin.com]

From: Joao Marcos Costa <joaomarc...@bootlin.com>

Add helper to version, base_url and filename from buildtools entry in
the project configuration.

Signed-off-by: Joao Marcos Costa <joaomarc...@bootlin.com>
---

kas/config.py | 23 +++++++++++++++++++++++
1 file changed, 23 insertions(+)

diff --git a/kas/config.py b/kas/config.py
index fe5d1f0..94bb373 100644
--- a/kas/config.py
+++ b/kas/config.py
@@ -253,3 +253,26 @@ class Config:
else:
return {k: v for k, v in signers.items()
if v.get('type', 'gpg') == keytype}
+
+ def get_buildtools(self):
+ """
+ Returns the buildtools keys: version, download URL and
+ archive filename. These are provided so kas knows which
+ buildtools archive to fetch and from what source.
+ """
+ buildtools = self._config.get('buildtools', {})
+ if not buildtools:
+ return {}
+
+ conf = {'version': buildtools['version']}
+ if 'base_url' in buildtools:
+ conf['base_url'] = buildtools['base_url']
+ else:
+ conf['base_url'] = None
+
+ if 'filename' in buildtools:
+ conf['filename'] = buildtools['filename']
+ else:
+ conf['filename'] = None
+
+ return conf
--
2.47.0

[Jan Kiszka]

I think we also need a mandatory sha256 key. There would otherwise be no
integrity validation of the fetched artifact possible.

> }
> }
> }

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[Jan Kiszka]

On 12.05.25 16:11, joaomarcos.costa via kas-devel wrote:

This complete method can be simplified to

return self._config.get('buildtools', {})

You don't perform any modifications on what you retrieve from the config.

[Jan Kiszka]

On 12.05.25 16:11, joaomarcos.costa via kas-devel wrote:

> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>
> Introduce buildtools configuration scheme:
>
> - version: sets the buildtools version to be fetched
> - base_url: optional url property. If not set, kas will use downloads.yoctoproject.org
> - filename: optional archive filename.
>
> All of the properties above are further documented in a follow-up commit.

Looks like you wanted to add this commit message to patch 1 instead.

Missing kas-container support for this new env variable. Please add that
as well.

> +
> +
> +def get_buildtools_upstream():

"Upstream" is a bit misleading term here. Just call this
"download_buildtools".

> + arch = platform.machine()
> + ctx = get_context()
> + conf_buildtools = ctx.config.get_buildtools()
> + version = conf_buildtools['version']
> + buildtools_dir = get_buildtools_path()
> +
> + if conf_buildtools['base_url']:
> + base_url = conf_buildtools['base_url']
> + else:
> + yocto_releases = "https://downloads.yoctoproject.org/releases/yocto"
> + base_url = f"{yocto_releases}/yocto-{version}/buildtools/"

I think we should also use KAS_PREMIRRORS to translate this URL. This
can be done later on in separate commit, but it will be fairly important
for projects that prefer internal mirrors over upstream, e.g. in
corporate environments.

> +
> + if conf_buildtools['filename']:
> + filename = conf_buildtools['filename']
> + else:
> + filename = f"{arch}-buildtools-extended-nativesdk-standalone-{version}.sh"
> +
> + # Enable extended buildtools tarball
> + buildtools_url = f"{base_url}/{quote(filename)}"
> + tmpbuildtools = os.path.join(buildtools_dir, filename)
> +
> + logging.info(f"Downloading Buildtools {version}")
> + # Download installer
> + fetch_cmd = ['wget', '-q', '-O', tmpbuildtools, buildtools_url]
> + (ret, _) = run_cmd(fetch_cmd, cwd=ctx.kas_work_dir)
> + if ret != 0:
> + raise InitBuildEnvError("Could not download buildtools installer")
> +

Here you should add integrity validation.

It's more robust to simple scan for "Distro Version:" and parse that
line only. Please also update the documentation then.

> +
> +
> def get_build_environ(build_system):
> """
> Creates the build environment variables.
> @@ -288,9 +361,42 @@ def get_build_environ(build_system):
> if not init_repo:
> raise InitBuildEnvError('Did not find any init-build-env script')
>
> + conf_buildtools = get_context().config.get_buildtools()
> + buildtools_env = ""
> +
> + if conf_buildtools:
> + # Create the dest. directory if it doesn't exist
> + buildtools_dir = get_buildtools_path()
> + pathlib.Path(buildtools_dir).mkdir(parents=True, exist_ok=True)
> +
> + if not os.listdir(buildtools_dir):
> + # Directory is empty, try to fetch from upstream
> + logging.info(f"Buildtools ({buildtools_dir}): directory is empty")
> + if "version" in conf_buildtools:
> + get_buildtools_upstream()
> + else:
> + raise InitBuildEnvError('Unable to fetch buildtools from upstream:'
> + 'version was not set')

Version is mandatory now, so this branch is impossible.

> + else:
> + # Directory is not empty: fetch buildtools if the versions don't match
> + if "version" in conf_buildtools:

Same as above: always true.

> + found_version = get_buildtools_version()
> + if found_version != conf_buildtools['version']:
> + logging.warning("Buildtools: version mismatch")
> + logging.info(f"Required version: {conf_buildtools['version']}")
> + logging.info(f"Found version: {found_version}")
> + shutil.rmtree(os.path.realpath(buildtools_dir))
> + os.makedirs(os.path.realpath(buildtools_dir))
> + get_buildtools_upstream()
> +
> + envfiles = glob.glob(os.path.join(buildtools_dir, "environment-setup-*"))
> + if len(envfiles) == 1:
> + buildtools_env = "source {}\n".format(os.path.realpath(envfiles[0]))

And if len(envfiles) != 1, I guess we should rather fail and alarm the
user about an invalid buildtools package.

> +
> with tempfile.TemporaryDirectory() as temp_dir:
> script = f"""#!/bin/bash
> set -e
> + {buildtools_env}
> source {init_script} $1 > /dev/null
> env
> """

[Joao Marcos Costa]

Hello Jan,

I hope this email finds you well.

On 5/18/25 19:53, Jan Kiszka wrote:
> On 12.05.25 16:11, joaomarcos.costa via kas-devel wrote:
>> From: Joao Marcos Costa <joaomarc...@bootlin.com>
>>

(...)

>>
>>
>> +def get_buildtools_path():
>> + # Set the dest. directory for buildtools's setup
>> + env = os.environ.get("KAS_BUILDTOOLS_DIR")
>> + if env:
>> + buildtools_dir = os.path.realpath(env)
>> + else:
>> + buildtools_dir = get_context().build_dir + "/buildtools"
>> +
>> + return buildtools_dir
>
> Missing kas-container support for this new env variable. Please add that
> as well.

I wasn't really sure about this one, so I added it in a separate commit.

>> +
>> +
>> +def get_buildtools_upstream():
>
> "Upstream" is a bit misleading term here. Just call this
> "download_buildtools".
>
>> + arch = platform.machine()
>> + ctx = get_context()
>> + conf_buildtools = ctx.config.get_buildtools()
>> + version = conf_buildtools['version']
>> + buildtools_dir = get_buildtools_path()
>> +
>> + if conf_buildtools['base_url']:
>> + base_url = conf_buildtools['base_url']
>> + else:
>> + yocto_releases = "https://downloads.yoctoproject.org/releases/yocto"
>> + base_url = f"{yocto_releases}/yocto-{version}/buildtools/"
>
> I think we should also use KAS_PREMIRRORS to translate this URL. This
> can be done later on in separate commit, but it will be fairly important
> for projects that prefer internal mirrors over upstream, e.g. in
> corporate environments.
>

Setting 'base_url' should suffise to handle mirrors. If not, what should
be the priority here? The KAS_PREMIRRORS would take precedence over
'base_url'?

>
> Jan
>

Thanks!

--
Best regards,
João Marcos Costa

[Jan Kiszka]

KAS_PREMIRRORS would be tried first, then the configured URL.