[PATCH 0/4] Add support for upstream bitbake hashservers

[Felix Moessbauer]

This series simplifies the use of upstream bitbake hashservers by forwarding
the relevant environment variables into the bitbake environment.

Configuring the hashserver is still up to the user, but in later versions,
kas might provide sensible defaults. As this relies on external
infrastructure, we don't do this for now.

Best regards,
Felix Moessbauer
Siemens AG

Felix Moessbauer (3):
forward env vars related to upstream bitbake hashservers
example(oe): update to yocto-6.0.1 tag
ci: test hash equivalence server support against OE servers

Ross Burton (1):
kas: add BB_HASHSERVE_DB_DIR to allowed environment variables

.github/workflows/next.yml | 11 +++++++++++
docs/userguide/environment-variables.inc | 9 ++++++---
examples/oe-core-nodistro.yml | 8 ++++----
kas-container | 3 +++
kas/libkas.py | 3 ++-
5 files changed, 26 insertions(+), 8 deletions(-)

--
2.53.0

[Felix Moessbauer]

To reduce the build times, federated (upstream) hash equivalence
servers can be used in bitbake. To also make this possible in kas,
we forward the corresponding environment variables.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
docs/userguide/environment-variables.inc | 4 +++-
kas/libkas.py | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/docs/userguide/environment-variables.inc b/docs/userguide/environment-variables.inc
index fecf9f504..42ede08bf 100644
--- a/docs/userguide/environment-variables.inc
+++ b/docs/userguide/environment-variables.inc
@@ -133,7 +133,9 @@ overwritten using the ``env`` section of the config file.
| ``SSTATE_DIR`` | the bitbake environment. The ``DL_DIR``, |
| ``SSTATE_MIRRORS`` | ``SSTATE_DIR`` and ``BB_HASHSERVE_DB_DIR`` |
| ``BB_HASHSERVE_DB_DIR`` | directories are created along with their |
-| (C,K,E,c) | parents, if set. |
+| ``BB_HASHSERVE`` | parents, if set. |
+| ``BB_HASHSERVE_UPSTREAM``| |
+| (C,K,E,c) | |
+--------------------------+--------------------------------------------------+
| ``TMPDIR`` (K,E,c) | Directory for temporary files. |
+--------------------------+--------------------------------------------------+
diff --git a/kas/libkas.py b/kas/libkas.py
index b9b2d70e7..5aaf99ba9 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -495,7 +495,7 @@ def get_build_environ(build_system, ctx):
conf_env = ctx.config.get_environment()

env_vars = ['SSTATE_DIR', 'SSTATE_MIRRORS', 'BB_HASHSERVE_DB_DIR',
- 'DL_DIR', 'TMPDIR']
+ 'BB_HASHSERVE', 'BB_HASHSERVE_UPSTREAM', 'DL_DIR', 'TMPDIR']
env_vars.extend(conf_env)

env.update(conf_env)
--
2.53.0

[Felix Moessbauer]

From: Ross Burton <ross....@arm.com>

Kas allows SSTATE_DIR to be set via an environment variable, but the
latest oe-core will now warn if hashequiv has been enabled[1], the
sstate is outside the build tree, but the hashequiv database is inside
the build tree. This is because the hashequiv data is needed to make
full use of the sstate objects.

The solution here is to also set the hashserver database location, but
this can't be done in the same way as SSTATE_DIR as we can't set it as
an environment variable. Add BB_HASHSERVE_DB_DIR to the list of
variables that can be set in the environment so that callers can set
them both if needed.

[1] oe-core 491de0db64a ("sanity.bbclass: warn when sstate is outside of
build dir, but hash equiv database is inside it")

[Felix: support kas-container as well, document feature]

Signed-off-by: Ross Burton <ross....@arm.com>

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

docs/userguide/environment-variables.inc | 7 ++++---

kas-container | 3 +++
kas/libkas.py | 3 ++-

3 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/docs/userguide/environment-variables.inc b/docs/userguide/environment-variables.inc
index 0f01d98a6..fecf9f504 100644
--- a/docs/userguide/environment-variables.inc
+++ b/docs/userguide/environment-variables.inc
@@ -130,9 +130,10 @@ overwritten using the ``env`` section of the config file.
| | ``SSH_PRIVATE_KEY_FILE``). |
+--------------------------+--------------------------------------------------+
| ``DL_DIR`` | Environment variables that are transferred to |
-| ``SSTATE_DIR`` | the bitbake environment. The ``DL_DIR`` and |
-| ``SSTATE_MIRRORS`` | ``SSTATE_DIR`` directories are created along |
-| (C,K,E,c) | with their parents, if set. |
+| ``SSTATE_DIR`` | the bitbake environment. The ``DL_DIR``, |
+| ``SSTATE_MIRRORS`` | ``SSTATE_DIR`` and ``BB_HASHSERVE_DB_DIR`` |
+| ``BB_HASHSERVE_DB_DIR`` | directories are created along with their |
+| (C,K,E,c) | parents, if set. |

+--------------------------+--------------------------------------------------+
| ``TMPDIR`` (K,E,c) | Directory for temporary files. |
+--------------------------+--------------------------------------------------+

diff --git a/kas-container b/kas-container
index 75315a8a9..4af3c5bbd 100755
--- a/kas-container
+++ b/kas-container
@@ -242,6 +242,7 @@ setup_kas_dirs()
KAS_REPO_REF_DIR="$(check_and_expand KAS_REPO_REF_DIR required)"
DL_DIR="$(check_and_expand DL_DIR createrec)"
SSTATE_DIR="$(check_and_expand SSTATE_DIR createrec)"
+ BB_HASHSERVE_DB_DIR="$(check_and_expand BB_HASHSERVE_DB_DIR createrec)"
KAS_BUILDTOOLS_DIR="$(check_and_expand KAS_BUILDTOOLS_DIR createrec)"
}

@@ -632,6 +633,7 @@ forward_dir KAS_BUILD_DIR "/build" "rw"
forward_dir DL_DIR "/downloads" "rw"
forward_dir KAS_REPO_REF_DIR "/repo-ref" "rw"
forward_dir SSTATE_DIR "/sstate" "rw"
+forward_dir BB_HASHSERVE_DB_DIR "/bb-hashserve-db" "rw"
forward_dir KAS_BUILDTOOLS_DIR "/buildtools" "rw"

if git_com_dir=$(git -C "${KAS_REPO_DIR}" rev-parse --git-common-dir 2>/dev/null) \
@@ -753,6 +755,7 @@ fi
for var in TERM KAS_DISTRO KAS_MACHINE KAS_TARGET KAS_TASK KAS_CLONE_DEPTH \
KAS_PREMIRRORS DISTRO_APT_PREMIRRORS BB_NUMBER_THREADS PARALLEL_MAKE \
GIT_CREDENTIAL_USEHTTPPATH \
+ BB_HASHSERVE BB_HASHSERVE_UPSTREAM \
TZ; do
if [ -n "$(eval echo \$${var})" ]; then
set -- "$@" -e "${var}=$(eval echo \"\$${var}\")"
diff --git a/kas/libkas.py b/kas/libkas.py
index 58321288c..b9b2d70e7 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -494,7 +494,8 @@ def get_build_environ(build_system, ctx):

conf_env = ctx.config.get_environment()

- env_vars = ['SSTATE_DIR', 'SSTATE_MIRRORS', 'DL_DIR', 'TMPDIR']
+ env_vars = ['SSTATE_DIR', 'SSTATE_MIRRORS', 'BB_HASHSERVE_DB_DIR',
+ 'DL_DIR', 'TMPDIR']
env_vars.extend(conf_env)

env.update(conf_env)
--
2.53.0

[Felix Moessbauer]

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

.github/workflows/next.yml | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/.github/workflows/next.yml b/.github/workflows/next.yml
index 40ed0838e..95c21f370 100644
--- a/.github/workflows/next.yml
+++ b/.github/workflows/next.yml
@@ -107,6 +107,17 @@ jobs:
if [[ "${{ matrix.image-name }}" == "kas" ]]; then
echo "Test oe-core nodistro"
../../kas-container build ../../examples/oe-core-nodistro.yml
+ ../../kas-container cleansstate ../../examples/oe-core-nodistro.yml
+
+ echo "Test oe-core nodistro with upstream hash equivalence server"
+ export BB_HASHSERVE="auto"
+ export BB_HASHSERVE_UPSTREAM="wss://hashserv.yoctoproject.org/ws"
+ export SSTATE_MIRRORS="file://.* https://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
+ ../../kas-container build ../../examples/oe-core-nodistro.yml
+ grep -q $BB_HASHSERVE_UPSTREAM build/tmp/log/eventlog/*.json || {
+ echo "Upstream hash equivalence server not used"
+ exit 1
+ }
fi
- name: Complete build and deploy ${{ matrix.image-name }} image
if: github.ref == 'refs/heads/next'
--
2.53.0

[Felix Moessbauer]

By that, we get improved hash equivalence server support.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

examples/oe-core-nodistro.yml | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/examples/oe-core-nodistro.yml b/examples/oe-core-nodistro.yml
index 1e59430af..7b44755af 100644
--- a/examples/oe-core-nodistro.yml
+++ b/examples/oe-core-nodistro.yml
@@ -39,8 +39,8 @@ target: zlib-native
repos:
bitbake:
url: https://github.com/openembedded/bitbake.git
- tag: yocto-5.2.3
- commit: 710f98844ae30416bdf6a01b655df398b49574ec
+ tag: yocto-6.0.1
+ commit: 22021758e66737bcf68dfd2b74adc6a0cb1d42d9
signed: true
allowed_signers:
- YoctoBuildandRelease
@@ -49,8 +49,8 @@ repos:

core:
url: https://github.com/openembedded/openembedded-core.git
- tag: yocto-5.2.3
- commit: 347cb0861dde58613541ce692778f907943a60ea
+ tag: yocto-6.0.1
+ commit: 06dd66e6220e5ce4ed4b9af4d8231ae5f0a8ce80
signed: true
allowed_signers:
- YoctoBuildandRelease
--
2.53.0

[Ross Burton]

Thanks for finishing the series Felix, other stuff took priority and I entirely forgot about this!

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

[Jan Kiszka]

This hunk seems to belong to patch 2 - I'm moving it there on merge.

Jan

> TZ; do
> if [ -n "$(eval echo \$${var})" ]; then
> set -- "$@" -e "${var}=$(eval echo \"\$${var}\")"
> diff --git a/kas/libkas.py b/kas/libkas.py
> index 58321288c..b9b2d70e7 100644
> --- a/kas/libkas.py
> +++ b/kas/libkas.py
> @@ -494,7 +494,8 @@ def get_build_environ(build_system, ctx):
>
> conf_env = ctx.config.get_environment()
>
> - env_vars = ['SSTATE_DIR', 'SSTATE_MIRRORS', 'DL_DIR', 'TMPDIR']
> + env_vars = ['SSTATE_DIR', 'SSTATE_MIRRORS', 'BB_HASHSERVE_DB_DIR',
> + 'DL_DIR', 'TMPDIR']
> env_vars.extend(conf_env)
>
> env.update(conf_env)


--

Siemens AG, Foundational Technologies
Linux Expert Center

[Jan Kiszka]

On 24.06.26 14:35, 'Felix Moessbauer' via kas-devel wrote:

Thanks, applied with that minor internal shuffling.

Jan