[PATCH] kas-container: auto-forward declared environment variables

[Cedric Hombourger]

kas files may contain environment variables and their default values.
The env: block instructs bitbake to take values of the specified
variables from the environment and expand them in recipes and/or
classes. When using kas-container, kas is effectively running in a
clean environment (as intended!): each variable must be explicitly
passed from the host environment to the container environment. This
is already done for a number of "well-known" variables. Extend this
to also forward environment variables listed in first-level kas files
(those included by the generated .config.yaml file or listed on the
kas-container command line plus any included file from the top-level
repository).

Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
---
docs/userguide/kas-container-description.inc | 5 ++++
kas-container | 29 +++++++++++++++++++-
2 files changed, 33 insertions(+), 1 deletion(-)

diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
index 7f35013..7898bf6 100644
--- a/docs/userguide/kas-container-description.inc
+++ b/docs/userguide/kas-container-description.inc
@@ -4,6 +4,11 @@ decouples the build environment from the host system. For details, see
:ref:`env-vars-label`. The wrapper also takes care of mounting the necessary
directories and setting up the environment variables inside the container.

+If the ``yq`` command is available, it will make a best-effort to determinei
+environment variables listed under `env:` from top-level files to forward them
+into the container. Additional environment variables may be explicitly passed
+using ``--runtime-args "-e VAR=VALUE"`.
+
.. note::
The ``kas-container`` script has limited support for Git worktrees. Regular
Git operations on the checked-out repository are supported. However, executing
diff --git a/kas-container b/kas-container
index 4289749..8f1f696 100755
--- a/kas-container
+++ b/kas-container
@@ -244,6 +244,25 @@ process_file_arg()
KAS_REPO_DIRS="${KAS_REPO_DIRS} ${_KAS_REPO_DIR}"
}

+list_includes()
+{
+ yq -r '(.header.includes // []) | map(if . | type == "string" then . else .file end) | join("\n")' "${1}"
+}
+
+list_env_vars()
+{
+ for FILE in "${@}"; do
+ [ -f "${FILE}" ] || continue
+ yq -r '(.env // {}) | keys | join("\n")' "${FILE}"
+
+ # SC2046: Quote this to prevent word splitting.
+ # shellcheck disable=2046
+ # SC2086: Double quote to prevent globbing and word splitting.
+ # shellcheck disable=2086
+ list_env_vars $(list_includes "${FILE}")
+ done
+}
+
# Params: NAME CONTAINER_PATH MODE
# If the dir is not below KAS_WORK_DIR, the dir is mounted into the container.
forward_dir()
@@ -726,10 +745,18 @@ if command -v timedatectl >/dev/null; then
set -- "$@" -e "KAS_HOST_TZ=$(timedatectl show -p Timezone --value 2>/dev/null)"
fi

+# propagate environment variables listed in our first-level files if we have yq
+KAS_EXTRA_VARS=""
+if command -v yq >/dev/null; then
+ # SC2086: Double quote to prevent globbing and word splitting.
+ # shellcheck disable=2086
+ KAS_EXTRA_VARS=$(list_env_vars ${KAS_FIRST_FILES} | sort -u)
+fi
+
for var in TERM KAS_DISTRO KAS_MACHINE KAS_TARGET KAS_TASK KAS_CLONE_DEPTH \
KAS_PREMIRRORS DISTRO_APT_PREMIRRORS BB_NUMBER_THREADS PARALLEL_MAKE \
GIT_CREDENTIAL_USEHTTPPATH \
- TZ; do
+ TZ ${KAS_EXTRA_VARS}; do
if [ -n "$(eval echo \$${var})" ]; then
set -- "$@" -e "${var}=$(eval echo \"\$${var}\")"
fi
--
2.47.3

[Florian Bezdeka]

On Tue, 2025-11-25 at 16:11 +0100, 'Cedric Hombourger' via kas-devel
wrote:

> kas files may contain environment variables and their default values.
> The env: block instructs bitbake to take values of the specified
> variables from the environment and expand them in recipes and/or
> classes. When using kas-container, kas is effectively running in a
> clean environment (as intended!): each variable must be explicitly
> passed from the host environment to the container environment. This
> is already done for a number of "well-known" variables. Extend this
> to also forward environment variables listed in first-level kas files
> (those included by the generated .config.yaml file or listed on the
> kas-container command line plus any included file from the top-level
> repository).
>
> Signed-off-by: Cedric Hombourger <cedric.h...@siemens.com>
> ---
> docs/userguide/kas-container-description.inc | 5 ++++
> kas-container | 29 +++++++++++++++++++-
> 2 files changed, 33 insertions(+), 1 deletion(-)
>
> diff --git a/docs/userguide/kas-container-description.inc b/docs/userguide/kas-container-description.inc
> index 7f35013..7898bf6 100644
> --- a/docs/userguide/kas-container-description.inc
> +++ b/docs/userguide/kas-container-description.inc
> @@ -4,6 +4,11 @@ decouples the build environment from the host system. For details, see
> :ref:`env-vars-label`. The wrapper also takes care of mounting the necessary
> directories and setting up the environment variables inside the container.
>
> +If the ``yq`` command is available, it will make a best-effort to determinei

s/determinei/determine

> +environment variables listed under `env:` from top-level files to forward them
> +into the container. Additional environment variables may be explicitly passed
> +using ``--runtime-args "-e VAR=VALUE"`.
> +

Assuming two systems: one with yp installed on the host, one without.
The build result might be very different without any warning, right?

Note to the debian maintainers: This comes with an additional (optional)
dependency to yp. Maybe we should handle that as "real" dependency to
get at least Debian systems consistent.

> --
> You received this message because you are subscribed to the Google Groups "kas-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to kas-devel+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/kas-devel/20251125151100.1527838-1-cedric.hombourger%40siemens.com.

[cedric.h...@siemens.com]

thanks

>
> > +environment variables listed under `env:` from top-level files to
> > forward them
> > +into the container. Additional environment variables may be
> > explicitly passed
> > +using ``--runtime-args "-e VAR=VALUE"`.
> > +
>
> Assuming two systems: one with yp installed on the host, one without.
> The build result might be very different without any warning, right?

yes I am afraid and so could values. Would you suggest we make yq a
hard-requirement?

--
Cedric Hombourger
Siemens AG
http://www.siemens.com/

[Jan Kiszka]

You are not handling the full inclusion and override chain this way. If
things in kas were that simple, kas/includehandler.py would look fairly
different. :)

So, this would create an inconsistency and was therefore never
considered for kas-container. We can really only handle top-level config
file keys there without causing a mess.

Jan

Jan


--
Siemens AG, Foundational Technologies
Linux Expert Center

[cedric.h...@siemens.com]

Yes it was noted to be a best-effort :)

> If
> things in kas were that simple, kas/includehandler.py would look
> fairly
> different. :)

Exactly and it would be very hard to match includehandler.py

>
> So, this would create an inconsistency and was therefore never
> considered for kas-container. We can really only handle top-level
> config
> file keys there without causing a mess.

Sure thing.

We seem to have several ways to enter into the container

1) "menu" - the input file is then Kconfig and we would not be able to
determine environment variables to be auto-forwarded (building from the
menu could fail)

2) "build" or "shell" from a configured project. The input file is then
.config.yaml. Should we also forget about determining includes from
that file? We would also not auto-forward any variables

3) "build" or "shell" with an explicit list of .yml file. Per your
suggestion, we could extract environment variables listed there.

Neither the current situation (where --runtime-args "-e VAR=VALUE"
needs to be used if .yml have env: variables) or possibly future
situation are ideal (some level of auto-forwarding).

Talking from the perspective of the user-experience (people may be
surprised that kas and kas-container behave differently with respect to
environment variables). We may need to look at the documentation of the
env: section to warn kas-container users.

Your guidance would be much appreciated!

Cedric Hombourger
Siemens AG
www.siemens.com

[Florian Bezdeka]

I don't have an clear idea how to handle that.

I just want to avoid that someone is searching for hours why his/her
build result is completely different while using a completely identical
kas-container cmdline.

I mean this is what we promise: Invoke kas-container and everything is
reproducible as everything happens within a clean environment. IMHO this
should hold...

Jan had some comments about the "faked" include handling below. Crazy
idea - maybe I shouldn't even post that...: Could we invoke kas (running
inside the container) upfront, so that it could do the proper parsing
for us? Could we re-inject the outcome into a second run then?

OTOH, maybe it's better to minimize all the dependencies that are
injected by the environment. Which brings me to a couple of questions:

- env variables are defined within the kas files already, so they will
be taken into account by kas. Why do we have to forward them from the
host to the container? Values of the host overwrite some defaults from
the kas files?

- What is the concrete use case behind that patch? Which kind of
variables need to be defined in those kas files? Is that really the
right place to define those?

[Jan Kiszka]

...which is primarily about inclusion of files that are step-wise
checked out while processing other files. On top comes merging /
overwriting of keys.

>>
>> So, this would create an inconsistency and was therefore never
>> considered for kas-container. We can really only handle top-level
>> config
>> file keys there without causing a mess.
>
> Sure thing.
>
> We seem to have several ways to enter into the container
>
> 1) "menu" - the input file is then Kconfig and we would not be able to
> determine environment variables to be auto-forwarded (building from the
> menu could fail)

menu is not a build entrypoint itself. If you select "build" there, the
actual build plugin will be started on exit.

>
> 2) "build" or "shell" from a configured project. The input file is then
> .config.yaml. Should we also forget about determining includes from
> that file? We would also not auto-forward any variables
>
> 3) "build" or "shell" with an explicit list of .yml file. Per your
> suggestion, we could extract environment variables listed there.
>
> Neither the current situation (where --runtime-args "-e VAR=VALUE"
> needs to be used if .yml have env: variables) or possibly future
> situation are ideal (some level of auto-forwarding).
>
> Talking from the perspective of the user-experience (people may be
> surprised that kas and kas-container behave differently with respect to
> environment variables). We may need to look at the documentation of the
> env: section to warn kas-container users.
>

https://kas.readthedocs.io/en/latest/command-line.html#environment-variables,
"Note".

> Your guidance would be much appreciated!
>

The situation is not optimal but also not really improvable. We clearly
state that kas-container requires explicit forward if one actually wants
env vars in the build. We cannot automate that up to the level of kas
native, so whatever we try to automate will only create another special
case between explicit forwarding with kas-container and automatic
forwarding with kas.

Actually, I'm not a big fan of random env vars influencing the build as
it is too easy to construct non-reproducible setups this way. A primary
use case for such vars were so far credential handling, but that we
should have covered fairly well by now via handling of well-known vars
without env:, and that also via kas-container. So, what are the exact
use cases for which you would like to see configurable env: vars?

[cedric.h...@siemens.com]

I should have started with the "why"

This is mostly for authentication tokens; in other words, things that
should be kept outside of the layer.

Isar has gained support for ISAR_APT_CREDS for credentials to be placed
in /etc/apt/auth.d/ of the build environment.

I have been using explicit -e options to pass them around and it is not
great from a user experience (long command line) but also from a
security aspect (ps aux on my build machine would reveal my token). My
initial proposal only improved the user-experience (well kind of) but
did change anything with respect to security.

I could use some ideas?

Should we recommend people to create a kas file in their project tree
and have that file listed in .gitignore to avoid accidental checkins?

>
> Jan
>

--
Cedric Hombourger
Siemens AG

http://www.siemens.com/

[Jan Kiszka]

I didn't follow that in details - this is another layer of cred passing,
besides standard .netrc? Can't we derive the apt auth from .netrc?

>
> I have been using explicit -e options to pass them around and it is not
> great from a user experience (long command line) but also from a
> security aspect (ps aux on my build machine would reveal my token). My
> initial proposal only improved the user-experience (well kind of) but
> did change anything with respect to security.
>
> I could use some ideas?
>
> Should we recommend people to create a kas file in their project tree
> and have that file listed in .gitignore to avoid accidental checkins?

Credential should come from env vars (even more in CI) or (protected,
out-of-tree) standard conf files that are explicitly forwarded.

If we need the new ISAR_APT_CREDS variable in the end, then we should
make kas-container aware of it. Better would be reusing existing channels.