[PATCH 0/1] conditional environment variables

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

Kas requires to set a default value for each defined
environment variable within the 'env' section.
This may not work well with bitbakes weak assignment
feature in some cases.

For example bitbake allows us to use
BB_ENV_EXTRAWHITE (deprecated) or BB_ENV_PASSTHROUGH_ADDITIONS
and if we add a variable VAR to it, bitbake allows forwarding the
variable to the bitbake environment.

A recipe for example using this variable may look like:

```
VAR ?= "default"
```

In case we set VAR in our environment to "foo", VAR will
contain "foo" and if we do not set VAR in our environment
then VAR will contain "default".

With kas this behaviour is not quite possible since we need
to set a default string value for each environment variable,
means the kas configs' default value is assigned instead
of the bitbakes weak assignment.

My proposal now is to support the possibility of assigning nothing
to an env variable like this:

```
env:
VAR:
```

In this case the variable is only added to BB_ENV_PASSTHROUGH_ADDITIONS
but not to the kas environment itself. This opens the door
for explicit forwarding of selected environment variables from
the host environment.

I'm not yet a kas expert, so in case this behaviour is somehow already
achievable without this change, I'm fine and we can skip it.


Benedikt Niedermayr (1):
add conditional environment variables

docs/userguide.rst | 11 ++++++++---
kas/libkas.py | 5 ++++-
kas/schema-kas.json | 2 +-
3 files changed, 13 insertions(+), 5 deletions(-)

--
2.25.1

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

Until now all environment variables in the kas configs
env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---

docs/userguide.rst | 11 ++++++++---
kas/libkas.py | 5 ++++-
kas/schema-kas.json | 2 +-
3 files changed, 13 insertions(+), 5 deletions(-)

diff --git a/docs/userguide.rst b/docs/userguide.rst
index 6c058f1..a8c72ea 100644
--- a/docs/userguide.rst
+++ b/docs/userguide.rst
@@ -354,9 +354,14 @@ Configuration reference
should be specified via the environment variable.

* ``env``: dict [optional]
- Contains environment variable names with the default values. These
- variables are made available to bitbake via ``BB_ENV_EXTRAWHITE`` and can
- be overwritten by the variables of the environment in which kas is started.
+ Contains environment variable names with either default values or None.
+ These variables are made available to bitbake via ``BB_ENV_EXTRAWHITE``
+ and can be overwritten by the variables of the environment in which
+ kas is started.
+ Either a string or nothing (None) can be assigned as value.
+ The former one serves as a default value whereas the latter one will lead
+ to add the variable only to ``BB_ENV_EXTRAWHITE`` and not to the
+ environment where kas is started.

* ``task``: string [optional]
Contains the task to build by bitbake. Can be overwritten by the
diff --git a/kas/libkas.py b/kas/libkas.py
index 4801d32..96ba946 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -275,6 +275,9 @@ def get_build_environ(build_system):
if env_var in os.environ:
env[env_var] = os.environ[env_var]

+ # filter out 'None' values
+ env = {k: v for (k, v) in env.items() if v is not None}
+
return env


@@ -394,7 +397,7 @@ def setup_parser_common_args(parser):

def setup_parser_preserve_env_arg(parser):
parser.add_argument('-E', '--preserve-env',
- help='Keep current user enviornment block',
+ help='Keep current user environment block',
action='store_true')


diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index ca4b513..e42a236 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -94,7 +94,7 @@
"env": {
"type": "object",
"additionalProperties": {
- "type": "string"
+ "type": ["string", "null"]
}
},
"target": {
--
2.25.1

[Henning Schild]

Am Wed, 21 Sep 2022 15:48:19 +0200
schrieb "B. Niedermayr" <benedikt....@siemens.com>:

> From: Benedikt Niedermayr <benedikt....@siemens.com>
>
> Until now all environment variables in the kas configs
> env section require a default value.
> In some cases we want to rely on the weak assignment
> of bitbake instead of relying on the default value of the
> variable in the kas config.
>
> This means if we assign the 'None' value or nothing to the variable
> in the kas config then it is only added to BB_ENV_EXTRAWHITE or
> BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
> variable.

I usually set the variable to "" as default value. But yes it would
become part of env that way, with the default value "".

I wonder if we do have a test case for that stuff, and should extend
it. Or whether we should introduce one that deals with env.

Henning

[Jan Kiszka]

On 21.09.22 16:37, Henning Schild wrote:
> Am Wed, 21 Sep 2022 15:48:19 +0200
> schrieb "B. Niedermayr" <benedikt....@siemens.com>:
>
>> From: Benedikt Niedermayr <benedikt....@siemens.com>
>>
>> Until now all environment variables in the kas configs
>> env section require a default value.
>> In some cases we want to rely on the weak assignment
>> of bitbake instead of relying on the default value of the
>> variable in the kas config.
>>
>> This means if we assign the 'None' value or nothing to the variable
>> in the kas config then it is only added to BB_ENV_EXTRAWHITE or
>> BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
>> variable.
>
> I usually set the variable to "" as default value. But yes it would
> become part of env that way, with the default value "".
>

I can follow the argumentation and would accept the extension. But it
also needs an update of docs/format-changelog.rst.

> I wonder if we do have a test case for that stuff, and should extend
> it. Or whether we should introduce one that deals with env.

That would also be very welcome.

As you have to send a v2 anyway, please factor this typo fix out into a
separate patch.

>> action='store_true')
>>
>>
>> diff --git a/kas/schema-kas.json b/kas/schema-kas.json
>> index ca4b513..e42a236 100644
>> --- a/kas/schema-kas.json
>> +++ b/kas/schema-kas.json
>> @@ -94,7 +94,7 @@
>> "env": {
>> "type": "object",
>> "additionalProperties": {
>> - "type": "string"
>> + "type": ["string", "null"]
>> }
>> },
>> "target": {
>

Thanks,
Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

[Niedermayr, BENEDIKT]

On Fri, 2022-09-30 at 17:52 +0200, Jan Kiszka wrote:
> On 21.09.22 16:37, Henning Schild wrote:
> > Am Wed, 21 Sep 2022 15:48:19 +0200
> > schrieb "B. Niedermayr" <benedikt....@siemens.com>:
> >
> > > From: Benedikt Niedermayr <benedikt....@siemens.com>
> > >
> > > Until now all environment variables in the kas configs
> > > env section require a default value.
> > > In some cases we want to rely on the weak assignment
> > > of bitbake instead of relying on the default value of the
> > > variable in the kas config.
> > >
> > > This means if we assign the 'None' value or nothing to the variable
> > > in the kas config then it is only added to BB_ENV_EXTRAWHITE or
> > > BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
> > > variable.
> >
> > I usually set the variable to "" as default value. But yes it would
> > become part of env that way, with the default value "".
> >
>
> I can follow the argumentation and would accept the extension. But it
> also needs an update of docs/format-changelog.rst.

Does this introduce Version 13 then?

Makes sense.

> > > action='store_true')
> > >
> > >
> > > diff --git a/kas/schema-kas.json b/kas/schema-kas.json
> > > index ca4b513..e42a236 100644
> > > --- a/kas/schema-kas.json
> > > +++ b/kas/schema-kas.json
> > > @@ -94,7 +94,7 @@
> > > "env": {
> > > "type": "object",
> > > "additionalProperties": {
> > > - "type": "string"
> > > + "type": ["string", "null"]
> > > }
> > > },
> > > "target": {
>
> Thanks,
> Jan
>

Cheers,
Benedikt

[Jan Kiszka]

On 21.10.22 13:50, Niedermayr, Benedikt (T CED SES-DE) wrote:
> On Fri, 2022-09-30 at 17:52 +0200, Jan Kiszka wrote:
>> On 21.09.22 16:37, Henning Schild wrote:
>>> Am Wed, 21 Sep 2022 15:48:19 +0200
>>> schrieb "B. Niedermayr" <benedikt....@siemens.com>:
>>>
>>>> From: Benedikt Niedermayr <benedikt....@siemens.com>
>>>>
>>>> Until now all environment variables in the kas configs
>>>> env section require a default value.
>>>> In some cases we want to rely on the weak assignment
>>>> of bitbake instead of relying on the default value of the
>>>> variable in the kas config.
>>>>
>>>> This means if we assign the 'None' value or nothing to the variable
>>>> in the kas config then it is only added to BB_ENV_EXTRAWHITE or
>>>> BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
>>>> variable.
>>>
>>> I usually set the variable to "" as default value. But yes it would
>>> become part of env that way, with the default value "".
>>>
>>
>> I can follow the argumentation and would accept the extension. But it
>> also needs an update of docs/format-changelog.rst.
>
> Does this introduce Version 13 then?

As we didn't have any other update to the format since the last release:
yes.

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

This introduces conditional environment variables as well as test cases for that.
Since 'BB_ENV_EXTRAWHITE' is still around out there, a test case covering that
has also been implemented.

One patch which fixes a little typo is also included.

Benedikt Niedermayr (3):
add conditional environment variables
tests: Add test case for environment variables
libkas: Fix typo

docs/format-changelog.rst | 9 ++++
docs/userguide.rst | 11 ++--
kas/libkas.py | 5 +-
kas/schema-kas.json | 2 +-
tests/test_environment_variables.py | 52 +++++++++++++++++++
tests/test_environment_variables/test_env.yml | 9 ++++
6 files changed, 83 insertions(+), 5 deletions(-)
create mode 100644 tests/test_environment_variables/test_env.yml

--
2.25.1

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

These additional test cases check if the env section
can:

- export variables with their default value
- add variables with 'None' assigned only to
BB_ENV_PASSTHROUGH_ADDITIONS or to the deprecated
BB_ENV_EXTRAWHITE

The BB_ENV_EXTRAWHITE variable is still present, so a test case has also
been added for this.

Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---

tests/test_environment_variables.py | 52 +++++++++++++++++++
tests/test_environment_variables/test_env.yml | 9 ++++

2 files changed, 61 insertions(+)
create mode 100644 tests/test_environment_variables/test_env.yml

diff --git a/tests/test_environment_variables.py b/tests/test_environment_variables.py
index d965f7a..b2318af 100644
--- a/tests/test_environment_variables.py
+++ b/tests/test_environment_variables.py
@@ -1,5 +1,6 @@
import os
import shutil
+import pathlib
from kas import kas


@@ -28,3 +29,54 @@ def test_build_dir_can_be_specified_by_environment_variable(changedir, tmpdir):
del os.environ['KAS_BUILD_DIR']

assert os.path.exists(os.path.join(build_dir, 'conf'))
+
+
+def _test_env_section_export(changedir, tmpdir, bb_env_var):
+ conf_dir = str(tmpdir.mkdir('test_env_variables'))
+ build_dir = str(tmpdir.mkdir('test_build_dir'))
+ env_out = pathlib.Path(build_dir) / 'env_out'
+ init_build_env = pathlib.Path(conf_dir) / 'oe-init-build-env'
+
+ shutil.rmtree(conf_dir, ignore_errors=True)
+ shutil.rmtree(build_dir, ignore_errors=True)
+ shutil.copytree('tests/test_environment_variables', conf_dir)
+ os.chdir(conf_dir)
+
+ # Overwrite oe-init-build-env script
+ # BB_ENV_EXTRAWHITE or BB_ENV_PASSTHROUGH_ADDITIONS is only exported by
+ # kas when it has already been exported in the buildenv script
+ script = """#!/bin/sh
+ export %s="FOO"
+ """ % bb_env_var
+ init_build_env.unlink()
+ init_build_env.write_text(script)
+ init_build_env.chmod(0o775)
+
+ os.environ['KAS_BUILD_DIR'] = build_dir
+ kas.kas(['shell', '-c', f'env >{env_out}', 'test_env.yml'])
+ del os.environ['KAS_BUILD_DIR']
+
+ test_env = {}
+ with env_out.open() as f:
+ for line in f.readlines():
+ key, val = line.split("=", 1)
+ test_env[key] = val.strip()
+
+ # Variables with 'None' assigned should not be added to environment
+ try:
+ _ = test_env["TESTVAR_ONLY_WHITELIST"]
+ assert False
+ except KeyError:
+ assert True
+
+ assert test_env["TESTVAR_DEFAULT_VAL"] == "BAR"
+ assert "TESTVAR_ONLY_WHITELIST" in test_env[bb_env_var]
+
+
+# BB_ENV_EXTRAWHITE is deprecated but may still be used
+def test_env_section_export_bb_extra_white(changedir, tmpdir):
+ _test_env_section_export(changedir, tmpdir, "BB_ENV_EXTRAWHITE")
+
+
+def test_env_section_export_bb_env_passthrough_additions(changedir, tmpdir):
+ _test_env_section_export(changedir, tmpdir, "BB_ENV_PASSTHROUGH_ADDITIONS")
diff --git a/tests/test_environment_variables/test_env.yml b/tests/test_environment_variables/test_env.yml
new file mode 100644
index 0000000..6ab9b41
--- /dev/null
+++ b/tests/test_environment_variables/test_env.yml
@@ -0,0 +1,9 @@
+header:
+ version: 13
+
+env:
+ TESTVAR_DEFAULT_VAL: "BAR"
+ TESTVAR_ONLY_WHITELIST:
+
+repos:
+ this:
--
2.25.1

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

Until now all environment variables in the kas config's

env section require a default value.
In some cases we want to rely on the weak assignment
of bitbake instead of relying on the default value of the
variable in the kas config.

This means if we assign the 'None' value or nothing to the variable
in the kas config then it is only added to BB_ENV_EXTRAWHITE or
BB_ENV_PASSTHROUGH_ADDITIONS without setting it as environment
variable.

Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---

docs/format-changelog.rst | 9 +++++++++
docs/userguide.rst | 11 ++++++++---
kas/libkas.py | 3 +++
kas/schema-kas.json | 2 +-
4 files changed, 21 insertions(+), 4 deletions(-)

diff --git a/docs/format-changelog.rst b/docs/format-changelog.rst
index c8a7218..edaf7b5 100644
--- a/docs/format-changelog.rst
+++ b/docs/format-changelog.rst
@@ -121,3 +121,12 @@ Added
- For repositories, ``url`` and ``path`` can now be overridden with a
null-value to switch between version-controlled repositories and unversioned
local folders.
+
+Version 13
+----------
+
+Added
+~~~~~
+
+- Variables used in the ``env`` section can now be assigned 'None' as value. In this
+ case the variables are only exported to the bb env whitelist.

diff --git a/docs/userguide.rst b/docs/userguide.rst
index 6c058f1..a8c72ea 100644
--- a/docs/userguide.rst
+++ b/docs/userguide.rst
@@ -354,9 +354,14 @@ Configuration reference
should be specified via the environment variable.

* ``env``: dict [optional]
- Contains environment variable names with the default values. These
- variables are made available to bitbake via ``BB_ENV_EXTRAWHITE`` and can
- be overwritten by the variables of the environment in which kas is started.
+ Contains environment variable names with either default values or None.
+ These variables are made available to bitbake via ``BB_ENV_EXTRAWHITE``
+ and can be overwritten by the variables of the environment in which
+ kas is started.
+ Either a string or nothing (None) can be assigned as value.
+ The former one serves as a default value whereas the latter one will lead
+ to add the variable only to ``BB_ENV_EXTRAWHITE`` and not to the
+ environment where kas is started.

* ``task``: string [optional]
Contains the task to build by bitbake. Can be overwritten by the
diff --git a/kas/libkas.py b/kas/libkas.py

index 4801d32..3cf2249 100644

--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -275,6 +275,9 @@ def get_build_environ(build_system):
if env_var in os.environ:
env[env_var] = os.environ[env_var]

+ # filter out 'None' values
+ env = {k: v for (k, v) in env.items() if v is not None}
+
return env

diff --git a/kas/schema-kas.json b/kas/schema-kas.json
index ca4b513..e42a236 100644
--- a/kas/schema-kas.json
+++ b/kas/schema-kas.json
@@ -94,7 +94,7 @@
"env": {
"type": "object",
"additionalProperties": {
- "type": "string"
+ "type": ["string", "null"]
}
},
"target": {

--
2.25.1

[B. Niedermayr]

From: Benedikt Niedermayr <benedikt....@siemens.com>

Signed-off-by: Benedikt Niedermayr <benedikt....@siemens.com>
---

kas/libkas.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/kas/libkas.py b/kas/libkas.py
index 3cf2249..96ba946 100644
--- a/kas/libkas.py
+++ b/kas/libkas.py
@@ -397,7 +397,7 @@ def setup_parser_common_args(parser):

def setup_parser_preserve_env_arg(parser):
parser.add_argument('-E', '--preserve-env',
- help='Keep current user enviornment block',
+ help='Keep current user environment block',

action='store_true')


--
2.25.1

[Jan Kiszka]

Shouldn't we also test if a value set on the env will make it into bitbake?

Jan

> +
> +# BB_ENV_EXTRAWHITE is deprecated but may still be used
> +def test_env_section_export_bb_extra_white(changedir, tmpdir):
> + _test_env_section_export(changedir, tmpdir, "BB_ENV_EXTRAWHITE")
> +
> +
> +def test_env_section_export_bb_env_passthrough_additions(changedir, tmpdir):
> + _test_env_section_export(changedir, tmpdir, "BB_ENV_PASSTHROUGH_ADDITIONS")
> diff --git a/tests/test_environment_variables/test_env.yml b/tests/test_environment_variables/test_env.yml
> new file mode 100644
> index 0000000..6ab9b41
> --- /dev/null
> +++ b/tests/test_environment_variables/test_env.yml
> @@ -0,0 +1,9 @@
> +header:
> + version: 13
> +
> +env:
> + TESTVAR_DEFAULT_VAL: "BAR"
> + TESTVAR_ONLY_WHITELIST:
> +
> +repos:
> + this:

--

[Niedermayr, BENEDIKT]

Currently we are not testing bitbake directly in any testcase, or?

On the one hand testing bitbake directly makes sense, since some features are completely tested then,
one the other hand I'm a bit afraid of it. If we are introducing bitbake tests we also need to test different versions
of bitbake (e.g. BB_ENV_EXTRAWHITE vs. BB_ENV_PASSTHROUGH_ADDITIONS). If we don't end up with a bunch of different bitbake
versions im completely fine with that.

> Jan
>
> > +
> > +# BB_ENV_EXTRAWHITE is deprecated but may still be used
> > +def test_env_section_export_bb_extra_white(changedir, tmpdir):
> > + _test_env_section_export(changedir, tmpdir, "BB_ENV_EXTRAWHITE")
> > +
> > +
> > +def test_env_section_export_bb_env_passthrough_additions(changedir, tmpdir):
> > + _test_env_section_export(changedir, tmpdir, "BB_ENV_PASSTHROUGH_ADDITIONS")
> > diff --git a/tests/test_environment_variables/test_env.yml b/tests/test_environment_variables/test_env.yml
> > new file mode 100644
> > index 0000000..6ab9b41
> > --- /dev/null
> > +++ b/tests/test_environment_variables/test_env.yml
> > @@ -0,0 +1,9 @@
> > +header:
> > + version: 13
> > +
> > +env:
> > + TESTVAR_DEFAULT_VAL: "BAR"
> > + TESTVAR_ONLY_WHITELIST:
> > +
> > +repos:
> > + this:

Cheers,
Benedikt

[Jan Kiszka]

I didn't check the setup for this test in details but we are even
building things in others. It should be doable to have at least a
minimal config that permits calling bitbake -e, checking the outcome
this way.

Jan