Setting BB_NUMBER_THREADS via environment

[Ross Burton]

I'm trying to remove some hardcoded assumptions from our kas files that are used for CI.  Specifically we have BB_NUMBER_THREADS and PARALLEL_MAKE in our local_conf_header as on our CI machine the default values that BitBake provides are not suitable.  However this is a huge assumption that doesn't apply for everyone, so my plan was to put BB_NUMBER_THREADS/PARALLEL_MAKE in the env block and let the caller potentially override them, for example in the CI environment.

But I can't see a way of saying "only set this variable if set". By which I meant I need to give env items a value:

env:

- BB_NUMBER_THREADS: ""

But this is now the default and overrides the actual default in Bitbake, so if the variable isn't set then:

$ kas shell -c 'bitbake -e |grep BB_NUMBER_THREADS='

BB_NUMBER_THREADS=""

That's not going to work.

So, is there a way to say that a variable should be passed through if set but not to set a default value if it isn't set?  I also can't set the default in the kas file to the BitBake default as it is expanded too early:

$ kas shell -c 'bitbake -e |grep BB_NUMBER_THREADS='

bb.data_smart.ExpansionError: Failure expanding variable PARALLEL_MAKE, expression was -j ${@oe.utils.cpu_count()} which triggered exception NameError: name 'oe' is not defined

Any ideas?

Ross

[Jan Kiszka]

As this is a standard bitbake variable, we could consider white-listing
this one as well (like SSTATE_DIR, DL_DIR etc.).

Or we permit

env:
BB_NUMBER_THREADS:

i.e. an empty default. I'm not even sure if that was possible before we
introduce the schema. Maybe the code would do the right thing (it
already checks if the var is in the environment).

Jan

--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[Ross Burton]

On Wed, 28 Apr 2021 at 13:01, Jan Kiszka <jan.k...@siemens.com> wrote:
> Or we permit
>
> env:
> BB_NUMBER_THREADS:
>
> i.e. an empty default. I'm not even sure if that was possible before we
> introduce the schema. Maybe the code would do the right thing (it
> already checks if the var is in the environment).

Just BB_NUMBER_THREADS: isn't valid YAML, but ~ or null are parsed as
None, the schema can be told to allow null, and then the environment
function can filter out variables with no value set.

I tried to post a patch but I think its in the moderation queue.
Something like this works:

diff --git a/kas/config.py b/kas/config.py
index 0119f68..e5cb1a3 100644
--- a/kas/config.py
+++ b/kas/config.py
@@ -163,8 +163,13 @@ class Config:
Returns the configured environment variables from the configuration
file with possible overwritten values from the environment.
"""
+ # Defaults from the configuration
env = self._config.get('env', {})
- return {var: os.environ.get(var, env[var]) for var in env}
+ # Update from the environment
+ env = {var: os.environ.get(var, env[var]) for var in env}
+ # Remove entries with no value set
+ env = {k:v for k,v in env.items() if v is not None}
+ return env

def get_multiconfig(self):
"""
diff --git a/kas/configschema.py b/kas/configschema.py
index b97ef42..e9867d1 100644
--- a/kas/configschema.py
+++ b/kas/configschema.py
@@ -109,7 +109,14 @@ CONFIGSCHEMA = {
'env': {
'type': 'object',
'additionalProperties': {
- 'type': 'string',
+ 'oneOf': [
+ {
+ 'type': 'null',
+ },
+ {
+ 'type': 'string',
+ },
+ ],
},
},
'target': {

Ross

[Jan Kiszka]

On 28.04.21 18:09, Ross Burton wrote:
> On Wed, 28 Apr 2021 at 13:01, Jan Kiszka <jan.k...@siemens.com> wrote:
>> Or we permit
>>
>> env:
>> BB_NUMBER_THREADS:
>>
>> i.e. an empty default. I'm not even sure if that was possible before we
>> introduce the schema. Maybe the code would do the right thing (it
>> already checks if the var is in the environment).
>
> Just BB_NUMBER_THREADS: isn't valid YAML, but ~ or null are parsed as
> None, the schema can be told to allow null, and then the environment
> function can filter out variables with no value set.

We already support (and validate)

...
layers:
meta:

so this should work as well.

>
> I tried to post a patch but I think its in the moderation queue.

Nope, it made to the list.

Jan