[PATCH] lists.bbclass,bitbake.conf: use features lists

[chris....@siemens.com]

From: Christopher Larson <ker...@gmail.com>

The intention behind this commit is to ease and encourage the use of Yocto-style
features variables, beyond our current usage:

- Add a bbclass to ease the handling of list variables in general
- Add default values for the features variables
- Add the features variables to the list variables
- Add a combined features variable

The intention is that a downstream layer will use `bb.utils.contains` or
`bb.utils.contains_any` to enable or disable functionality based on the presence
of defined features, rather than adding new variables in each case.

Signed-off-by: Christopher Larson <ker...@gmail.com>
---
meta/classes/lists.bbclass | 97 ++++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 19 ++++++++
2 files changed, 116 insertions(+)
create mode 100644 meta/classes/lists.bbclass

diff --git a/meta/classes/lists.bbclass b/meta/classes/lists.bbclass
new file mode 100644
index 00000000..c872f4bd
--- /dev/null
+++ b/meta/classes/lists.bbclass
@@ -0,0 +1,97 @@
+# Functions to improve the functionality of bitbake list variables.
+#
+# - Add the ability to remove items from a list variable without using :remove.
+# - Add the ability for a list item to imply the addition of other list items.
+#
+
+# Usage requires either adding the variable name to LIST_VARIABLES, or manually
+# adding a :remove and a :prepend to each fully supported list variable.
+#
+# To remove items from a configured list, simply append the item to be removed
+# to the variable with a '-' or '~' prefix. For example, to remove 'alpha' from
+# IMAGE_FEATURES, add '-alpha' to IMAGE_FEATURES.
+#
+# To support implied list items, create a mapping of items to be appended to
+# the variable when a specific item is present. For example, to append 'beta'
+# to IMAGE_FEATURES when 'alpha' is present, configure IMAGE_FEATURES as such,
+# then set IMPLIED_IMAGE_FEATURES[alpha] = "beta".
+#
+# Boilerplate example:
+#
+# # Either this:
+# LIST_VARIABLES += "IMAGE_FEATURES"
+#
+# # Or this:
+# IMAGE_FEATURES:remove = "${@remove_prefixed_items('IMAGE_FEATURES', d)}"
+# IMAGE_FEATURES:prepend = "${@add_implied_items('IMAGE_FEATURES', 'IMPLIED_IMAGE_FEATURES', d)} "
+#
+# Usage example:
+#
+# # IMAGE_FEATURES will be "beta alpha" if the following configuration is used:
+# IMPLIED_IMAGE_FEATURES[alpha] = "beta"
+# IMAGE_FEATURES += "alpha"
+#
+# # IMAGE_FEATURES will be "first" if the following configuration is used:
+# IMAGE_FEATURES = "first second"
+# IMAGE_FEATURES += "-second"
+
+
+def remove_prefixed_items(var, d):
+ """Return the items to be removed from var with :remove.
+
+ This function is intended to be used in a :remove handler to remove
+ items from a variable. It will interpret items prefixed with a '-'
+ or '~' as items to be removed.
+ """
+ # Use a flag to avoid infinite recursion.
+ if d.getVarFlag(var, 'remove_prefixed_items_internal') == '1':
+ return ''
+
+ from collections import Counter
+
+ d.setVarFlag(var, 'remove_prefixed_items_internal', '1')
+ try:
+ value = d.getVar(var)
+ counter = Counter()
+ for v in value.split():
+ if v.startswith('-') or v.startswith('~'):
+ counter[v[1:]] -= 1
+ counter[v] -= 1
+ else:
+ counter[v] += 1
+ return ' '.join(v for v, c in counter.items() if c < 1)
+ finally:
+ d.delVarFlag(var, 'remove_prefixed_items_internal')
+
+
+def add_implied_items(var, implied_var, d):
+ """Return the items to be appended due to the presence of other items in var.
+
+ This function is intended to be used in a :append handler to append
+ items from a variable. It will rely on the supplied mapping of implied items
+ to append the corresponding items.
+ """
+ # Use a flag to avoid infinite recursion.
+ if d.getVarFlag(var, 'add_implied_items_internal') == '1':
+ return ''
+
+ def implied_items(item, implied_mapping, d, seen=None):
+ """Return the implied items for a given item."""
+ if seen is None:
+ seen = set()
+ if item in seen:
+ return ''
+ seen.add(item)
+ implied = implied_mapping.get(item, '').split()
+ return ' '.join(implied + [implied_items(f, implied_mapping, d, seen) for f in implied])
+
+ d.setVarFlag(var, 'add_implied_items_internal', '1')
+ try:
+ value = d.getVar(var)
+ implied_mapping = d.getVarFlags(implied_var)
+ if implied_mapping is None:
+ return ''
+
+ return ' '.join(implied_items(f, implied_mapping, d) for f in value.split())
+ finally:
+ d.delVarFlag(var, 'add_implied_items_internal')
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index ef408faa..5ab8ced7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -175,6 +175,25 @@ BBINCLUDELOGS ??= "yes"
# Add event handlers for bitbake
INHERIT += "isar-events sstate"

+# Make features variables available
+INHERIT += "lists"
+
+LIST_VARIABLES += "BASE_REPO_FEATURES MACHINE_FEATURES DISTRO_FEATURES ROOTFS_FEATURES"
+
+BASE_REPO_FEATURES ??= ""
+BASE_REPO_FEATURES[doc] = "Specifies the list of features for the base-apt repository."
+
+MACHINE_FEATURES ??= ""
+MACHINE_FEATURES[doc] = "Specifies the list of hardware features the MACHINE is capable of supporting."
+
+DISTRO_FEATURES ??= ""
+DISTRO_FEATURES[doc] = "The software support you want in your distribution for various features."
+
+COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}"
+
+ROOTFS_FEATURES ??= ""
+ROOTFS_FEATURES[doc] = "The list of features to be included in a root filesystem. Typically, you configure this variable in an image recipe or class."
+
# Buildstats requires IMAGE_ROOTFS to be always defined
IMAGE_ROOTFS ??= "${WORKDIR}/rootfs"
INHERIT += "${@'buildstats' if bb.utils.to_boolean(d.getVar('USE_BUILDSTATS')) else ''}"
--
2.47.1.windows.1

[Larson, Chris]

Ignore, this, I'll follow-up. Accidentally missed the LIST_VARIABLES handling somehow.
--
Christopher Larson
Siemens AG
http://www.siemens.com/

> +manually # adding a :remove and a :prepend to each fully supported list variable.

> +#
> +# To remove items from a configured list, simply append the item to be

> +removed # to the variable with a '-' or '~' prefix. For example, to
> +remove 'alpha' from # IMAGE_FEATURES, add '-alpha' to IMAGE_FEATURES.

> +#
> +# To support implied list items, create a mapping of items to be

> +appended to # the variable when a specific item is present. For example, to

> append 'beta'
> +# to IMAGE_FEATURES when 'alpha' is present, configure IMAGE_FEATURES

> +as such, # then set IMPLIED_IMAGE_FEATURES[alpha] = "beta".

> + seen) for f in implied])

> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> isar-users+...@googlegroups.com.
> To view this discussion visit
> https://groups.goog/
> le.com%2Fd%2Fmsgid%2Fisar-users%2F20241216201602.619-1-
> chris.larson%2540siemens.com&data=05%7C02%7Cchris.larson%40siemens.co
> m%7Cb93bac82bcdf48d0efb708dd1e0e8b96%7C38ae3bcd95794fd4addab42e14
> 95d55a%7C1%7C0%7C638699770030857193%7CUnknown%7CTWFpbGZsb3d
> 8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjo
> iTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=TYJgrABE1gVPkjCAN
> QGPp%2B9yYpHdVmqXbJLlmA%2BFPZU%3D&reserved=0.

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

The intention behind this commit is to ease and encourage the use of Yocto-style
features variables, beyond our current usage:

- Add a bbclass to ease the handling of list variables in general
- Add default values for the features variables
- Add the features variables to the list variables
- Add a combined features variable

The intention is that a downstream layer will use `bb.utils.contains` or
`bb.utils.contains_any` to enable or disable functionality based on the presence
of defined features, rather than adding new variables in each case.

Signed-off-by: Christopher Larson <chris....@siemens.com>
---
meta/classes/lists.bbclass | 105 +++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 19 +++++++
2 files changed, 124 insertions(+)
create mode 100644 meta/classes/lists.bbclass

v2 changes:
- Corrected email address.
- Added missing LIST_VARIABLES event handler
- Changed the examples to the more appropriate ROOTFS_FEATURES

diff --git a/meta/classes/lists.bbclass b/meta/classes/lists.bbclass
new file mode 100644

index 00000000..db8f5837
--- /dev/null
+++ b/meta/classes/lists.bbclass
@@ -0,0 +1,105 @@

+# Functions to improve the functionality of bitbake list variables.
+#
+# - Add the ability to remove items from a list variable without using :remove.
+# - Add the ability for a list item to imply the addition of other list items.
+#
+
+# Usage requires either adding the variable name to LIST_VARIABLES, or manually
+# adding a :remove and a :prepend to each fully supported list variable.
+#
+# To remove items from a configured list, simply append the item to be removed
+# to the variable with a '-' or '~' prefix. For example, to remove 'alpha' from

+# ROOTFS_FEATURES, add '-alpha' to ROOTFS_FEATURES.

+#
+# To support implied list items, create a mapping of items to be appended to
+# the variable when a specific item is present. For example, to append 'beta'

+# to ROOTFS_FEATURES when 'alpha' is present, configure ROOTFS_FEATURES as such,
+# then set IMPLIED_ROOTFS_FEATURES[alpha] = "beta".

+#
+# Boilerplate example:
+#
+# # Either this:

+# LIST_VARIABLES += "ROOTFS_FEATURES"

+#
+# # Or this:

+# ROOTFS_FEATURES:remove = "${@remove_prefixed_items('ROOTFS_FEATURES', d)}"
+# ROOTFS_FEATURES:prepend = "${@add_implied_items('ROOTFS_FEATURES', 'IMPLIED_ROOTFS_FEATURES', d)} "

+#
+# Usage example:
+#

+# # ROOTFS_FEATURES will be "beta alpha" if the following configuration is used:
+# IMPLIED_ROOTFS_FEATURES[alpha] = "beta"
+# ROOTFS_FEATURES += "alpha"
+#
+# # ROOTFS_FEATURES will be "first" if the following configuration is used:
+# ROOTFS_FEATURES = "first second"
+# ROOTFS_FEATURES += "-second"
+
+python enable_list_variables() {
+ """Enable list variable functionality."""
+ for variable in d.getVar("LIST_VARIABLES").split():
+ d.setVar(variable + ':remove', ' ${@remove_prefixed_items("%s", d)}' % variable)
+ d.setVar(variable + ':prepend', '${@add_implied_items("%s", "IMPLIED_%s", d)} ' % (variable, variable))
+}
+enable_list_variables[eventmask] = "bb.event.ConfigParsed"
+addhandler enable_list_variables

index cda98035..9f3b8a4e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -172,6 +172,25 @@ BBINCLUDELOGS ??= "yes"

[Uladzimir Bely]

Hello.

Could you please resend properly formatted [PATCH v2]? Current one is
broken due to multiple extra empty lines and is not recognized as a
patch.

--
Best regards,
Uladzimir.

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

The intention behind this commit is to ease and encourage the use of Yocto-style
features variables, beyond our current usage:

- Add a bbclass to ease the handling of list variables in general
- Add default values for the features variables
- Add the features variables to the list variables
- Add a combined features variable

The intention is that a downstream layer will use `bb.utils.contains` or
`bb.utils.contains_any` to enable or disable functionality based on the presence
of defined features, rather than adding new variables in each case.

Signed-off-by: Christopher Larson <chris....@siemens.com>
---
meta/classes/lists.bbclass | 105 +++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 19 +++++++

2 files changed, 124 insertions(+)
create mode 100644 meta/classes/lists.bbclass

v2 changes:
- Corrected email address.
- Added missing LIST_VARIABLES event handler
- Changed the examples to the more appropriate ROOTFS_FEATURES

diff --git a/meta/classes/lists.bbclass b/meta/classes/lists.bbclass
new file mode 100644

index 00000000..db8f5837
--- /dev/null
+++ b/meta/classes/lists.bbclass
@@ -0,0 +1,105 @@

+# Functions to improve the functionality of bitbake list variables.
+#
+# - Add the ability to remove items from a list variable without using :remove.
+# - Add the ability for a list item to imply the addition of other list items.
+#
+
+# Usage requires either adding the variable name to LIST_VARIABLES, or manually
+# adding a :remove and a :prepend to each fully supported list variable.
+#
+# To remove items from a configured list, simply append the item to be removed
+# to the variable with a '-' or '~' prefix. For example, to remove 'alpha' from

+# ROOTFS_FEATURES, add '-alpha' to ROOTFS_FEATURES.

+#
+# To support implied list items, create a mapping of items to be appended to
+# the variable when a specific item is present. For example, to append 'beta'

+# to ROOTFS_FEATURES when 'alpha' is present, configure ROOTFS_FEATURES as such,
+# then set IMPLIED_ROOTFS_FEATURES[alpha] = "beta".

+#
+# Boilerplate example:
+#
+# # Either this:

+# LIST_VARIABLES += "ROOTFS_FEATURES"

+#
+# # Or this:

+# ROOTFS_FEATURES:remove = "${@remove_prefixed_items('ROOTFS_FEATURES', d)}"
+# ROOTFS_FEATURES:prepend = "${@add_implied_items('ROOTFS_FEATURES', 'IMPLIED_ROOTFS_FEATURES', d)} "

+#
+# Usage example:
+#

+# # ROOTFS_FEATURES will be "beta alpha" if the following configuration is used:
+# IMPLIED_ROOTFS_FEATURES[alpha] = "beta"
+# ROOTFS_FEATURES += "alpha"
+#
+# # ROOTFS_FEATURES will be "first" if the following configuration is used:
+# ROOTFS_FEATURES = "first second"

+# ROOTFS_FEATURES += "-second"
+
+python enable_list_variables() {
+ """Enable list variable functionality."""
+ for variable in d.getVar("LIST_VARIABLES").split():
+ d.setVar(variable + ':remove', ' ${@remove_prefixed_items("%s", d)}' % variable)
+ d.setVar(variable + ':prepend', '${@add_implied_items("%s", "IMPLIED_%s", d)} ' % (variable, variable))
+}
+enable_list_variables[eventmask] = "bb.event.ConfigParsed"
+addhandler enable_list_variables

index cda98035..9f3b8a4e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -172,6 +172,25 @@ BBINCLUDELOGS ??= "yes"

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

This class provides functions to improve the functionality of bitbake
list variables.

- Add the ability to remove items from a list variable without using :remove.

- Add the ability for a list item to imply the addition of other list items.

Usage requires either adding the variable name to LIST_VARIABLES, or manually

adding a :remove and a :prepend to each fully supported list variable.

To remove items from a configured list, simply append the item to be removed

to the variable with a '-' or '~' prefix. For example, to remove 'alpha' from

a configured variable, append '-alpha' to it.

To support implied list items, create a mapping of items to be appended to

the variable when a specific item is present. For example, to append 'beta'

to ROOTFS_FEATURES when 'alpha' is present, configure ROOTFS_FEATURES as such,

then set IMPLIED_ROOTFS_FEATURES[alpha] = "beta".

Signed-off-by: Christopher Larson <chris....@siemens.com>
---
meta/classes/lists.bbclass | 105 +++++++++++++++++++++++++++++++++++++

1 file changed, 105 insertions(+)

create mode 100644 meta/classes/lists.bbclass

diff --git a/meta/classes/lists.bbclass b/meta/classes/lists.bbclass
new file mode 100644

index 00000000..db8f5837
--- /dev/null
+++ b/meta/classes/lists.bbclass
@@ -0,0 +1,105 @@

+# Functions to improve the functionality of bitbake list variables.
+#
+# - Add the ability to remove items from a list variable without using :remove.
+# - Add the ability for a list item to imply the addition of other list items.
+#
+
+# Usage requires either adding the variable name to LIST_VARIABLES, or manually
+# adding a :remove and a :prepend to each fully supported list variable.
+#
+# To remove items from a configured list, simply append the item to be removed
+# to the variable with a '-' or '~' prefix. For example, to remove 'alpha' from

+# ROOTFS_FEATURES, add '-alpha' to ROOTFS_FEATURES.

+#
+# To support implied list items, create a mapping of items to be appended to
+# the variable when a specific item is present. For example, to append 'beta'

+# to ROOTFS_FEATURES when 'alpha' is present, configure ROOTFS_FEATURES as such,
+# then set IMPLIED_ROOTFS_FEATURES[alpha] = "beta".

+#
+# Boilerplate example:
+#
+# # Either this:

+# LIST_VARIABLES += "ROOTFS_FEATURES"

+#
+# # Or this:

+# ROOTFS_FEATURES:remove = "${@remove_prefixed_items('ROOTFS_FEATURES', d)}"

+# ROOTFS_FEATURES:prepend = "${@add_implied_items('ROOTFS_FEATURES', 'IMPLIED_ROOTFS_FEATURES', d)} "

+#
+# Usage example:
+#

+# # ROOTFS_FEATURES will be "beta alpha" if the following configuration is used:
+# IMPLIED_ROOTFS_FEATURES[alpha] = "beta"

+# ROOTFS_FEATURES += "alpha"
+#

+# # ROOTFS_FEATURES will be "first" if the following configuration is used:
+# ROOTFS_FEATURES = "first second"
+# ROOTFS_FEATURES += "-second"
+
+python enable_list_variables() {
+ """Enable list variable functionality."""
+ for variable in d.getVar("LIST_VARIABLES").split():
+ d.setVar(variable + ':remove', ' ${@remove_prefixed_items("%s", d)}' % variable)
+ d.setVar(variable + ':prepend', '${@add_implied_items("%s", "IMPLIED_%s", d)} ' % (variable, variable))
+}
+enable_list_variables[eventmask] = "bb.event.ConfigParsed"
+addhandler enable_list_variables

--
2.47.1

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

Add BASE_REPO_FEATURES and ROOTFS_FEATURES to LIST_VARIABLES.
Define fallback values and documentation flags for those variables.

Signed-off-by: Christopher Larson <chris....@siemens.com>
---

meta/conf/bitbake.conf | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cda98035..4906858a 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -172,6 +172,16 @@ BBINCLUDELOGS ??= "yes"

# Add event handlers for bitbake
INHERIT += "isar-events sstate"

+INHERIT += "lists"
+
+LIST_VARIABLES += "BASE_REPO_FEATURES ROOTFS_FEATURES"

+
+BASE_REPO_FEATURES ??= ""
+BASE_REPO_FEATURES[doc] = "Specifies the list of features for the base-apt repository."
+

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

These variables are useful to allow toggling of functionality in their
respective configuration files to alter how images are constructed.
Common usage would be for an image recipe to alter ROOTFS_FEATURES basd
on configured machine, distro, or combined features.

Signed-off-by: Christopher Larson <chris....@siemens.com>
---

meta/conf/bitbake.conf | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 4906858a..9f3b8a4e 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -172,13 +172,22 @@ BBINCLUDELOGS ??= "yes"

# Add event handlers for bitbake
INHERIT += "isar-events sstate"

+# Make features variables available

INHERIT += "lists"

-LIST_VARIABLES += "BASE_REPO_FEATURES ROOTFS_FEATURES"

+LIST_VARIABLES += "BASE_REPO_FEATURES MACHINE_FEATURES DISTRO_FEATURES ROOTFS_FEATURES"

BASE_REPO_FEATURES ??= ""

BASE_REPO_FEATURES[doc] = "Specifies the list of features for the base-apt repository."

+MACHINE_FEATURES ??= ""
+MACHINE_FEATURES[doc] = "Specifies the list of hardware features the MACHINE is capable of supporting."
+
+DISTRO_FEATURES ??= ""
+DISTRO_FEATURES[doc] = "The software support you want in your distribution for various features."
+
+COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}"
+

ROOTFS_FEATURES ??= ""

ROOTFS_FEATURES[doc] = "The list of features to be included in a root filesystem. Typically, you configure this variable in an image recipe or class."

--
2.47.1

[Uladzimir Bely]

Hello, Chris.

Could this be resend ones more as one series (v4) of patches?

V3 for some reasons was sent as two different series:
- https://patchwork.isar-build.org/project/isar/list/?series=1476 -
this includes only 1st patch
- https://patchwork.isar-build.org/project/isar/list/?series=1440 -
this includes two other patches.

The problem is that both series, if taken as "series" from patchwork
(i.e. https://patchwork.isar-build.org/series/1476/mbox/ and
https://patchwork.isar-build.org/series/1440/mbox/ , conflict with each
other.

I'm currently testing in CI all three latest (v3) patches picked
separately, but I'm not sure that they are exactly what should be meant
for merge.

--
Best regards,
Uladzimir.

[chris....@siemens.com]

[chris....@siemens.com]

From: Christopher Larson <chris....@siemens.com>

These variables are useful to allow toggling of functionality in their
respective configuration files to alter how images are constructed.
Common usage would be for an image recipe to alter ROOTFS_FEATURES basd
on configured machine, distro, or combined features.

Signed-off-by: Christopher Larson <chris....@siemens.com>
---

meta/conf/bitbake.conf | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf

index d444b802..5ab8ced7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -175,13 +175,22 @@ BBINCLUDELOGS ??= "yes"

[Larson, Chris]

On 2/12/2025 12:49 AM, Uladzimir Bely wrote:
> On Fri, 2025-01-24 at 14:20 -0700, chris.larson via isar-users wrote:
>> From: Christopher Larson <chris....@siemens.com>
>>
>> This class provides functions to improve the functionality of bitbake
>> list variables.

[snip]

>>
>
> Hello, Chris.
>
> Could this be resend ones more as one series (v4) of patches?

Re-sent as v4 in a new thread, thanks!

--
Christopher Larson
Siemens AG

www.siemens.com

[Uladzimir Bely]

Applied to next, thanks.

--
Best regards,
Uladzimir.