[PATCH 0/6] Model isar-exclude-docs as rootfs feature

[Felix Moessbauer]

The isar-exclude-docs package provides mechanisms to exclude
documentation from the image. Technically this is implemented
by a dpkg filter, as well as a postinst script to strip the
documentation installed prior to installing the isar-exclude-docs
package.

This has the major drawback of first installing the documentation
(as isar-exclude-docs is installed along with ROOTFS_PACKAGES) and
then removing it again - which is even an emulated task and by that
super slow on a larger rootfs. Further, it does not handle cases
where docs are generated in a postinst script.

We now change this to configure apt prior to installing packages.
This results in less being installed into the rootfs. After that,
a final cleanup path from outside the chroot strip the remaining bits.

Best regards,
Felix Moessbauer

Felix Moessbauer (6):
rootfs: add feature to clean isar generated dpkg config
image: add IMAGE_ROOTFS_FEATURES to set rootfs features of the image
isar-exclude-docs: model via rootfs feature
rootfs: also purge package changelogs on exclude-docs
kas: port isar-exclude-docs to rootfs feature
testsuite: replace isar-exclude-docs by rootfs feature

RECIPE-API-CHANGELOG.md | 14 ++++++
kas/opt/Kconfig | 9 ++++
.../exclude-docs.yaml} | 4 +-
kas/package/Kconfig | 11 -----
meta-isar/conf/local.conf.sample | 6 ++-
.../recipes-core/images/isar-image-ci.bb | 2 +
.../recipes-core/images/isar-rootfs-ci.bb | 1 +
meta/classes-recipe/image.bbclass | 3 ++
meta/classes-recipe/rootfs.bbclass | 45 +++++++++++++++++++
.../isar-exclude-docs_0.2.3.bb | 4 ++
scripts/generate_yaml.sh | 1 -
testsuite/cibuilder.py | 1 -
12 files changed, 85 insertions(+), 16 deletions(-)
rename kas/{package/pkg_isar-exclude-docs.yaml => opt/exclude-docs.yaml} (58%)

--
2.55.0

[Felix Moessbauer]

Previously this could only be controlled from the image recipe itself.
By that, the set of used features could not be set from a local config
(or any other external context). To make that possible, we add this
variable.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
RECIPE-API-CHANGELOG.md | 5 +++++
meta/classes-recipe/image.bbclass | 2 ++
2 files changed, 7 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index abb67fa8..08ef328a 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1211,3 +1211,8 @@ ROOTFS_FEATURES:remove = "clean-apt-credentials"
All `ROOTFS_POSTPROCESS_COMMAND` steps are now executed within the `do_rootfs_install`
task to avoid idempotency issues on partial rebuilds. Task that previously had a
dependency to `do_rootfs_postprocess` shall now be changed to run after `do_rootfs_install`.
+
+### Add IMAGE_ROOTFS_FEATURES variable to control rootfs features of the image
+
+To easily select rootfs features from a local conf that should only apply to the image
+recipe, we provide the `IMAGE_ROOTFS_FEATURES`.
diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index 290d6d5c..d9125d05 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -93,6 +93,7 @@ inherit multiarch
inherit essential

ROOTFSDIR = "${IMAGE_ROOTFS}"
+IMAGE_ROOTFS_FEATURES ?= ""
ROOTFS_FEATURES += "\
clean-apt-lists \
generate-manifest \
@@ -101,6 +102,7 @@ ROOTFS_FEATURES += "\
generate-sbom \
clean-apt-credentials \
clean-dpkg-config \
+ ${IMAGE_ROOTFS_FEATURES} \
"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
ROOTFS_VARDEPS += "IMAGE_INSTALL"
--
2.55.0

[Felix Moessbauer]

We previously used a package to control if the docs should be included
in a rootfs or not. This has the drawback, that the expensive cleanup of
the existing docs needs to happen from a postinst script, which is
emulated on non native builds. Also, what is removed when depends on the
install order of the packages.

We change this by making the logic a rootfs feature, which - when
enabled - sets a dpkg config prior to the installation of packages. By
that, the docs are not extracted into the rootfs and only a cheap
cleanup of the docs we got from bootstrapping is needed. This also
cleanly aligns with other rootfs features we provide.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 6 +++
meta-isar/conf/local.conf.sample | 6 ++-
meta/classes-recipe/rootfs.bbclass | 40 +++++++++++++++++++
.../isar-exclude-docs_0.2.3.bb | 4 ++
4 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 08ef328a..0c98a33e 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1216,3 +1216,9 @@ dependency to `do_rootfs_postprocess` shall now be changed to run after `do_root

To easily select rootfs features from a local conf that should only apply to the image

recipe, we provide the `IMAGE_ROOTFS_FEATURES`.

+
+### Replace isar-exclude-docs with rootfs feature exclude-docs
+
+The `isar-exclude-docs` package provided mechanisms to remove documentation from
+the rootfs. This has been replaced by the `exclude-docs` rootfs feature. The
+`isar-exclude-docs` package should no longer be used.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 6e1e1546..a984e28f 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -152,7 +152,7 @@ CONF_VERSION = "1"

#
# The default list of extra packages to be installed.
-IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck isar-exclude-docs samefile hello isar-disable-apt-cache cowsay example-prebuilt"
+IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck samefile hello isar-disable-apt-cache cowsay example-prebuilt"

#
# Container and WSL machines don't need example module and enable-fsck.
@@ -164,6 +164,10 @@ IMAGE_INSTALL:remove:wsl = "example-module-${KERNEL_NAME} enable-fsck"
IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"

+#
+# Exclude documentation files from the image
+IMAGE_ROOTFS_FEATURES += "exclude-docs"
+
#
# Uncomment this to disable cross-compilation support
#ISAR_CROSS_COMPILE ?= "0"
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index f7822351..771d9b2e 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -41,6 +41,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
# 'populate-systemd-preset' - enable systemd units according to systemd presets
# 'clean-apt-credentials' - remove apt auth credentials written by ISAR_APT_CREDS
# 'clean-dpkg-config' - remove ISAR-specific dpkg configuration files from the rootfs
+# 'exclude-docs' - exclude most documentation files from the rootfs

# convenience variable to enable all features needed for a reproducible rootfs build
ROOTFS_FEATURES_REPRODUCIBLE = " \
@@ -312,6 +313,37 @@ rootfs_configure_apt() {
EOSUDO
}

+rootfs_exclude_docs_drop() {
+ if [ -d '${ROOTFSDIR}/usr/share/man' ]; then
+ find '${ROOTFSDIR}/usr/share/man/' -mindepth 1 ! -type d -delete
+ find '${ROOTFSDIR}/usr/share/man/' -depth -mindepth 1 -type d -empty -delete
+ fi
+ if [ -d '${ROOTFSDIR}/usr/share/doc' ]; then
+ find '${ROOTFSDIR}/usr/share/doc/' -mindepth 1 ! -type d ! -name "copyright" ! -name "changelog.*" -delete
+ find '${ROOTFSDIR}/usr/share/doc/' -depth -mindepth 1 -type d -empty -delete
+ fi
+}
+
+ROOTFS_CONFIGURE_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'exclude-docs', 'rootfs_configure_exclude_docs_filter', '', d)}"
+rootfs_configure_exclude_docs_filter() {
+ run_privileged_heredoc <<'EOSUDO'
+ set -e
+ mkdir -p '${ROOTFSDIR}/etc/dpkg/dpkg.cfg.d'
+ cat > '${ROOTFSDIR}/etc/dpkg/dpkg.cfg.d/55isar-exclude-docs' << 'EOF'
+path-exclude=/usr/share/man/*
+path-exclude=/usr/share/doc/*
+path-include=/usr/share/doc/*/copyright
+path-include=/usr/share/doc/*/changelog.*
+EOF
+
+EOSUDO
+ # drop docs from bootstrap
+ run_privileged_heredoc <<'EOSUDO'
+ set -e
+ ${rootfs_exclude_docs_drop}
+EOSUDO
+}
+
ROOTFS_CONFIGURE_COMMAND += "rootfs_disable_initrd_generation"
rootfs_disable_initrd_generation[weight] = "1"
rootfs_disable_initrd_generation() {
@@ -583,6 +615,14 @@ rootfs_postprocess_clean_dpkg_config() {
run_privileged find "${ROOTFSDIR}/etc/dpkg/dpkg.cfg.d" -type f -name '*isar*.cfg' -delete
}

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'exclude-docs', 'rootfs_postprocess_exclude_docs', '', d)}"
+rootfs_postprocess_exclude_docs() {
+ run_privileged_heredoc <<'EOSUDO'
+ set -e
+ ${rootfs_exclude_docs_drop}
+EOSUDO
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-pycache', 'rootfs_postprocess_clean_pycache', '', d)}"
rootfs_postprocess_clean_pycache() {
run_privileged find ${ROOTFSDIR}/usr -type f -name '*.pyc' -delete -print
diff --git a/meta/recipes-support/isar-exclude-docs/isar-exclude-docs_0.2.3.bb b/meta/recipes-support/isar-exclude-docs/isar-exclude-docs_0.2.3.bb
index a5aa06f4..5b641fba 100644
--- a/meta/recipes-support/isar-exclude-docs/isar-exclude-docs_0.2.3.bb
+++ b/meta/recipes-support/isar-exclude-docs/isar-exclude-docs_0.2.3.bb
@@ -14,3 +14,7 @@ do_install[cleandirs] += "${D}/etc/dpkg/dpkg.cfg.d/"
do_install() {
install -v -m 644 "${WORKDIR}/${BPN}" "${D}/etc/dpkg/dpkg.cfg.d/99${BPN}"
}
+
+do_prepare_build() {
+ bbwarn "This package is deprecated. Use the corresponding exclude-docs rootfs feature instead."
+}
--
2.55.0

[Felix Moessbauer]

It depends on the use-case if the isar generated dpkg config should be
preserved or not. At least for the image is should not be preserved, but
for other rootfs (like the sbuild ones) it should be.

We now add a rootfs feature to model this.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/classes-recipe/image.bbclass | 1 +
meta/classes-recipe/rootfs.bbclass | 6 ++++++

2 files changed, 7 insertions(+)

diff --git a/meta/classes-recipe/image.bbclass b/meta/classes-recipe/image.bbclass
index d17d1860..290d6d5c 100644
--- a/meta/classes-recipe/image.bbclass
+++ b/meta/classes-recipe/image.bbclass
@@ -100,6 +100,7 @@ ROOTFS_FEATURES += "\
populate-systemd-preset \
generate-sbom \
clean-apt-credentials \
+ clean-dpkg-config \

"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"
ROOTFS_VARDEPS += "IMAGE_INSTALL"

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index d3b0d471..f7822351 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -40,6 +40,7 @@ ROOTFS_BASE_DISTRO ?= "${BASE_DISTRO}"
# 'clean-log-files' - delete log files that are not owned by packages

# 'populate-systemd-preset' - enable systemd units according to systemd presets
# 'clean-apt-credentials' - remove apt auth credentials written by ISAR_APT_CREDS

+# 'clean-dpkg-config' - remove ISAR-specific dpkg configuration files from the rootfs

# convenience variable to enable all features needed for a reproducible rootfs build
ROOTFS_FEATURES_REPRODUCIBLE = " \

@@ -577,6 +578,11 @@ rootfs_postprocess_clean_apt_credentials() {
run_privileged rm -f "${ROOTFSDIR}/etc/apt/auth.conf.d/isar.conf"
}

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-dpkg-config', 'rootfs_postprocess_clean_dpkg_config', '', d)}"
+rootfs_postprocess_clean_dpkg_config() {
+ run_privileged find "${ROOTFSDIR}/etc/dpkg/dpkg.cfg.d" -type f -name '*isar*.cfg' -delete

+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-pycache', 'rootfs_postprocess_clean_pycache', '', d)}"
rootfs_postprocess_clean_pycache() {
run_privileged find ${ROOTFSDIR}/usr -type f -name '*.pyc' -delete -print

--
2.55.0

[Felix Moessbauer]

These can be quite big and usually do not provide value in a
docu-less system.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 3 +++
meta/classes-recipe/rootfs.bbclass | 3 +--
2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 0c98a33e..3798a9d1 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1222,3 +1222,6 @@ recipe, we provide the `IMAGE_ROOTFS_FEATURES`.

The `isar-exclude-docs` package provided mechanisms to remove documentation from

the rootfs. This has been replaced by the `exclude-docs` rootfs feature. The

`isar-exclude-docs` package should no longer be used.

+
+When enabled, the package changelogs are now removed as well (copyright is still
+kept for legal reasons).
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 771d9b2e..79bd70b2 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -319,7 +319,7 @@ rootfs_exclude_docs_drop() {

find '${ROOTFSDIR}/usr/share/man/' -depth -mindepth 1 -type d -empty -delete

fi

if [ -d '${ROOTFSDIR}/usr/share/doc' ]; then

- find '${ROOTFSDIR}/usr/share/doc/' -mindepth 1 ! -type d ! -name "copyright" ! -name "changelog.*" -delete
+ find '${ROOTFSDIR}/usr/share/doc/' -mindepth 1 ! -type d ! -name "copyright" -delete

find '${ROOTFSDIR}/usr/share/doc/' -depth -mindepth 1 -type d -empty -delete

fi
}
@@ -333,7 +333,6 @@ rootfs_configure_exclude_docs_filter() {
path-exclude=/usr/share/man/*
path-exclude=/usr/share/doc/*
path-include=/usr/share/doc/*/copyright
-path-include=/usr/share/doc/*/changelog.*
EOF

EOSUDO
--
2.55.0

[Felix Moessbauer]

The logic to exclude documentation has been ported over to a rootfs
feature. The former isar-exclude-docs package is just kept for
compatibility.

By that, we also port over the kas menu logic to use the rootfs feature
instead of installing the package.

For compatibility, we still keep the isar-exclude-docs package but issue
a warning if it is build.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

kas/opt/Kconfig | 9 +++++++++
.../exclude-docs.yaml} | 4 ++--
kas/package/Kconfig | 11 -----------
scripts/generate_yaml.sh | 1 -
4 files changed, 11 insertions(+), 14 deletions(-)

rename kas/{package/pkg_isar-exclude-docs.yaml => opt/exclude-docs.yaml} (58%)

diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index bc4ed997..ec010757 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -213,3 +213,12 @@ config KAS_INCLUDE_USE_DRACUT
string
default "kas/opt/dracut.yaml"
depends on USE_DRACUT
+
+config IMAGE_EXCLUDE_DOCS
+ bool "Exclude docs from the image"
+ default y
+
+config KAS_INCLUDE_IMAGE_EXCLUDE_DOCS
+ string
+ default "kas/opt/exclude-docs.yaml"
+ depends on IMAGE_EXCLUDE_DOCS
diff --git a/kas/package/pkg_isar-exclude-docs.yaml b/kas/opt/exclude-docs.yaml
similarity index 58%
rename from kas/package/pkg_isar-exclude-docs.yaml
rename to kas/opt/exclude-docs.yaml
index 0c24e982..c86b93ab 100644
--- a/kas/package/pkg_isar-exclude-docs.yaml
+++ b/kas/opt/exclude-docs.yaml
@@ -5,5 +5,5 @@ header:
version: 14

local_conf_header:
- package-isar-exclude-docs: |
- IMAGE_INSTALL:append = " isar-exclude-docs"
+ image-exclude-docs: |
+ IMAGE_ROOTFS_FEATURES += "exclude-docs"
diff --git a/kas/package/Kconfig b/kas/package/Kconfig
index 527aec0f..e342f8ef 100644
--- a/kas/package/Kconfig
+++ b/kas/package/Kconfig
@@ -89,17 +89,6 @@ config KAS_INCLUDE_PACKAGE_HELLO_ISAR
default "kas/package/pkg_hello-isar.yaml"
depends on PACKAGE_HELLO_ISAR

-
-config PACKAGE_ISAR_EXCLUDE_DOCS
- bool "isar-exclude-docs"
- default y
-
-config KAS_INCLUDE_PACKAGE_ISAR_EXCLUDE_DOCS
- string
- default "kas/package/pkg_isar-exclude-docs.yaml"
- depends on PACKAGE_ISAR_EXCLUDE_DOCS
-
-
config PACKAGE_ISAR_DISABLE_APT_CACHE
bool "isar-disable-apt-cache"
default y
diff --git a/scripts/generate_yaml.sh b/scripts/generate_yaml.sh
index aaa8bcba..904d241d 100755
--- a/scripts/generate_yaml.sh
+++ b/scripts/generate_yaml.sh
@@ -109,7 +109,6 @@ PKGS=" \
hello-isar \
hello \
isar-disable-apt-cache \
- isar-exclude-docs \
kselftest \
samefile \
sshd-regen-keys \
--
2.55.0

[Felix Moessbauer]

The isar-exclude-docs package is deprecated and needs to be replaced by
the exclude-docs rootfs feature.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta-test/recipes-core/images/isar-image-ci.bb | 2 ++
meta-test/recipes-core/images/isar-rootfs-ci.bb | 1 +
testsuite/cibuilder.py | 1 -
3 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta-test/recipes-core/images/isar-image-ci.bb b/meta-test/recipes-core/images/isar-image-ci.bb
index a2456bec..b88e2b45 100644
--- a/meta-test/recipes-core/images/isar-image-ci.bb
+++ b/meta-test/recipes-core/images/isar-image-ci.bb
@@ -5,6 +5,8 @@

require recipes-core/images/isar-image-base.bb

+ROOTFS_FEATURES += "exclude-docs"
+
FILESEXTRAPATHS:append = ":${LAYERDIR_isar}/recipes-core/images:"

# Avoid ISAR_RELEASE_CMD warning in image.bbclass
diff --git a/meta-test/recipes-core/images/isar-rootfs-ci.bb b/meta-test/recipes-core/images/isar-rootfs-ci.bb
index 4aeed71f..35f5005c 100644
--- a/meta-test/recipes-core/images/isar-rootfs-ci.bb
+++ b/meta-test/recipes-core/images/isar-rootfs-ci.bb
@@ -13,5 +13,6 @@ inherit multiarch
inherit rootfs

# behave similar to image class, so we can reuse the testing infrastructure
+ROOTFS_FEATURES += "exclude-docs"
DEPENDS += "${IMAGE_INSTALL}"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${@isar_multiarch_packages('IMAGE_INSTALL', d)}"

diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index 1d415084..0565f158 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -35,7 +35,6 @@ IMAGE_INSTALL_DEFAULT = [
'example-raw',
'example-module-${KERNEL_NAME}',
'enable-fsck',
- 'isar-exclude-docs',
'samefile',
'hello',
'isar-disable-apt-cache',
--
2.55.0