[PATCH 0/2] Fix data races in sstate tasks

[Felix Moessbauer]

While extensively testing the rootless builds in a CI running with
sstate cache, I was finally able to pin-point some sporadic issues.

These issues are completely unrelated to the rootless builds, though.
They have been around for quite some time, but were quite unlikely
to hit on normal (non multiconfig) builds.

Best regards,
Felix Moessbauer

Felix Moessbauer (2):
fix(rootfs): protect rootfs_install sstate prepare with lock
sstate: guard sstate funcs against execution in wrong context

meta/classes-recipe/dpkg-base.bbclass | 4 ++++
meta/classes-recipe/rootfs.bbclass | 6 +++++-
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
3 files changed, 13 insertions(+), 1 deletion(-)

--
2.53.0

[Felix Moessbauer]

Previously the lock was a no-op, as it was attached to a task that does
not exist (the task is named do_rootfs_install_setscene). Instead attach
it to the function that is executed by bitbake on sstate prepare.

This might explain some sporadic data races on sstate tasks that
silently lead to the task being re-executed.

Fixes: 085efc7d ("sstate: add caching to rootfs")
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
meta/classes-recipe/rootfs.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index a58ad031..8a4aad16 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -702,7 +702,7 @@ rootfs_install_sstate_prepare() {
sudo umount ${WORKDIR}/mnt/rootfs
sudo chown $(id -u):$(id -g) rootfs.tar
}
-do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
+rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

rootfs_install_sstate_finalize() {
# this runs in SSTATE_INSTDIR
--
2.53.0

[Felix Moessbauer]

In sstate.bbclass, sstate_package() sets SSTATE_CURRTASK from
the sstate task being packaged, then runs every function listed
in SSTATECREATEFUNCS. This happens independent of the context the sstate
task is scoped to for all tasks in a single recipe.

While this pattern is not problematic in isar itself (we only have one
SSTATECREATEFUNC per recipe), it becomes problematic when downstream
users define further manual sstate tasks. In this case, the sstate tasks
are executed in the wrong context, leading to races and failures in all
colors.

We fix this by guarding each manually defined sstate func to only run in
its intended context.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/classes-recipe/dpkg-base.bbclass | 4 ++++
meta/classes-recipe/rootfs.bbclass | 4 ++++
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 4 ++++
3 files changed, 12 insertions(+)

diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index f7a12302..85c37a5a 100644
--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -184,6 +184,8 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"

dpkg_build_sstate_prepare() {
+ [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
+
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
cp -f ${WORKDIR}/*.deb -t .
@@ -191,6 +193,8 @@ dpkg_build_sstate_prepare() {
}

dpkg_build_sstate_finalize() {
+ [ "${SSTATE_CURRTASK}" = "dpkg_build" ] || return 0
+

# this runs in SSTATE_INSTDIR

if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
mv -f ./*.deb -t ${WORKDIR}/
diff --git a/meta/classes-recipe/rootfs.bbclass b/meta/classes-recipe/rootfs.bbclass
index 8a4aad16..936f56fe 100644
--- a/meta/classes-recipe/rootfs.bbclass
+++ b/meta/classes-recipe/rootfs.bbclass
@@ -691,6 +691,8 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"

# the rootfs is owned by root, so we need some sudoing to pack and unpack
rootfs_install_sstate_prepare() {
+ [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
+
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
# tar --one-file-system will cross bind-mounts to the same filesystem,
# so we use some mount magic to prevent that
@@ -705,6 +707,8 @@ rootfs_install_sstate_prepare() {

rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

rootfs_install_sstate_finalize() {

+ [ "${SSTATE_CURRTASK}" = "rootfs_install" ] || return 0
+

# this runs in SSTATE_INSTDIR

# - after building the rootfs, the tar won't be there, but we also don't need to unpack
# - after restoring from cache, there will be a tar which we unpack and then delete
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index e746f469..98f6c090 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -264,12 +264,16 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
+ [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
+
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
+ [ "${SSTATE_CURRTASK}" = "bootstrap" ] || return 0
+

# this runs in SSTATE_INSTDIR

# we should restore symlinks after using tar
if [ -f bootstrap.tar.zst ]; then
--
2.53.0

[Jan Kiszka]

Out of curiosity: Does OE have similar setup and solve them the same
way? Or are we doing something ... special here? This pattern looks like
it does not scale very well (easily forgotten when adding another
function in Isar and even more easily in downstream).

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[MOESSBAUER, Felix]

They use (or nowadays mostly used) this pattern as well. However, this
is only needed when implementing the low-level sstate functions
manually, which should be avoided. When using the task flags (sstate-
inputdirs / sstate-outputdirs) this is not needed. Unfortunately we
need to implement the rootfs part manually, as tarring and untarring
the rootfs can only be performed by root (due to the permissions) and
we also want to avoid having rootfs' tarballs lingering in the workdir.

My hope is that sometime in the future we can solely rely on the task
flags, but this requires more refactoring. The rootless story is part
of it, but does not fully solve this until we drop the old privileged
part (and more ...).

You can also call it technical dept.

> This pattern looks like
> it does not scale very well (easily forgotten when adding another
> function in Isar and even more easily in downstream).

Well... you rarely need to write sstate functions by yourself, but rely
on the inputdirs / outputdirs flags instead. Then it is not needed.

Felix