[PATCH] fix: add currenttask-guards to SSTATE(CREATE|POSTINST)FUNCS

[Adriaan Schmidt]

We use SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS to insert
code for preparation/postprocessing of sstate artifacts.
As it turns out, those are not per-job, but global, so if
a recipe has multiple sstate-cacheable tasks, all of those
functions are executed, and we need to make sure to bail out
of all unintended calls.
Because all invocations are visible in the logs, we add a
message in case a function was skipped, to make logs more
readable.

Signed-off-by: Adriaan Schmidt <adriaan...@siemens.com>
---
meta/classes/base.bbclass | 9 +++++++++
meta/classes/dpkg-base.bbclass | 2 ++
meta/classes/rootfs.bbclass | 2 ++
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 ++
meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc | 2 ++
5 files changed, 17 insertions(+)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index c730eec9..d5a3142b 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -204,6 +204,15 @@ def isar_export_ccache(d):
else:
os.environ['CCACHE_DISABLE'] = '1'

+isar_check_current_task() {
+ if [ "do_${BB_CURRENTTASK}" = "$1" ]; then
+ return 0
+ else
+ echo "skipping because current task is not $1"
+ return 1
+ fi
+}
+
do_fetch[dirs] = "${DL_DIR}"
do_fetch[file-checksums] = "${@bb.fetch.get_checksum_file_list(d)}"
do_fetch[vardeps] += "SRCREV"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4468a49a..7f2dcd3f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -184,6 +184,7 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"

dpkg_build_sstate_prepare() {
+ (isar_check_current_task "do_dpkg_build" || return) || true
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
if [ -n "$(find ${WORKDIR} -maxdepth 1 -name '*.deb' -print -quit)" ]; then
cp -f ${WORKDIR}/*.deb -t .
@@ -191,6 +192,7 @@ dpkg_build_sstate_prepare() {
}

dpkg_build_sstate_finalize() {
+ (isar_check_current_task "do_dpkg_build_setscene" || return) || true
# this runs in SSTATE_INSTDIR
if [ -n "$(find . -maxdepth 1 -name '*.deb' -print -quit)" ]; then
mv -f ./*.deb -t ${WORKDIR}/
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cd8fd118..aed50a91 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -437,6 +437,7 @@ SSTATE_TAR_ATTR_FLAGS ?= "--xattrs --xattrs-include='*'"

# the rootfs is owned by root, so we need some sudoing to pack and unpack
rootfs_install_sstate_prepare() {
+ (isar_check_current_task "do_rootfs_install" || return) || true
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
# tar --one-file-system will cross bind-mounts to the same filesystem,
# so we use some mount magic to prevent that
@@ -450,6 +451,7 @@ rootfs_install_sstate_prepare() {
do_rootfs_install_sstate_prepare[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

rootfs_install_sstate_finalize() {
+ (isar_check_current_task "do_rootfs_install_setscene" || return) || true
# this runs in SSTATE_INSTDIR
# - after building the rootfs, the tar won't be there, but we also don't need to unpack
# - after restoring from cache, there will be a tar which we unpack and then delete
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 08d61a84..a3ab28ca 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -219,12 +219,14 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
+ (isar_check_current_task "do_bootstrap" || return) || true
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
+ (isar_check_current_task "do_bootstrap_setscene" || return) || true
# this runs in SSTATE_INSTDIR
# we should restore symlinks after using tar
if [ -f bootstrap.tar.zst ]; then
diff --git a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
index 931f6f13..44775bc5 100644
--- a/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
+++ b/meta/recipes-core/isar-mmdebstrap/isar-mmdebstrap.inc
@@ -222,12 +222,14 @@ SSTATECREATEFUNCS += "bootstrap_sstate_prepare"
SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
+ (isar_check_current_task "do_bootstrap" || return) || true
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst
sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
+ (isar_check_current_task "do_bootstrap_setscene" || return) || true
# this runs in SSTATE_INSTDIR
# we should restore symlinks after using tar
if [ -f bootstrap.tar.zst ]; then
--
2.39.5

[Baurzhan Ismagulov]

On 2025-04-11 16:33, 'Adriaan Schmidt' via isar-users wrote:
> We use SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS to insert
> code for preparation/postprocessing of sstate artifacts.
> As it turns out, those are not per-job, but global, so if
> a recipe has multiple sstate-cacheable tasks, all of those
> functions are executed, and we need to make sure to bail out
> of all unintended calls.

Thanks Adriaan for the patch,

we've had a look and have a few comments (numbering for easier referencing):

1. For this use case, I'd like to have a testcase. Would you like to submit one
or provide quick hints what to add to trigger the behavior? We've started
checking some simple cases but haven't reproduced it so far.

> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -204,6 +204,15 @@ def isar_export_ccache(d):
> else:
> os.environ['CCACHE_DISABLE'] = '1'
>
> +isar_check_current_task() {
> + if [ "do_${BB_CURRENTTASK}" = "$1" ]; then
> + return 0
> + else
> + echo "skipping because current task is not $1"
> + return 1
> + fi
> +}

2. Could we name "isar_check_current_task" to reflect when it returns what?
E.g., "fail_if_task_not" or something. (Minor but easy and helpful)

3. Technically, this function itself doesn't skip, so the message might be
misleading if the caller makes a mistake. (Just a comment; I don't have a
better wording on the spot)

> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -184,6 +184,7 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
> SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
>
> dpkg_build_sstate_prepare() {
> + (isar_check_current_task "do_dpkg_build" || return) || true

4. If isar_check_current_task succeeds, the execution should continue. If it
fails, we return. Why is || true needed? (Just double-checking)

Coincidentally, many || true occurrences needed fixing in the past, so I'm
not impartial towards those :) , although it's just a handier form of "if".

5. I wonder if there are ways to remove hard-coding of "do_dpkg_build" into the
function (something similar to overrides, flags, etc.). (Just brainstorming)

(I've always wanted to separate hard-coded cases like "i386", "stretch",
etc. from the code = separate generic mechanism from the specific machine,
distro, etc. -- haven't come to that so far)


6. Just for my understanding, what are some examples where *CREATE* and
*POSTINST* are useful?

7. Given that this touches central classes and sstate bugs could silently
produce incorrect deliverables, what are the opinions regarding applying
this after the 0.11 release? The use case doesn't sound (to me, in any case)
like it would affect the most downstreams, and they are using next anyway.


With kind regards,
Baurzhan

[Schmidt, Adriaan]

Baurzhan Ismagulov, Mittwoch, 16. April 2025 16:13:

> On 2025-04-11 16:33, 'Adriaan Schmidt' via isar-users wrote:
> > We use SSTATECREATEFUNCS and SSTATEPOSTINSTFUNCS to insert
> > code for preparation/postprocessing of sstate artifacts.
> > As it turns out, those are not per-job, but global, so if
> > a recipe has multiple sstate-cacheable tasks, all of those
> > functions are executed, and we need to make sure to bail out
> > of all unintended calls.
>
> Thanks Adriaan for the patch,
>
> we've had a look and have a few comments (numbering for easier referencing):
>
> 1. For this use case, I'd like to have a testcase. Would you like to submit one
> or provide quick hints what to add to trigger the behavior? We've started
> checking some simple cases but haven't reproduced it so far.

It's actually not a problem in current Isar, as the behavior is only triggered when
we have a recipe that has more than one addition to SSTATECREATEFUNCS
or SSTATEPOSTINSTFUNCS.
Felix discovered the issue while working on his initrd patches.

>
> > --- a/meta/classes/base.bbclass
> > +++ b/meta/classes/base.bbclass
> > @@ -204,6 +204,15 @@ def isar_export_ccache(d):
> > else:
> > os.environ['CCACHE_DISABLE'] = '1'
> >
> > +isar_check_current_task() {
> > + if [ "do_${BB_CURRENTTASK}" = "$1" ]; then
> > + return 0
> > + else
> > + echo "skipping because current task is not $1"
> > + return 1
> > + fi
> > +}
>
> 2. Could we name "isar_check_current_task" to reflect when it returns what?
> E.g., "fail_if_task_not" or something. (Minor but easy and helpful)

Sure, I also wasn't sure what to name the function.
The pattern was copied from OE. There they don't have a generic function,
but instead copy the check directly to all relevant locations.

> 3. Technically, this function itself doesn't skip, so the message might be
> misleading if the caller makes a mistake. (Just a comment; I don't have a
> better wording on the spot)

That's true, it relies on correct usage by the caller, but I don't have a better idea.
Even if we "correct" the wording, the caller still needs to do the right thing...
What that output addresses is that we see the invocation of all registered functions
in the log of the bitbake task. The output makes it clear that the function was
called but didn't do anything.

>
> > --- a/meta/classes/dpkg-base.bbclass
> > +++ b/meta/classes/dpkg-base.bbclass
> > @@ -184,6 +184,7 @@ SSTATECREATEFUNCS += "dpkg_build_sstate_prepare"
> > SSTATEPOSTINSTFUNCS += "dpkg_build_sstate_finalize"
> >
> > dpkg_build_sstate_prepare() {
> > + (isar_check_current_task "do_dpkg_build" || return) || true
>
> 4. If isar_check_current_task succeeds, the execution should continue. If it
> fails, we return. Why is || true needed? (Just double-checking)

If the check fails, bitbake not only returns from this one function, but terminates
the whole task.

> Coincidentally, many || true occurrences needed fixing in the past, so I'm
> not impartial towards those :) , although it's just a handier form of "if".

I tried to make the check a one-liner... But I'd also be ok with making it a
"normal" if/then/fi.

> 5. I wonder if there are ways to remove hard-coding of "do_dpkg_build" into the
> function (something similar to overrides, flags, etc.). (Just brainstorming)

I don't think so... we're deep in (undocumented) sstate logic here...
The approach was taken from OE, e.g. https://github.com/openembedded/openembedded-core/blob/master/meta/classes-recipe/native.bbclass#L234

> (I've always wanted to separate hard-coded cases like "i386", "stretch",
> etc. from the code = separate generic mechanism from the specific machine,
> distro, etc. -- haven't come to that so far)
>
>
> 6. Just for my understanding, what are some examples where *CREATE* and
> *POSTINST* are useful?

We need to use those hooks because in some cases we don't produce "nice"
artifacts that can be given to the sstate logic directly:

- our rootfs needs to be tar'ed (with sudo) to preserve file permissions
- our dpkg jobs create (any number of) .deb packages, that we need to collect before giving them to sstate

In other instances (e.g. bootstrap) we already have a nice artifact, and
don't need this additional processing.

Adriaan