[PATCH 1/1] fix(dpkg-build): deploy debs via shared sstate dir

[Felix Moessbauer]

The built debs were stored in a WORKDIR-internal plaindir and consumed
from there by do_deploy_deb. do_deploy_deb is a task that mutates
the external isar-apt repository and can therefore never be sstate
cached, so it always runs and relies on the plaindir being present on
disk.

When do_dpkg_build is served from the sstate cache, that WORKDIR-internal
plaindir is not guaranteed to be materialized (e.g. when the task is
covered via hash equivalence without a setscene restore). do_deploy_deb
then runs against an empty deploy dir, passes an unexpanded glob to
dpkg-deb and aborts the build.

Fix it by deploying the debs into a shared, sstate-tracked DEPLOY_DIR_DEB
via sstate-inputdirs/sstate-outputdirs.

Fixes: 3363ff57 ("dpkg: directly deploy via sstate cache")
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
RECIPE-API-CHANGELOG.md | 8 ++++++++
doc/technical_overview.md | 2 +-
meta/classes-recipe/barebox.bbclass | 4 ++--
meta/classes-recipe/dpkg-base.bbclass | 19 ++++++++++++++-----
4 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index f5b84c84..4edee1bd 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1129,6 +1129,14 @@ environment and cannot be deleted from the outside by the calling user. To simpl
the cleanup, we provide the `isar-clean-builddir` script that helps purging
directories with mixed ownerships (without requiring root privileges).

+### Built debs are deployed to `DEPLOY_DIR_DEB`
+
+The debs produced by `do_dpkg_build` are now exported into a shared, sstate-tracked
+deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
+
+Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
+artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.
+
### Add Hyper-V machine support

A new machine `hyper-v` has been introduced for building images
diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index fe427cae..5d2e1b8a 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -217,7 +217,7 @@ Both consist of the following steps:
finally unmount again (`dpkg_undo_mounts`).

6. Task `do_deploy_deb`: add successfully built packages
- `${DEPLOYDIR}/*.deb` to the isar-apt repository
+ `${DEPLOY_DIR_DEB}/*.deb` to the isar-apt repository
`${REPO_ISAR_DIR}/${DISTRO}`

## 3.6 Populate Target Filesystem
diff --git a/meta/classes-recipe/barebox.bbclass b/meta/classes-recipe/barebox.bbclass
index c9a767d5..094f8018 100644
--- a/meta/classes-recipe/barebox.bbclass
+++ b/meta/classes-recipe/barebox.bbclass
@@ -47,12 +47,12 @@ BAREBOX_BASE_BIN ?= "barebox"

do_deploy[dirs] = "${DEPLOY_DIR_IMAGE}"
do_deploy() {
- dpkg --fsys-tarfile ${DEPLOYDIR}/${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb | \
+ dpkg --fsys-tarfile ${DEPLOY_DIR_DEB}/${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb | \
tar xOf - ./usr/lib/barebox/barebox.bin \
> "${DEPLOY_DIR_IMAGE}/${BAREBOX_BASE_BIN}.img"
ln -sf ${BAREBOX_BASE_BIN}.img ${DEPLOY_DIR_IMAGE}/barebox.bin

- dpkg --fsys-tarfile ${DEPLOYDIR}/${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb | \
+ dpkg --fsys-tarfile ${DEPLOY_DIR_DEB}/${PN}_${CHANGELOG_V}_${DISTRO_ARCH}.deb | \
tar xOf - ./usr/lib/barebox/barebox.config \
> "${DEPLOY_DIR_IMAGE}/${BAREBOX_BASE_BIN}.config"
ln -sf ${BAREBOX_BASE_BIN}.config ${DEPLOY_DIR_IMAGE}/barebox.config
diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index bc53e4bd..4a81787e 100644
--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -12,7 +12,12 @@ inherit repository
inherit deb-dl-dir
inherit essential

+# Local (WORKDIR-internal) dir where do_dpkg_build collects the built debs.
+# Its content is exported into the shared, sstate-tracked DEPLOY_DIR_DEB so
+# that consumers (do_deploy_deb) always find the artifacts, independent of the
+# WORKDIR lifecycle.
DEPLOYDIR = "${WORKDIR}/deploy"
+DEPLOY_DIR_DEB = "${DEPLOY_DIR}/isar-deb/${DISTRO}-${DISTRO_ARCH}/${PN}"

DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
@@ -200,7 +205,8 @@ python do_dpkg_build() {
bb.build.exec_func('dpkg_chroot_finalize', d)
}
do_dpkg_build[cleandirs] = "${DEPLOYDIR}"
-do_dpkg_build[sstate-plaindirs] = "${DEPLOYDIR}"
+do_dpkg_build[sstate-inputdirs] = "${DEPLOYDIR}"
+do_dpkg_build[sstate-outputdirs] = "${DEPLOY_DIR_DEB}"
do_dpkg_build[network] = "${TASK_USE_NETWORK_AND_SUDO}"
do_dpkg_build[depends] = "${SCHROOT_DEP} base-apt:do_cache isar-apt:do_cache_config"
do_dpkg_build[postfuncs] += "dpkg_collect_debs"
@@ -218,7 +224,7 @@ addtask dpkg_build_setscene
CLEANFUNCS += "deb_clean"

deb_clean() {
- DEBS=$( find ${DEPLOYDIR} -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
+ DEBS=$( find ${DEPLOY_DIR_DEB} -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
if [ -n "${DEBS}" ]; then
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
@@ -232,8 +238,11 @@ do_clean[network] = "${TASK_USE_SUDO}"

do_deploy_deb() {
deb_clean
- repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
- "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${DEPLOYDIR}/*.deb
+ debs=$(find ${DEPLOY_DIR_DEB} -maxdepth 1 -name '*.deb')
+ if [ -n "${debs}" ]; then
+ repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
+ "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${debs}
+ fi
}

addtask deploy_deb after do_dpkg_build before do_build
@@ -241,7 +250,7 @@ do_deploy_deb[deptask] = "do_deploy_deb"
do_deploy_deb[rdeptask] = "do_deploy_deb"
do_deploy_deb[depends] += "isar-apt:do_cache_config"
do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_deb[dirs] = "${S}"
+do_deploy_deb[dirs] = "${S} ${DEPLOY_DIR_DEB}"

python do_devshell() {
isar_export_proxies(d)
--
2.53.0

[Jan Kiszka]

On 29.07.26 13:18, Felix Moessbauer wrote:
> The built debs were stored in a WORKDIR-internal plaindir and consumed
> from there by do_deploy_deb. do_deploy_deb is a task that mutates
> the external isar-apt repository and can therefore never be sstate
> cached, so it always runs and relies on the plaindir being present on
> disk.
>
> When do_dpkg_build is served from the sstate cache, that WORKDIR-internal
> plaindir is not guaranteed to be materialized (e.g. when the task is
> covered via hash equivalence without a setscene restore). do_deploy_deb
> then runs against an empty deploy dir, passes an unexpanded glob to
> dpkg-deb and aborts the build.

Not getting it: why is dpkg_build_setscene not properly filling
DEPLOYDIR then? We told it that the build will fill it, it picked things
up from there for caching, and it should refill it when it skips the task.

>
> Fix it by deploying the debs into a shared, sstate-tracked DEPLOY_DIR_DEB
> via sstate-inputdirs/sstate-outputdirs.

This is creating yet another copy of the generated deb files in our
build dirs. Sounds inefficient, specifically when we have larger artifacts.

Jan

--
Siemens AG, Foundational Technologies
Linux Expert Center

[MOESSBAUER, Felix]

On Thu, 2026-07-30 at 16:41 +0200, Jan Kiszka wrote:
> On 29.07.26 13:18, Felix Moessbauer wrote:
> > The built debs were stored in a WORKDIR-internal plaindir and consumed
> > from there by do_deploy_deb. do_deploy_deb is a task that mutates
> > the external isar-apt repository and can therefore never be sstate
> > cached, so it always runs and relies on the plaindir being present on
> > disk.
> >
> > When do_dpkg_build is served from the sstate cache, that WORKDIR-internal
> > plaindir is not guaranteed to be materialized (e.g. when the task is
> > covered via hash equivalence without a setscene restore). do_deploy_deb
> > then runs against an empty deploy dir, passes an unexpanded glob to
> > dpkg-deb and aborts the build.
>
> Not getting it: why is dpkg_build_setscene not properly filling
> DEPLOYDIR then? We told it that the build will fill it, it picked things
> up from there for caching, and it should refill it when it skips the task.

Well... That's unfortunately how bitbake works. The artifacts in
plaindirs are not under sstate manifest control (the thing that also
blames you if you deploy artifacts with equal name but different
content to the same location). That also means they are removed on
"clean".

In other words, the staging/copy step only runs when the task's
setscene is actually executed (i.e. the sstate object is unpacked).  It
does not run when the task is considered to be already executed based
on hash equivalence. In this case, the restore directly happens from
the sstate cache via it's manifest tracking.

Don't ask me WHY bitbake internally implements it like that...

Hash equivalence is described in [1], I still can only find this detail
in the sstate sources (and runqueue.py line 2324ff) ...

[1]
https://docs.yoctoproject.org/overview-manual/concepts.html#hash-equivalence

>
> >
> > Fix it by deploying the debs into a shared, sstate-tracked DEPLOY_DIR_DEB
> > via sstate-inputdirs/sstate-outputdirs.
>
> This is creating yet another copy of the generated deb files in our
> build dirs. Sounds inefficient, specifically when we have larger artifacts.

Yes, unfortunately. But we need to properly deploy these artifacts. One
optimization would be to hardlink the isar-apt cache pool files to
these deployed files, however I'm not sure if this can practically be
implemented.

I'm not so much worried about the additional space consumption. A way
bigger waste of space are the left-over chroots.

Felix

[Zhihang Wei]

On 7/29/26 13:18, 'Felix Moessbauer' via isar-users wrote:
> The built debs were stored in a WORKDIR-internal plaindir and consumed
> from there by do_deploy_deb. do_deploy_deb is a task that mutates
> the external isar-apt repository and can therefore never be sstate
> cached, so it always runs and relies on the plaindir being present on
> disk.
>
> When do_dpkg_build is served from the sstate cache, that WORKDIR-internal
> plaindir is not guaranteed to be materialized (e.g. when the task is
> covered via hash equivalence without a setscene restore). do_deploy_deb
> then runs against an empty deploy dir, passes an unexpanded glob to
> dpkg-deb and aborts the build.
>
> Fix it by deploying the debs into a shared, sstate-tracked DEPLOY_DIR_DEB
> via sstate-inputdirs/sstate-outputdirs.
>
> Fixes: 3363ff57 ("dpkg: directly deploy via sstate cache")
> Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
> ---
> RECIPE-API-CHANGELOG.md | 8 ++++++++
> doc/technical_overview.md | 2 +-
> meta/classes-recipe/barebox.bbclass | 4 ++--
> meta/classes-recipe/dpkg-base.bbclass | 19 ++++++++++++++-----
> 4 files changed, 25 insertions(+), 8 deletions(-)
>

<snip>

> diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
> index bc53e4bd..4a81787e 100644
> --- a/meta/classes-recipe/dpkg-base.bbclass
> +++ b/meta/classes-recipe/dpkg-base.bbclass
> @@ -12,7 +12,12 @@ inherit repository
> inherit deb-dl-dir
> inherit essential
>
> +# Local (WORKDIR-internal) dir where do_dpkg_build collects the built debs.
> +# Its content is exported into the shared, sstate-tracked DEPLOY_DIR_DEB so
> +# that consumers (do_deploy_deb) always find the artifacts, independent of the
> +# WORKDIR lifecycle.
> DEPLOYDIR = "${WORKDIR}/deploy"
> +DEPLOY_DIR_DEB = "${DEPLOY_DIR}/isar-deb/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> DEPENDS ?= ""
> RPROVIDES ?= "${PROVIDES}"
> @@ -200,7 +205,8 @@ python do_dpkg_build() {
> bb.build.exec_func('dpkg_chroot_finalize', d)
> }
> do_dpkg_build[cleandirs] = "${DEPLOYDIR}"
> -do_dpkg_build[sstate-plaindirs] = "${DEPLOYDIR}"
> +do_dpkg_build[sstate-inputdirs] = "${DEPLOYDIR}"
> +do_dpkg_build[sstate-outputdirs] = "${DEPLOY_DIR_DEB}"
> do_dpkg_build[network] = "${TASK_USE_NETWORK_AND_SUDO}"
> do_dpkg_build[depends] = "${SCHROOT_DEP} base-apt:do_cache isar-apt:do_cache_config"

Can you send a rebase? The line above does not match with current next.

Zhihang

[Felix Moessbauer]

The built debs were stored in a WORKDIR-internal plaindir and consumed

from there by do_deploy_deb. do_deploy_deb is a live task that mutates

the external isar-apt repository and can therefore never be sstate

cached, so it always runs and relies purely on the plaindir being

present on disk.

When do_dpkg_build is served from the sstate cache, that WORKDIR-internal
plaindir is not guaranteed to be materialized (e.g. when the task is
covered via hash equivalence without a setscene restore). do_deploy_deb
then runs against an empty deploy dir, passes an unexpanded glob to
dpkg-deb and aborts the build.

Fix it by deploying the debs into a shared, sstate-tracked DEPLOY_DIR_DEB

via sstate-inputdirs/sstate-outputdirs, the same pattern already used by
do_bootstrap, do_rootfs_install and friends. This makes the deploy dir a
first-class sstate output that bitbake reliably restores whenever a
consumer needs it, decoupled from the WORKDIR lifecycle. do_deploy_deb
additionally guards against an empty deploy dir so it can no longer feed
a literal glob to dpkg-deb.

Fixes: 3fc96652 ("dpkg: directly deploy via sstate cache")

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

Changes since v1:
- rebased on top of the latest next, no functional changes

RECIPE-API-CHANGELOG.md | 8 ++++++++
doc/technical_overview.md | 2 +-
meta/classes-recipe/barebox.bbclass | 4 ++--
meta/classes-recipe/dpkg-base.bbclass | 19 ++++++++++++++-----
4 files changed, 25 insertions(+), 8 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7b13cc7e..7e923e55 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1138,6 +1138,14 @@ network access. Recipes that still require network access during the build can s
`DPKG_BUILD_ENABLE_NETWORK = "1"` to explicitly enable it when using an sbuild
version newer than 0.89.

diff --git a/meta/classes-recipe/dpkg-base.bbclass b/meta/classes-recipe/dpkg-base.bbclass
index e2ed4dfb..610213b8 100644

--- a/meta/classes-recipe/dpkg-base.bbclass
+++ b/meta/classes-recipe/dpkg-base.bbclass
@@ -12,7 +12,12 @@ inherit repository
inherit deb-dl-dir
inherit essential

+# Local (WORKDIR-internal) dir where do_dpkg_build collects the built debs.
+# Its content is exported into the shared, sstate-tracked DEPLOY_DIR_DEB so
+# that consumers (do_deploy_deb) always find the artifacts, independent of the
+# WORKDIR lifecycle.
DEPLOYDIR = "${WORKDIR}/deploy"
+DEPLOY_DIR_DEB = "${DEPLOY_DIR}/isar-deb/${DISTRO}-${DISTRO_ARCH}/${PN}"

DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
@@ -200,7 +205,8 @@ python do_dpkg_build() {
bb.build.exec_func('dpkg_chroot_finalize', d)
}
do_dpkg_build[cleandirs] = "${DEPLOYDIR}"
-do_dpkg_build[sstate-plaindirs] = "${DEPLOYDIR}"
+do_dpkg_build[sstate-inputdirs] = "${DEPLOYDIR}"
+do_dpkg_build[sstate-outputdirs] = "${DEPLOY_DIR_DEB}"
do_dpkg_build[network] = "${TASK_USE_NETWORK_AND_SUDO}"
do_dpkg_build[depends] = "${SCHROOT_DEP}"

--
2.53.0

[MOESSBAUER, Felix]

Rebase sent. However I have two more fixes that relate to that
location. All of them are independent, except that they change lines
which are close.

Will send them on top of the v2 of this patch.

Felix

[Zhihang Wei]

Applied to next, thanks.

Zhihang