[PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate

[Felix Moessbauer]

Changes since v1:

- move changelog entry to correct patch
- rework caching logic to not break gbp or apt fetcher interfaces

The series has now been tested in fast CI.

Best regards,
Felix Moessbauer

Felix Moessbauer (2):
dpkg: let do_deploy_deb runtime depend on do_deploy_source
dpkg: cache do_dpkg_source results in sstate

RECIPE-API-CHANGELOG.md | 8 ++++++++
meta/classes-recipe/dpkg-source.bbclass | 21 +++++++++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
3 files changed, 28 insertions(+), 3 deletions(-)

--
2.55.0

[Felix Moessbauer]

Changes since v2:

- fix container-loader to build one source package per architecture
- add an API changelog entry regarding architecture-specific source packages

Changes since v1:

- move changelog entry to correct patch
- rework caching logic to not break gbp or apt fetcher interfaces

The series (v3) has been tested in -t fast and -t prebuilt CI.

Best regards,
Felix Moessbauer

Felix Moessbauer (3):

dpkg: let do_deploy_deb runtime depend on do_deploy_source

fix(container-loader): build one source package per architecture

dpkg: cache do_dpkg_source results in sstate

RECIPE-API-CHANGELOG.md | 29 ++++++++++++++++++++
meta/classes-recipe/container-loader.bbclass | 5 ++++
meta/classes-recipe/dpkg-source.bbclass | 21 ++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
4 files changed, 54 insertions(+), 3 deletions(-)

--
2.55.0

[Felix Moessbauer]

The isar-apt is an artifact that is deployed by isar and can be consumed
as-is as an apt repository by downstream users. As the isar-apt also
provides the debian source packages, we have to ensure that these are
also deployed when the do_dpkg_build task is provided by the SState
cache. Otherwise the isar-apt is not reproducible across runs with and
without the SState cache.

To fix this, we add a runtime dependency between do_deploy_deb and
do_deploy_source, to ensure all sources of our debs are deployed as
well.

Reported-by: Andreas Ziegler <ziegler...@siemens.com>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
meta/classes-recipe/dpkg-source.bbclass | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index b2b45ed3..92d06e1b 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -49,6 +49,8 @@ do_deploy_source() {
addtask deploy_source after do_dpkg_source

do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+# ensure that the source package is deployed into isar-apt
+do_deploy_deb[rdepends] += "${BPN}:do_deploy_source"

SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"

--
2.55.0

[Felix Moessbauer]

We currently build architecture specific source packages with varying
content under the same name. This is not allowed in Debian, but remained
silently unnoticed as long as the packages are not deployed into the
same isar-apt.

As a preparation to cache the source packages, we need to ensure all
packages we build are either identical across all architectures, or are
architecture specific, along with having the arch in the package name.

Fixes: cd4e9090 ("container-loader: make generated package arch ...")

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/classes-recipe/container-loader.bbclass | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/meta/classes-recipe/container-loader.bbclass b/meta/classes-recipe/container-loader.bbclass
index e5c82a74..8439db53 100644
--- a/meta/classes-recipe/container-loader.bbclass
+++ b/meta/classes-recipe/container-loader.bbclass
@@ -10,6 +10,11 @@ SRC_URI += " \
file://container-loader.service.tmpl \
file://container-loader.sh.tmpl"

+# make the source package architecture dependent to avoid
+# package collisions on multiconf / multiarch builds
+PROVIDES := "${BPN}"
+DEBIAN_PROVIDES := "${BPN}"
+PN .= "-${DISTRO_ARCH}"
DPKG_ARCH ?= "${DISTRO_ARCH}"
DEBIAN_MULTI_ARCH ?= "allowed"

--
2.55.0

[Felix Moessbauer]

As the do_dpkg_source task now runs for each corresponding
do_deploy_deb task, we also want to cache the generated results (to
avoid pulling in a much bigger dependency tree on otherwise cached
rebuilds).

For that, we cache the .dsc and the .tar.* artifacts in the deploy dir.
This is implemented similar to how do_dpkg_build artifacts (.debs) are
cached, which is also compatible with a hash equivalence server.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

RECIPE-API-CHANGELOG.md | 29 +++++++++++++++++++++++++
meta/classes-recipe/dpkg-source.bbclass | 19 ++++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
3 files changed, 47 insertions(+), 3 deletions(-)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7e923e55..2db05169 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1146,6 +1146,35 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.

+### Debian source packages are deployed to `DEPLOY_DIR_SRC`
+
+The source packages produced by `do_dpkg_source` are now exported into a shared,
+sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`.
+
+Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now
+reference the local copy in `${DEPLOY_DIR_SRC}` instead.
+
+### Architecture specific source packages
+
+Setting `DPKG_ARCH = "<arch>"` makes the source package architecture-specific,
+unlike `all` or `any`. Because Debian source packages are identified by name and
+version, package providers must ensure that a source package with the same name
+and version does not contain different content for different architectures. This
+was never allowed, but Isar now enforces it.
+
+In this case, you can use the following pattern to make the package architecture specific,
+while still maintaining backward compatibility:
+
+```
+inherit dpkg-raw

+PROVIDES := "${BPN}"
+DEBIAN_PROVIDES := "${BPN}"
+PN .= "-${DISTRO_ARCH}"

+DPKG_ARCH ?= "${DISTRO_ARCH}"
+```
+
+Firmware packages are an exception because they are built for only one architecture.
+
### Add Hyper-V machine support

A new machine `hyper-v` has been introduced for building images
diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index 92d06e1b..97cf9714 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \
DPKG_SOURCE_EXTRA_ARGS ?= "-I"

DEBIAN_SOURCE ?= "${BPN}"
+SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg"
+DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}"

+do_dpkg_source[cleandirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
do_dpkg_source() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
@@ -22,9 +27,19 @@ do_dpkg_source() {
fi
find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
+ # move packages to deploy directory
+ find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \;
}
addtask dpkg_source after do_prepare_build

+SSTATETASKS += "do_dpkg_source"
+
+python do_dpkg_source_setscene() {
+ sstate_setscene(d)
+}
+
+addtask dpkg_source_setscene
+
CLEANFUNCS += "deb_clean_source"

deb_clean_source() {
@@ -34,11 +49,11 @@ deb_clean_source() {

do_deploy_source[depends] += "isar-apt:do_cache_config"
do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_source[dirs] = "${S}"
+do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}"
do_deploy_source() {
repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
if [ -n "${DSC_FILE}" ]; then
repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index d8e129bb..a42703d5 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -116,7 +116,7 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}

- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)

# networking is automatically enabled on older versions of sbuild
sbuild_network_option=""
--
2.55.0

[Zhihang Wei]

Applied to next, thanks.

Zhihang

[MOESSBAUER, Felix]

Hi,

this series revealed another bug regarding source packages with same
name but different content:

Similar to upstream Debian, our kernel source packages are architecture
specific. By that, we also have to encode the architecture (or machine)
in the name of the package. Otherwise existing packages might get
reused on rebuilds with other targets (or multiarch)

In isar, this has ever been modeled incorrectly (the kernels .dsc file
was arch=any despite being arch specific). This is easy to fix, but
also depending on the kernels was done incorrectly (a KERNEL_NAME =
mainline is not specific enough, as mainline on arm64 is technically a
different kernel (source package) than mainline on amd64.

To fix this, we have the following options (which I tried):

1. encode the MACHINE or DISTRO_ARCH in KERNEL_NAME. Usually arch is
enough, but if a machine specific defconfig is used, the machine needs
to be encoded in the name => clean, works but might require downstream
changes

2. internally generate kernel source package with -${DISTRO_ARCH}
suffix, but keep DEPENDS / PROVIDES and binary package names as-is:
This is different to how Debian names the kernels. For the kernel build
itself this can be modeled, but the dependencies of custom modules back
to the "correct" kernel cannot be modeled, simply because the :native
and alike multiarch logic does not know about the internal changes and
is not able to dispatch to the correct kernel recipes. I tried various
workarounds, but it only got more and more complex without any solution
that is able to pass the CI.

3. Make DEPLOY_DIR_SRC architecture specific. This is a easy change,
but means we deploy all source packages multiple times. Along that, it
still means our deployed isar-apt is racy as the last built source
package wins.

Given that, I will send a fix based on (1), along with a an API
changelog. I'm pretty sure, most downstream layers don't have to change
anything. If they only build a single target in a single run, only a
single kernel is built and by that the bug has no effect.

Best regards,
Felix