[PATCH v2 0/2] ensure source packages are deployed on rebuild with sstate
Felix Moessbauer
- move changelog entry to correct patch
- rework caching logic to not break gbp or apt fetcher interfaces
The series has now been tested in fast CI.
Best regards,
Felix Moessbauer
Felix Moessbauer (2):
dpkg: let do_deploy_deb runtime depend on do_deploy_source
dpkg: cache do_dpkg_source results in sstate
RECIPE-API-CHANGELOG.md | 8 ++++++++
meta/classes-recipe/dpkg-source.bbclass | 21 +++++++++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
3 files changed, 28 insertions(+), 3 deletions(-)
--
2.55.0
Felix Moessbauer
- fix container-loader to build one source package per architecture
- add an API changelog entry regarding architecture-specific source packages
Changes since v1:
- move changelog entry to correct patch
- rework caching logic to not break gbp or apt fetcher interfaces
Best regards,
Felix Moessbauer
Felix Moessbauer (3):
meta/classes-recipe/container-loader.bbclass | 5 ++++
meta/classes-recipe/dpkg-source.bbclass | 21 ++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
4 files changed, 54 insertions(+), 3 deletions(-)
--
2.55.0
Felix Moessbauer
as-is as an apt repository by downstream users. As the isar-apt also
provides the debian source packages, we have to ensure that these are
also deployed when the do_dpkg_build task is provided by the SState
cache. Otherwise the isar-apt is not reproducible across runs with and
without the SState cache.
To fix this, we add a runtime dependency between do_deploy_deb and
do_deploy_source, to ensure all sources of our debs are deployed as
well.
Reported-by: Andreas Ziegler <ziegler...@siemens.com>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
meta/classes-recipe/dpkg-source.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index b2b45ed3..92d06e1b 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -49,6 +49,8 @@ do_deploy_source() {
addtask deploy_source after do_dpkg_source
do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+# ensure that the source package is deployed into isar-apt
+do_deploy_deb[rdepends] += "${BPN}:do_deploy_source"
SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"
--
2.55.0
Felix Moessbauer
content under the same name. This is not allowed in Debian, but remained
silently unnoticed as long as the packages are not deployed into the
same isar-apt.
As a preparation to cache the source packages, we need to ensure all
packages we build are either identical across all architectures, or are
architecture specific, along with having the arch in the package name.
Fixes: cd4e9090 ("container-loader: make generated package arch ...")
meta/classes-recipe/container-loader.bbclass | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/meta/classes-recipe/container-loader.bbclass b/meta/classes-recipe/container-loader.bbclass
index e5c82a74..8439db53 100644
--- a/meta/classes-recipe/container-loader.bbclass
+++ b/meta/classes-recipe/container-loader.bbclass
@@ -10,6 +10,11 @@ SRC_URI += " \
file://container-loader.service.tmpl \
file://container-loader.sh.tmpl"
+# make the source package architecture dependent to avoid
+# package collisions on multiconf / multiarch builds
+PROVIDES := "${BPN}"
+DEBIAN_PROVIDES := "${BPN}"
+PN .= "-${DISTRO_ARCH}"
DPKG_ARCH ?= "${DISTRO_ARCH}"
DEBIAN_MULTI_ARCH ?= "allowed"
--
2.55.0
Felix Moessbauer
do_deploy_deb task, we also want to cache the generated results (to
avoid pulling in a much bigger dependency tree on otherwise cached
rebuilds).
For that, we cache the .dsc and the .tar.* artifacts in the deploy dir.
This is implemented similar to how do_dpkg_build artifacts (.debs) are
cached, which is also compatible with a hash equivalence server.
RECIPE-API-CHANGELOG.md | 29 +++++++++++++++++++++++++
meta/classes-recipe/dpkg-source.bbclass | 19 ++++++++++++++--
meta/classes-recipe/dpkg.bbclass | 2 +-
3 files changed, 47 insertions(+), 3 deletions(-)
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7e923e55..2db05169 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1146,6 +1146,35 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.
+### Debian source packages are deployed to `DEPLOY_DIR_SRC`
+
+The source packages produced by `do_dpkg_source` are now exported into a shared,
+sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`.
+
+Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now
+reference the local copy in `${DEPLOY_DIR_SRC}` instead.
+
+### Architecture specific source packages
+
+Setting `DPKG_ARCH = "<arch>"` makes the source package architecture-specific,
+unlike `all` or `any`. Because Debian source packages are identified by name and
+version, package providers must ensure that a source package with the same name
+and version does not contain different content for different architectures. This
+was never allowed, but Isar now enforces it.
+
+In this case, you can use the following pattern to make the package architecture specific,
+while still maintaining backward compatibility:
+
+```
+inherit dpkg-raw
+DEBIAN_PROVIDES := "${BPN}"
+PN .= "-${DISTRO_ARCH}"
+```
+
+Firmware packages are an exception because they are built for only one architecture.
+
### Add Hyper-V machine support
A new machine `hyper-v` has been introduced for building images
diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index 92d06e1b..97cf9714 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \
DPKG_SOURCE_EXTRA_ARGS ?= "-I"
DEBIAN_SOURCE ?= "${BPN}"
+SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg"
+DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}"
+do_dpkg_source[cleandirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
do_dpkg_source() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
@@ -22,9 +27,19 @@ do_dpkg_source() {
fi
find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
+ # move packages to deploy directory
+ find ${WORKDIR} -maxdepth 1 \( -name "${DEBIAN_SOURCE}_*.tar.*" -o -name "${DEBIAN_SOURCE}_*.dsc" \) -exec mv {} ${SRCPKG_DIR}/ \;
}
addtask dpkg_source after do_prepare_build
+SSTATETASKS += "do_dpkg_source"
+
+python do_dpkg_source_setscene() {
+ sstate_setscene(d)
+}
+
+addtask dpkg_source_setscene
+
CLEANFUNCS += "deb_clean_source"
deb_clean_source() {
@@ -34,11 +49,11 @@ deb_clean_source() {
do_deploy_source[depends] += "isar-apt:do_cache_config"
do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_source[dirs] = "${S}"
+do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}"
do_deploy_source() {
repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
if [ -n "${DSC_FILE}" ]; then
repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index d8e129bb..a42703d5 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -116,7 +116,7 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
# networking is automatically enabled on older versions of sbuild
sbuild_network_option=""
--
2.55.0
Zhihang Wei
Zhihang
MOESSBAUER, Felix
Hi,
this series revealed another bug regarding source packages with same
name but different content:
Similar to upstream Debian, our kernel source packages are architecture
specific. By that, we also have to encode the architecture (or machine)
in the name of the package. Otherwise existing packages might get
reused on rebuilds with other targets (or multiarch)
In isar, this has ever been modeled incorrectly (the kernels .dsc file
was arch=any despite being arch specific). This is easy to fix, but
also depending on the kernels was done incorrectly (a KERNEL_NAME =
mainline is not specific enough, as mainline on arm64 is technically a
different kernel (source package) than mainline on amd64.
To fix this, we have the following options (which I tried):
1. encode the MACHINE or DISTRO_ARCH in KERNEL_NAME. Usually arch is
enough, but if a machine specific defconfig is used, the machine needs
to be encoded in the name => clean, works but might require downstream
changes
2. internally generate kernel source package with -${DISTRO_ARCH}
suffix, but keep DEPENDS / PROVIDES and binary package names as-is:
This is different to how Debian names the kernels. For the kernel build
itself this can be modeled, but the dependencies of custom modules back
to the "correct" kernel cannot be modeled, simply because the :native
and alike multiarch logic does not know about the internal changes and
is not able to dispatch to the correct kernel recipes. I tried various
workarounds, but it only got more and more complex without any solution
that is able to pass the CI.
3. Make DEPLOY_DIR_SRC architecture specific. This is a easy change,
but means we deploy all source packages multiple times. Along that, it
still means our deployed isar-apt is racy as the last built source
package wins.
Given that, I will send a fix based on (1), along with a an API
changelog. I'm pretty sure, most downstream layers don't have to change
anything. If they only build a single target in a single run, only a
single kernel is built and by that the bug has no effect.
Best regards,
Felix
MOESSBAUER, Felix
wrote:
Hi, after some extensive tests, I conclude that this is the only
feasible approach. The API change is also quite minimal, as we
technically don't even change an API. We just tell the users how to
correctly name their packages - which they theoretically should already
have done correctly.
>
> 2. internally generate kernel source package with -${DISTRO_ARCH}
> suffix, but keep DEPENDS / PROVIDES and binary package names as-is:
> This is different to how Debian names the kernels. For the kernel build
> itself this can be modeled, but the dependencies of custom modules back
> to the "correct" kernel cannot be modeled, simply because the :native
> and alike multiarch logic does not know about the internal changes and
> is not able to dispatch to the correct kernel recipes. I tried various
> workarounds, but it only got more and more complex without any solution
> that is able to pass the CI.
I tried this, but it does it comes with severe limitations. The main
issue is that we have to add additional provides, but just for the base
variant of the recipe. Detecting what the base variant is either
requires heuristics (bad!) or immediate expansion of BPN on first
parsing. This however breaks if kernel recipes are inherited / included
as then the immediate expansion always results in the first name
instead of the final one.
It also is problematic as we now create the conflicts on the binary
packages.
>
> 3. Make DEPLOY_DIR_SRC architecture specific. This is a easy change,
> but means we deploy all source packages multiple times. Along that, it
> still means our deployed isar-apt is racy as the last built source
> package wins.
Possible, but a lot of disk-space waste and against the single-source
package principles.
Felix
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/isar-users/b26fcda922ab9ab2341f11e5c951089ce8bc1fcc.camel%40siemens.com.