[PATCH 1/2] dpkg: let do_deploy_deb runtime depend on do_deploy_source

[Felix Moessbauer]

The isar-apt is an artifact that is deployed by isar and can be consumed
as-is as an apt repository by downstream users. As the isar-apt also
provides the debian source packages, we have to ensure that these are
also deployed when the do_dpkg_build task is provided by the SState
cache. Otherwise the isar-apt is not reproducible across runs with and
without the SState cache.

To fix this, we add a runtime dependency between do_deploy_deb and
do_deploy_source, to ensure all sources of our debs are deployed as
well.

Reported-by: Andreas Ziegler <ziegler...@siemens.com>
Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---
RECIPE-API-CHANGELOG.md | 8 ++++++++
meta/classes-recipe/dpkg-source.bbclass | 2 ++
2 files changed, 10 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7e923e55..7386f5d1 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -1146,6 +1146,14 @@ deploy directory `DEPLOY_DIR_DEB` instead of being deployed to the `WORKDIR`.
Recipes that accessed the built debs through `${WORKDIR}/*.deb` (e.g. to unpack an
artifact in a `do_deploy` task) must now reference `${DEPLOY_DIR_DEB}/*.deb` instead.

+### Debian source packages are deployed to `DEPLOY_DIR_SRC`
+
+The source packages produced by `do_dpkg_source` are now exported into a shared,
+sstate-tracked deploy dir `DEPLOY_DIR_SRC` instead of being deployed to the `WORKDIR`.
+
+Recipes that access the source packages through `${WORKDIR}/*.tar.{gz,xz}` must now
+reference the local copy in `${DEPLOY_DIR_SRC}` instead.
+
### Add Hyper-V machine support

A new machine `hyper-v` has been introduced for building images
diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index b2b45ed3..92d06e1b 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -49,6 +49,8 @@ do_deploy_source() {
addtask deploy_source after do_dpkg_source

do_dpkg_build[depends] += "${BPN}:do_deploy_source"
+# ensure that the source package is deployed into isar-apt
+do_deploy_deb[rdepends] += "${BPN}:do_deploy_source"

SCHROOT_MOUNTS = "${WORKDIR}:/work ${REPO_ISAR_DIR}/${DISTRO}:/isar-apt"

--
2.55.0

[Felix Moessbauer]

As the do_dpkg_source task now runs for each corresponding
do_deploy_deb task, we also want to cache the generated results (to
avoid pulling in a much bigger dependency tree on otherwise cached
rebuilds).

For that, we cache the .dsc and the .tar.* artifacts in the deploy dir.
This is implemented similar to how do_dpkg_build artifacts (.debs) are
cached, which is also compatible with a hash equivalence server.

Signed-off-by: Felix Moessbauer <felix.mo...@siemens.com>
---

meta/classes-recipe/dpkg-source.bbclass | 19 ++++++++++++++++---
meta/classes-recipe/dpkg.bbclass | 2 +-
2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/meta/classes-recipe/dpkg-source.bbclass b/meta/classes-recipe/dpkg-source.bbclass
index 92d06e1b..63828a53 100644
--- a/meta/classes-recipe/dpkg-source.bbclass
+++ b/meta/classes-recipe/dpkg-source.bbclass
@@ -13,7 +13,12 @@ TAR_REPRO_OPTS ?= "--exclude=.git --exclude=debian \
DPKG_SOURCE_EXTRA_ARGS ?= "-I"

DEBIAN_SOURCE ?= "${BPN}"
+SRCPKG_DIR = "${WORKDIR}/deploy-srcpkg"
+DEPLOY_DIR_SRC = "${DEPLOY_DIR}/isar-source/${DISTRO}/${BPN}"

+do_dpkg_source[cleandirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-inputdirs] = "${SRCPKG_DIR}"
+do_dpkg_source[sstate-outputdirs] = "${DEPLOY_DIR_SRC}"
do_dpkg_source() {
# Create a .dsc file from source directory to use it with sbuild
DEB_SOURCE_NAME=$(dpkg-parsechangelog --show-field Source --file ${WORKDIR}/${PPS}/debian/changelog)
@@ -21,10 +26,18 @@ do_dpkg_source() {
bbfatal "DEBIAN_SOURCE (${DEBIAN_SOURCE}) not aligned with source name used in control files (${DEB_SOURCE_NAME})"
fi
find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -delete
- sh -c "cd ${WORKDIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${PPS}"
+ sh -c "cd ${SRCPKG_DIR}; dpkg-source ${DPKG_SOURCE_EXTRA_ARGS} -b ${WORKDIR}/${PPS}"
}
addtask dpkg_source after do_prepare_build

+SSTATETASKS += "do_dpkg_source"
+
+python do_dpkg_source_setscene() {
+ sstate_setscene(d)
+}
+
+addtask dpkg_source_setscene
+
CLEANFUNCS += "deb_clean_source"

deb_clean_source() {
@@ -34,11 +47,11 @@ deb_clean_source() {

do_deploy_source[depends] += "isar-apt:do_cache_config"
do_deploy_source[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
-do_deploy_source[dirs] = "${S}"
+do_deploy_source[dirs] = "${S} ${DEPLOY_DIR_SRC}"
do_deploy_source() {
repo_del_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${DEBIAN_SOURCE}"
- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc")
if [ -n "${DSC_FILE}" ]; then
repo_add_srcpackage "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" \
diff --git a/meta/classes-recipe/dpkg.bbclass b/meta/classes-recipe/dpkg.bbclass
index d8e129bb..a42703d5 100644
--- a/meta/classes-recipe/dpkg.bbclass
+++ b/meta/classes-recipe/dpkg.bbclass
@@ -116,7 +116,7 @@ dpkg_runbuild() {
echo '$apt_keep_downloaded_packages = 1;' >> ${SBUILD_CONFIG}
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}

- DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ DSC_FILE=$(find ${DEPLOY_DIR_SRC} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)

# networking is automatically enabled on older versions of sbuild
sbuild_network_option=""
--
2.55.0

[MOESSBAUER, Felix]

This unforunately breaks gbp. Will send a v2.

Felix