[PATCH v4 00/12] Sbuild/Schroot migration

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:17 AM1/25/22

to isar-...@googlegroups.com

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, official `kas` images don't include all required software, so
there was an alternative docker image prepared:
`docker pull ghcr.io/wiselord/kas/kas-isar:next-sbuild`

To use it with KAS:
- export KAS_IMAGE_VERSION=3.0.0-sbuild
- export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord/kas

If 'gitlab' is used, the similar changes are required:
- 'image: ghcr.io/wiselord/kas/kas-isar:3.0.0-sbuild' in .gitlab-ci.yml
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams. Some patches are required for these downstreams,
appropriate pull requests will be created soon.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,
Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
dpkg: Build packages with sbuild

Uladzimir Bely (11):
dpkg-gbp: Use separate command to export tarball
dpkg-gbp: Use host tools for dsc preparation
sbuild: Add recipes for host and target rootfs to run sbuild
sbuild: Introduce a class for another build method
sbuild: support of DEB_BUILD_PROFILES
sbuild: add ccache support
dpkg: Remove builddeps install task.
dpkg-base: Switch devshell to use schroot
dpkg-base: Switch apt_fetch and apt_unpack to use schroot
dpkg-base: Cleanup from buildchroot parts.
doc: Add sbuild-related documentation

doc/user_manual.md | 22 ++-
meta/classes/dpkg-base.bbclass | 81 ++++----
meta/classes/dpkg-gbp.bbclass | 26 +--
meta/classes/dpkg.bbclass | 62 +++---
meta/classes/sbuild.bbclass | 185 ++++++++++++++++++
meta/conf/bitbake.conf | 2 +
.../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 +
.../sbuild-chroot/sbuild-chroot.inc | 39 ++++
9 files changed, 359 insertions(+), 81 deletions(-)
create mode 100644 meta/classes/sbuild.bbclass
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:18 AM1/25/22

to isar-...@googlegroups.com

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index caaab8cc..97270167 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -79,7 +79,9 @@ apt install \
qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```

If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"

PATCHTOOL ?= "git"

-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

-do_install_builddeps_append() {
- dpkg_do_mounts
- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
-}
-
dpkg_runbuild_prepend() {
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ sh -c "
+ cd ${WORKDIR}/${PPS}
+ gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+ "
# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
# for compatibility with gbp version froms debian-stretch. In newer distros
# it's possible to use a subcommand `export-orig --pristine-tar`
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:18 AM1/25/22

to isar-...@googlegroups.com

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
3 files changed, 59 insertions(+)

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+ fakeroot \
+ build-essential \
+ debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+ ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:18 AM1/25/22

to isar-...@googlegroups.com

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
`cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 2 +
2 files changed, 162 insertions(+)
create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@

+# This software is a part of ISAR.

+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+ import pwd
+ d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+ d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+ d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ dep = "sbuild-chroot-target:do_build"
+ else:
+ d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ dep = "sbuild-chroot-host:do_build"
+ d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+ cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+ mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+ cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+ # Prepare mount points
+ cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+ if [ -d ${DL_DIR} ]; then
+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+ if [ -d "${SBUILD_CONF_DIR}" ]; then
+ rm -rf "${SBUILD_CONF_DIR}"
+ fi
+ rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${SCHROOT_CONF_FILE}"
+ rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+ VAR=${1}; shift
+ VAR_LINE="'${VAR}' => '${@}',"
+ if [ -s "${SBUILD_CONFIG}" ]; then
+ sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+ else
+ echo "\$build_environment = {" > ${SBUILD_CONFIG}
+ echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ echo "};" >> ${SBUILD_CONFIG}
+ fi
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
+
+schroot_install() {
+ schroot_create_configs
+ APTS="$1"
+
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+
+ deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends --download-only ${APTS}
+ deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends ${APTS}
+ schroot_delete_configs
+}
+
+insert_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+remove_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+schroot_run() {
+ schroot_create_configs
+ insert_mounts
+ schroot $@
+ remove_mounts
+ schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "vmlinuz"
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:18 AM1/25/22

to isar-...@googlegroups.com

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
}

dpkg_runbuild_prepend() {
- export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"

+ # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used

+ # for compatibility with gbp version froms debian-stretch. In newer distros
+ # it's possible to use a subcommand `export-orig --pristine-tar`
}
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:18 AM1/25/22

to isar-...@googlegroups.com

From: Anton Mikanovich <ami...@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 16 +++++++++++++++-
meta/classes/dpkg-gbp.bbclass | 2 ++
meta/classes/dpkg.bbclass | 13 +++++++++++--
3 files changed, 28 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 76b1290a..e1c27b9e 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT

+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""

DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"

+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
python do_adjust_git() {
import subprocess

@@ -196,13 +199,22 @@ dpkg_undo_mounts() {
sudo rmdir ${BUILDROOT}
}

+do_prepare_build_append() {
+ # Make a local copy of isar-apt repo that is not affected by other parallel builds
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
# Basic export implementation supporting multiword values
dpkg_build_export() {
var=$1
shift
value=$@

- export $var="$value"
+ sbuild_export $var "$value"
}

# Placeholder for actual dpkg_runbuild() implementation
@@ -250,6 +262,8 @@ python do_dpkg_build_setscene() {
addtask dpkg_build_setscene
do_dpkg_build_setscene[dirs] += "${S}/.."

+do_dpkg_build[depends] = "${SCHROOT_DEP}"
+
KEEP_INSTALLED_ON_CLEAN ?= "0"

CLEANFUNCS += "deb_clean"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"

GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {
sh -c "

cd ${WORKDIR}/${PPS}

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index e8c7e2eb..66539f48 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,11 +29,20 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

addtask devshell after do_install_builddeps

+
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
E="${@ isar_export_ccache(d)}"
dpkg_build_export PARALLEL_MAKE "${PARALLEL_MAKE}"
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+
+ schroot_create_configs
+
+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --debbuildopts="--source-option=-I" \
+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+ schroot_delete_configs
}
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:19 AM1/25/22

to isar-...@googlegroups.com

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 66539f48..8647d077 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -38,8 +38,16 @@ dpkg_runbuild() {

schroot_create_configs

+ profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

+ profiles="${profiles} cross nocheck"
+ fi
+ if [ ! -z "$profiles" ]; then
+ profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+ fi

+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

- --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
--debbuildopts="--source-option=-I" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:19 AM1/25/22

to isar-...@googlegroups.com

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 22 +++++++++++++++++++
.../sbuild-chroot/sbuild-chroot.inc | 1 +
3 files changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 8647d077..e761078d 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -38,6 +38,10 @@ dpkg_runbuild() {

schroot_create_configs

+ if [ ${USE_CCACHE} -eq 1 ]; then
+ schroot_configure_ccache
+ fi

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

profiles="${profiles} cross nocheck"

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..c9676bdf 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,25 @@ schroot_run() {
remove_mounts
schroot_delete_configs
}
+
+schroot_configure_ccache() {

+ sudo -s <<'EOSUDO'
+ set -e
+

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+ cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export CCACHE_DIR=/ccache
+export PATH="/usr/lib/ccache:\$PATH"
+exec "\$@"
+END
+ chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+ echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
build-essential \
debhelper \
+ ccache \
"

SBUILD_CHROOT_PREINSTALL ?= " \
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:20 AM1/25/22

to isar-...@googlegroups.com

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4f8c3256..ea914cd4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -99,19 +99,12 @@ python() {
}

do_apt_fetch() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_fetch after do_unpack before do_apt_unpack
@@ -120,13 +113,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
# Add dependency from the correct buildchroot: host or target
do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"

+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
do_apt_unpack() {
rm -rf ${S}
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -135,8 +129,7 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_unpack after do_apt_fetch before do_patch
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:20 AM1/25/22

to isar-...@googlegroups.com

This removes remaining buildchroot-related tasks and variables
from dpkg-base class while there are not used anymore.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 19 +++----------------
meta/classes/dpkg.bbclass | 4 ----
2 files changed, 3 insertions(+), 20 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ea914cd4..513d5217 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -167,6 +167,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"

BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"

@@ -216,14 +217,11 @@ dpkg_runbuild() {
}

python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)
- bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func('schroot_create_configs', d)
try:
bb.build.exec_func("dpkg_runbuild", d)
finally:
- bb.build.exec_func("dpkg_undo_mounts", d)
- bb.utils.unlockfile(lock)
+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask dpkg_build
@@ -257,8 +255,6 @@ do_dpkg_build_setscene[dirs] += "${S}/.."

do_dpkg_build[depends] = "${SCHROOT_DEP}"

-KEEP_INSTALLED_ON_CLEAN ?= "0"
-
CLEANFUNCS += "deb_clean"

deb_clean() {
@@ -267,15 +263,6 @@ deb_clean() {
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 38845a16..99d1a074 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -13,8 +13,6 @@ dpkg_runbuild() {

E="${@ isar_export_ccache(d)}"
dpkg_build_export PARALLEL_MAKE "${PARALLEL_MAKE}"

- schroot_create_configs
-
distro="${DISTRO}"

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

distro="${HOST_DISTRO}"
@@ -52,6 +50,4 @@ dpkg_runbuild() {

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-
- schroot_delete_configs
}
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:20 AM1/25/22

to isar-...@googlegroups.com

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 43 ++++++++++++++++++---------------------
1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index e761078d..38845a16 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,7 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-# Install build dependencies for package
-do_install_builddeps() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-

# Build package from sources using build script
dpkg_runbuild() {

@@ -38,6 +15,21 @@ dpkg_runbuild() {

schroot_create_configs

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives/"
+ ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+ ( flock 9
+ grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+ ) 9>"${TMPDIR}/sbuildrc.lock"

+
if [ ${USE_CCACHE} -eq 1 ]; then

schroot_configure_ccache
fi
@@ -53,8 +45,13 @@ dpkg_runbuild() {

sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--debbuildopts="--source-option=-I" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
schroot_delete_configs
}
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:20 AM1/25/22

to isar-...@googlegroups.com

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 23 ++++++++++++++-----
meta/classes/sbuild.bbclass | 3 +++
.../sbuild-chroot/sbuild-chroot.inc | 2 ++
3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e1c27b9e..4f8c3256 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -300,18 +300,29 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[dirs] = "${S}"

python do_devshell() {

- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)

isar_export_proxies(d)
isar_export_ccache(d)

- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
+ isar_apt = d.getVar('ISAR_APT_REPO')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- # the PATH variable is not forwarded by sudo -E.
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)

- bb.build.exec_func('dpkg_undo_mounts', d)

+ install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+ \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+ debian/control"
+
+ termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+ cd {1}; \
+ echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+ apt-get -y -q update; \
+ {3}; \
+ $SHELL -i \
+ '"
+ oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index c9676bdf..b4024852 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,6 +76,9 @@ EOF

fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"

grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}

+
if [ -d ${DL_DIR} ]; then

fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -15,6 +15,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
build-essential \
debhelper \
ccache \
+ devscripts \
+ equivs \
"

SBUILD_CHROOT_PREINSTALL ?= " \
--
2.20.1

Uladzimir Bely

unread,

Jan 25, 2022, 7:39:21 AM1/25/22

to isar-...@googlegroups.com

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 97270167..17758d17 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -84,6 +84,24 @@ apt install \
pristine-tar
```

+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+

If your host is >= buster, also install the following package.

```
apt install python3-distutils
--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:00:44 PM2/1/22

to isar-...@googlegroups.com

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, sbuild-related stuff is supported in kas-isar:next image

To use the patchset with KAS:
- export KAS_IMAGE_VERSION=next

If 'gitlab' is used, the similar changes are required:

- `next` tag for kas image should be set

- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'

downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,
Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
dpkg: Build packages with sbuild

Uladzimir Bely (11):
dpkg-gbp: Use separate command to export tarball
dpkg-gbp: Use host tools for dsc preparation
sbuild: Add recipes for host and target rootfs to run sbuild
sbuild: Introduce a class for another build method
sbuild: support of DEB_BUILD_PROFILES

sbuild: support of shell exports from dpkg_runbuild_prepend

dpkg: Remove builddeps install task.

sbuild: add ccache support

dpkg-base: Switch devshell to use schroot
dpkg-base: Switch apt_fetch and apt_unpack to use schroot

doc: Add sbuild-related documentation

doc/user_manual.md | 22 ++-

meta/classes/dpkg-base.bbclass | 79 ++++----
meta/classes/dpkg-gbp.bbclass | 26 +--
meta/classes/dpkg.bbclass | 80 +++++---
meta/classes/sbuild.bbclass | 184 ++++++++++++++++++

meta/conf/bitbake.conf | 2 +
.../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 +
.../sbuild-chroot/sbuild-chroot.inc | 39 ++++

9 files changed, 375 insertions(+), 80 deletions(-)
create mode 100644 meta/classes/sbuild.bbclass

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:00:49 PM2/1/22

to isar-...@googlegroups.com

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass

Uladzimir Bely

unread,

Feb 1, 2022, 12:00:55 PM2/1/22

to isar-...@googlegroups.com

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 4d4fa763..df73ef89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -80,7 +80,9 @@ apt install \

qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```

If your host is >= buster, also install the following package.

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass

@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"

PATCHTOOL ?= "git"

-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

-do_install_builddeps_append() {
- dpkg_do_mounts

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
-}
-
dpkg_runbuild_prepend() {

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ sh -c "
+ cd ${WORKDIR}/${PPS}
+ gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}

+ "
# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used

# for compatibility with gbp version froms debian-stretch. In newer distros

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:00 PM2/1/22

to isar-...@googlegroups.com

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
3 files changed, 59 insertions(+)

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644

index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb

new file mode 100644

index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

new file mode 100644

index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#

+# This software is a part of ISAR.

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:01 PM2/1/22

to isar-...@googlegroups.com

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
`cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 2 +
2 files changed, 162 insertions(+)
create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass

new file mode 100644

index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@

+# This software is a part of ISAR.

+ sudo -s <<'EOSUDO'
+ set -e
+

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+

+ if [ -d ${DL_DIR} ]; then

+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {

+ sudo -s <<'EOSUDO'
+ set -e

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends --download-only ${APTS}
+ deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \

+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends ${APTS}
+ schroot_delete_configs
+}
+
+insert_mounts() {

+ sudo -s <<'EOSUDO'
+ set -e

+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+remove_mounts() {

+ sudo -s <<'EOSUDO'
+ set -e

+ for mp in ${SCHROOT_MOUNTS}; do

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:06 PM2/1/22

to isar-...@googlegroups.com

From: Anton Mikanovich <ami...@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
meta/classes/dpkg-gbp.bbclass | 2 ++
meta/classes/dpkg.bbclass | 10 ++++++++--
meta/classes/sbuild.bbclass | 4 ++--
4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2add0b22..048a09c0 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass

@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT

+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""

DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"

+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
python do_adjust_git() {
import subprocess

@@ -171,6 +174,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build

# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"

BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"

@@ -196,20 +200,26 @@ dpkg_undo_mounts() {

sudo rmdir ${BUILDROOT}
}

+do_prepare_build_append() {
+ # Make a local copy of isar-apt repo that is not affected by other parallel builds
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+

+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+

# Placeholder for actual dpkg_runbuild() implementation

dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"

}

python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)

- bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func('schroot_create_configs', d)

try:
bb.build.exec_func("dpkg_runbuild", d)
finally:

- bb.build.exec_func("dpkg_undo_mounts", d)

- bb.utils.unlockfile(lock)

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask dpkg_build
@@ -241,7 +251,7 @@ python do_dpkg_build_setscene() {
addtask dpkg_build_setscene

do_dpkg_build_setscene[dirs] += "${S}/.."

-KEEP_INSTALLED_ON_CLEAN ?= "0"

+do_dpkg_build[depends] = "${SCHROOT_DEP}"

CLEANFUNCS += "deb_clean"

@@ -251,15 +261,6 @@ deb_clean() {

for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass

@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"

GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {
sh -c "

cd ${WORKDIR}/${PPS}

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 27fe84f4..d7afea18 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -34,6 +34,12 @@ dpkg_runbuild() {

E="${@ isar_export_proxies(d)}"
E="${@ isar_export_ccache(d)}"

export PARALLEL_MAKE="${PARALLEL_MAKE}"

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+

+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+

+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --debbuildopts="--source-option=-I" \

+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..6fff664f 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,6 +31,8 @@ SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"

SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"

SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"

+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
schroot_create_configs() {
sudo -s <<'EOSUDO'
set -e
@@ -96,7 +98,6 @@ EOSUDO
}

sbuild_export() {
- SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
VAR=${1}; shift

VAR_LINE="'${VAR}' => '${@}',"

if [ -s "${SBUILD_CONFIG}" ]; then

@@ -106,7 +107,6 @@ sbuild_export() {

echo "${VAR_LINE}" >> ${SBUILD_CONFIG}

echo "};" >> ${SBUILD_CONFIG}
fi
- export SBUILD_CONFIG="${SBUILD_CONFIG}"
}

schroot_install() {
--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:06 PM2/1/22

to isar-...@googlegroups.com

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d7afea18..66db7ec5 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -35,10 +35,18 @@ dpkg_runbuild() {

E="${@ isar_export_ccache(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

+ profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

+ profiles="${profiles} cross nocheck"
+ fi
+ if [ ! -z "$profiles" ]; then
+ profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+ fi

+
export SBUILD_CONFIG="${SBUILD_CONFIG}"

sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

- --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
--debbuildopts="--source-option=-I" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:11 PM2/1/22

to isar-...@googlegroups.com

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 44 ++++++++++++++++++---------------------
1 file changed, 20 insertions(+), 24 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index c252e9b3..8f311deb 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,6 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-# Install build dependencies for package
-do_install_builddeps() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build

-do_install_builddeps[depends] += "isar-apt:do_cache_config"

-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-

DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"

do_prepare_build_append() {
@@ -53,6 +29,21 @@ dpkg_runbuild() {
sbuild_export $var "$value"
done

+ distro="${DISTRO}"

+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives/"
+ ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+ ( flock 9
+ grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+ ) 9>"${TMPDIR}/sbuildrc.lock"

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

profiles="${profiles} cross nocheck"

@@ -66,6 +57,11 @@ dpkg_runbuild() {

sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \

--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+

+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
}

--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:11 PM2/1/22

to isar-...@googlegroups.com

Many of recipes often use shell exports done in dpkg_run_prepend, so that
this changed environment is used during build.

While sbuild is performed in isolated environment, we need a way to
pass these variables to it. This is done by storing environment before
dpkg_runbuild (after prepare_build) and finding just before the actual
build what was changed or added.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 18 ++++++++++++++++++

1 file changed, 18 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 66db7ec5..c252e9b3 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,12 +29,30 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

addtask devshell after do_install_builddeps

+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+ env > ${DPKG_PREBUILD_ENV_FILE}
+}
+

# Build package from sources using build script

dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"

E="${@ isar_export_ccache(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

+ env | while read -r line; do
+ # Filter the same lines
+ grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+ # Filter some standard variables
+ echo ${line} | grep -q "^HOME=" && continue
+ echo ${line} | grep -q "^PWD=" && continue
+
+ var=$(echo "${line}" | cut -d '=' -f1)
+ value=$(echo "${line}" | cut -d '=' -f2-)
+ sbuild_export $var "$value"
+ done

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
profiles="${profiles} cross nocheck"

--
2.20.1

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:15 PM2/1/22

to isar-...@googlegroups.com

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 21 +++++++++++++++++++
.../sbuild-chroot/sbuild-chroot.inc | 1 +
3 files changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 8f311deb..ed0e6e35 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -44,6 +44,10 @@ dpkg_runbuild() {

echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc

) 9>"${TMPDIR}/sbuildrc.lock"

+ if [ ${USE_CCACHE} -eq 1 ]; then

+ schroot_configure_ccache
+ fi

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
profiles="${profiles} cross nocheck"

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6fff664f..9896b772 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,24 @@ schroot_run() {
remove_mounts
schroot_delete_configs
}
+
+schroot_configure_ccache() {

+ sudo -s <<'EOSUDO'
+ set -e
+

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+ cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh

+export PATH="\$PATH_PREPEND:\$PATH"

+exec "\$@"
+END
+ chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+ echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:21 PM2/1/22

to isar-...@googlegroups.com

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 23 ++++++++++++++-----
meta/classes/sbuild.bbclass | 3 +++
.../sbuild-chroot/sbuild-chroot.inc | 2 ++

3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 048a09c0..b1a16f67 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -278,18 +278,29 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

do_deploy_deb[dirs] = "${S}"

python do_devshell() {

- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)

isar_export_proxies(d)
isar_export_ccache(d)

- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
+ isar_apt = d.getVar('ISAR_APT_REPO')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- # the PATH variable is not forwarded by sudo -E.
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)

- bb.build.exec_func('dpkg_undo_mounts', d)

+ install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+ \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+ debian/control"
+
+ termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+ cd {1}; \
+ echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+ apt-get -y -q update; \
+ {3}; \
+ $SHELL -i \
+ '"
+ oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 9896b772..eb358c71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -78,6 +78,9 @@ EOF

fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"

grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}

+
if [ -d ${DL_DIR} ]; then

fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 74811e22..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:27 PM2/1/22

to isar-...@googlegroups.com

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index b1a16f67..39f20263 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass

@@ -99,19 +99,12 @@ python() {
}

do_apt_fetch() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_fetch after do_unpack before do_apt_unpack
@@ -120,13 +113,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
# Add dependency from the correct buildchroot: host or target
do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"

+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
do_apt_unpack() {
rm -rf ${S}

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-

+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

Uladzimir Bely

unread,

Feb 1, 2022, 12:01:31 PM2/1/22

to isar-...@googlegroups.com

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 18 ++++++++++++++++++

1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index df73ef89..966e7125 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -85,6 +85,24 @@ apt install \

pristine-tar
```

+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+

If your host is >= buster, also install the following package.

Jan Kiszka

unread,

Feb 1, 2022, 1:09:11 PM2/1/22

to Uladzimir Bely, isar-...@googlegroups.com

So, this basically decouples "Avoid using shell environment during the
build" and similar conversions downstream from this series, right? It's
indeed good to have a compat path now.

However, what are the patterns we want to push? Avoiding exports? Then
we should probably warn here that this compat path should be avoided and
might be removed in the future.

If we want to keep both, we can probably leave several recipes alone in
that other series.

Jan

--
Siemens AG, Technology
Competence Center Embedded Linux

Jan Kiszka

unread,

Feb 1, 2022, 1:14:16 PM2/1/22

to Uladzimir Bely, isar-...@googlegroups.com

On 01.02.22 18:00, Uladzimir Bely wrote:

Does this functionally obsoletes buildchroot completely? I see that the
recipes are still there. If they aren't used anymore by core Isar, only
kept to make some special, not-yet-converted downstream layers happy,
they may still quickly regress.

Uladzimir Bely

unread,

Feb 1, 2022, 1:33:40 PM2/1/22

to isar-...@googlegroups.com, Jan Kiszka

In the email from Tuesday, 1 February 2022 21:14:12 +03 user Jan Kiszka wrote:
> Does this functionally obsoletes buildchroot completely? I see that the
> recipes are still there. If they aren't used anymore by core Isar, only
> kept to make some special, not-yet-converted downstream layers happy,
> they may still quickly regress.

Not competely. Sbuild is used only for _building_ packages.
Buildchroot is still used, for example, by image.bbclass.

Also, in meta-iot2050 npm.bbclass still uses buildchroot mounts.

Uladzimir Bely

unread,

Feb 1, 2022, 1:41:28 PM2/1/22

to isar-...@googlegroups.com, Jan Kiszka

You might be right. Current solution doesn't force users to avoid using
exports while they are silently supported.

The first internal solution was keeping dpkg_build_export function but also
passing custom shell exports (from dpkg_runbuild_prepend) to sbuild
environment. And a warning about migration was shown.

Later I removed this to make new patchset less invasive for downstreams.
Anyway, if we agree that "dpkg_build_export" (like API function) is more
preferable, I can get back this solution with warnings.

Jan Kiszka

unread,

Feb 2, 2022, 1:57:28 AM2/2/22

to Uladzimir Bely, isar-...@googlegroups.com

This is not necessarily what I'm suggesting. Your conversion series
works without that API, primarily using template. We also have build
profiles. But we first of all need to define what we actually want.

The benefit of pushing things into the build env, rather than feeding it
from the recipe, is likely that the resulting deb-src of such generated
packages can actually be used for rebuilding independently of Isar. So
there is likely value in deprecating classic export.

Jan Kiszka

unread,

Feb 2, 2022, 2:02:02 AM2/2/22

to Uladzimir Bely, isar-...@googlegroups.com

On 01.02.22 19:33, Uladzimir Bely wrote:
> In the email from Tuesday, 1 February 2022 21:14:12 +03 user Jan Kiszka wrote:
>> Does this functionally obsoletes buildchroot completely? I see that the
>> recipes are still there. If they aren't used anymore by core Isar, only
>> kept to make some special, not-yet-converted downstream layers happy,
>> they may still quickly regress.
>
> Not competely. Sbuild is used only for _building_ packages.
> Buildchroot is still used, for example, by image.bbclass.
>

So, buildchroot becomes "imagechroot" - no longer used for _building_.

> Also, in meta-iot2050 npm.bbclass still uses buildchroot mounts.
>

Then this needs to be fully converted to sbuild, also regarding the
"install deps via npm" step. That analogous to things like
"linux-custom: Move cfg fragments applying to debian/rules".

Moessbauer, Felix

unread,

Feb 2, 2022, 3:46:56 AM2/2/22

to jan.k...@siemens.com, Uladzimir Bely, isar-...@googlegroups.com

> -----Original Message-----
> From: isar-...@googlegroups.com <isar-...@googlegroups.com> On
> Behalf Of Jan Kiszka
> Sent: Wednesday, February 2, 2022 7:57 AM
> To: Uladzimir Bely <ub...@ilbers.de>; isar-...@googlegroups.com
> Subject: Re: [PATCH v5 07/12] sbuild: support of shell exports from
> dpkg_runbuild_prepend
>

Using environment variables to control the build in Debian/rules is generally discouraged (except for DEB_BUILD_PROFILES and DEB_BUILD_OPTIONS).
But even for these variables, semantics are documented [1] and we / an ISAR user should strictly follow them when packaging.

There are also reasons why you want to build a package in a clean environment (e.g. to avoid accidentally setting CC or CXX to icc or clang).
Hence, I highly vote for not forwarding the environment into the sbuild build environment.

Felix

[1] https://wiki.debian.org/BuildProfileSpec#Registered_profile_names

>
> Jan
>
> --
> Siemens AG, Technology
> Competence Center Embedded Linux
>

> --
> You received this message because you are subscribed to the Google Groups
> "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email
> to isar-users+...@googlegroups.com.
> To view this discussion on the web visit
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.g
> oogle.com%2Fd%2Fmsgid%2Fisar-users%2F9f8b4874-62b5-546f-5dc9-
> 8d9c0bd5b1cf%2540siemens.com&data=04%7C01%7Cfelix.moessbauer%4
> 0siemens.com%7C9ed643fe46b945b7d2a008d9e619471f%7C38ae3bcd95794fd
> 4addab42e1495d55a%7C1%7C0%7C637793819407336907%7CUnknown%7CTW
> FpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI
> 6Mn0%3D%7C3000&sdata=Wfb%2F4NfV1M9LjodozJ1V5p3riY8HHobGS31
> %2B6gJ7y6c%3D&reserved=0.

Jan Kiszka

unread,

Feb 2, 2022, 3:52:34 AM2/2/22

to Moessbauer, Felix (T CED SES-DE), Uladzimir Bely, isar-...@googlegroups.com

...without a warning about a grace period during that we will still do
that to allow downstream migration. Then we have a plan, I would say.

Uladzimir Bely

unread,

Feb 2, 2022, 4:13:32 AM2/2/22

to Moessbauer, Felix (T CED SES-DE), isar-...@googlegroups.com, Jan Kiszka

In the email from Wednesday, 2 February 2022 11:52:29 +03 user Jan Kiszka

To make things clear, there are now three options:

1) (Current) Just support passing shell exports user done in
dpkg_runbuild_prepend to (s)build environment. Users of downstreams are not
notified, so they continue using these exports.

2) Continue support custom shell exports by passing them to (s)build env, BUT
show warnings, that user should use some 'dpkg_build_export'. In general, this
doesn't differ much from the previous approach.

3) Continue support custom shell exports by passing them to (s)build env, BUT
show warnings, that this stuff should be converted to templates. After some
period, stop supporting exports.

4) Doesn't support shell exports passing to (s)build env, so it will
immediately break the builds.

As I understand correctly, the way (3) is the best solution.

--
Uladzimir Bely

Jan Kiszka

unread,

Feb 2, 2022, 4:46:15 AM2/2/22

to Uladzimir Bely, Moessbauer, Felix (T CED SES-DE), isar-...@googlegroups.com

Templates or build profiles.

>
> 4) Doesn't support shell exports passing to (s)build env, so it will
> immediately break the builds.
>
> As I understand correctly, the way (3) is the best solution.
>

That is also my understanding.

Thanks,

quirin.g...@siemens.com

unread,

Feb 4, 2022, 8:39:37 AM2/4/22

to isar-...@googlegroups.com, ub...@ilbers.de

Hi,

On 1/25/22 13:39, Uladzimir Bely wrote:
> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
>
> This patchset should be applied on top of "preparation" patchset
> named "Avoid using shell environment during the build".
>
> Base schroot image is created with sbuild-chroot-target (or
> sbuild-chroot-host in case of cross-build) recipe. These images are based
> on isar-bootstrap ones, but include some build-related stuff preinstalled.
>
> To use this changes you need to have sbuild and schroot installed and
> configured. Current user should be added to sbuild group.
>
> If 'kas-container' is used, it should be modified to support sbuild:
> - /var/lib/schroot/union/overlay should be externally mounted (-v option)
>

> Currently, official `kas` images don't include all required software, so
> there was an alternative docker image prepared:
> `docker pull ghcr.io/wiselord/kas/kas-isar:next-sbuild`
>
> To use it with KAS:
> - export KAS_IMAGE_VERSION=3.0.0-sbuild
> - export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord/kas

>
> If 'gitlab' is used, the similar changes are required:

> - 'image: ghcr.io/wiselord/kas/kas-isar:3.0.0-sbuild' in .gitlab-ci.yml

> - external dir for schroot overlay should also be specified in
> /etc/gitlab-runner/config.toml:
> volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]
>
> The patchset was tested with 'meta-iot2050' and 'xenomai-images'

> downstreams. Some patches are required for these downstreams,
> appropriate pull requests will be created soon.

>
> Current limitations:
> - parallel building of several packages requires more free space in
> comparison with buildchroot-based architecture. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are downloaded
> and installed).
>

> Changes since v3:
> - dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
> compatibility;
> - dpkg_build_export is used for adjusting sbuild environment;
> - DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.
> Changes since v2:
> - patches reworked/squashed for easier reading and understanding;
> - fixed building foreigh architectures with kas-docker;
> - implemented support of ccache;
> - fixed devshell and devshell_nodeps,
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported;
> - parallel multiconfig targets in one build are supported;
> - per-task schroot configuration in /etc/schroot/ is now used;
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH).
>
> Anton Mikanovich (1):
> dpkg: Build packages with sbuild
>
> Uladzimir Bely (11):
> dpkg-gbp: Use separate command to export tarball
> dpkg-gbp: Use host tools for dsc preparation
> sbuild: Add recipes for host and target rootfs to run sbuild
> sbuild: Introduce a class for another build method
> sbuild: support of DEB_BUILD_PROFILES

> sbuild: add ccache support

> dpkg: Remove builddeps install task.

> dpkg-base: Switch devshell to use schroot
> dpkg-base: Switch apt_fetch and apt_unpack to use schroot

> dpkg-base: Cleanup from buildchroot parts.

> doc: Add sbuild-related documentation
>
> doc/user_manual.md | 22 ++-

> meta/classes/dpkg-base.bbclass | 81 ++++----
> meta/classes/dpkg-gbp.bbclass | 26 +--
> meta/classes/dpkg.bbclass | 62 +++---
> meta/classes/sbuild.bbclass | 185 ++++++++++++++++++

> meta/conf/bitbake.conf | 2 +
> .../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
> .../sbuild-chroot/sbuild-chroot-target.bb | 10 +
> .../sbuild-chroot/sbuild-chroot.inc | 39 ++++

> 9 files changed, 359 insertions(+), 81 deletions(-)

> create mode 100644 meta/classes/sbuild.bbclass
> create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
> create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
> create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
>

any thoughts on the clean source behavior[1].

"When running sbuild from within an unpacked source tree, run the
'clean' target before generating the source package. This might require
some of the build dependencies necessary for running the 'clean' target
to be installed on the host machine. Only disable if you start from a
clean checkout and you know what you are doing."[1]
[1]:
https://manpages.debian.org/unstable/sbuild/sbuild.conf.5.en.html#CLEAN_SOURCE

If this is necessary you need to install dh-*(e.g. dh-lua, dh-python) in
the build environment.

Quirin

Jan Kiszka

unread,

Feb 4, 2022, 9:28:55 AM2/4/22

to quirin.g...@siemens.com, isar-...@googlegroups.com, ub...@ilbers.de, Schmidl, Tobias (T CED SES-DE)

Then we would need to expand the host side (builder) dependency list.
And also kas-isar.

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:22 AM2/10/22

to isar-...@googlegroups.com

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

This patchset should be applied on top of "preparation" patchset
named "Avoid using shell environment during the build".

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)

Currently, sbuild-related stuff is supported in kas-isar:next image

To use the patchset with KAS:
- export KAS_IMAGE_VERSION=next

If 'gitlab' is used, the similar changes are required:

- `next` tag for kas image should be set

- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'

downstreams in combination with kas image mentioned above.

Current limitations:
- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

Changes since v5:
- warning about shell exports in dpkg_runbuild_prepend. This should
avoided be reworked to templates.

Changes since v4:
- consider shell exports done in dpkg_runbuild_prepend and pass them
to sbuild environment;
- fixed ccache work.

Changes since v3:
- dpkg_do_mounts() and dpkg_undo_mounts() are not removed for downstreams
compatibility;
- dpkg_build_export is used for adjusting sbuild environment;
- DEB_BUILD_PROFILES support for sbuild is fixed in case of cross-build.

Changes since v2:
- patches reworked/squashed for easier reading and understanding;
- fixed building foreigh architectures with kas-docker;
- implemented support of ccache;
- fixed devshell and devshell_nodeps,

Changes since v1:
- parallel builds with different BUILD_DIR are supported;
- parallel multiconfig targets in one build are supported;
- per-task schroot configuration in /etc/schroot/ is now used;
- patchset now passes Jenkins CI (so patches changes RFC => PATCH).

Anton Mikanovich (1):
dpkg: Build packages with sbuild

Uladzimir Bely (11):
dpkg-gbp: Use separate command to export tarball
dpkg-gbp: Use host tools for dsc preparation
sbuild: Add recipes for host and target rootfs to run sbuild
sbuild: Introduce a class for another build method
sbuild: support of DEB_BUILD_PROFILES

sbuild: support of shell exports from dpkg_runbuild_prepend

dpkg: Remove builddeps install task.

sbuild: add ccache support

dpkg-base: Switch devshell to use schroot
dpkg-base: Switch apt_fetch and apt_unpack to use schroot

doc: Add sbuild-related documentation

doc/user_manual.md | 22 ++-

meta/classes/dpkg-base.bbclass | 79 ++++----
meta/classes/dpkg-gbp.bbclass | 26 +--
meta/classes/dpkg.bbclass | 88 ++++++---
meta/classes/sbuild.bbclass | 184 ++++++++++++++++++

meta/conf/bitbake.conf | 2 +
.../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 +
.../sbuild-chroot/sbuild-chroot.inc | 39 ++++

9 files changed, 382 insertions(+), 81 deletions(-)

create mode 100644 meta/classes/sbuild.bbclass
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:22 AM2/10/22

to isar-...@googlegroups.com

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 4d4fa763..df73ef89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

@@ -80,7 +80,9 @@ apt install \
qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```

If your host is >= buster, also install the following package.

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass

@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"

PATCHTOOL ?= "git"

-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

-do_install_builddeps_append() {
- dpkg_do_mounts

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
-}
-
dpkg_runbuild_prepend() {

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:23 AM2/10/22

to isar-...@googlegroups.com

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
3 files changed, 59 insertions(+)

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644

index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb

new file mode 100644

index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

new file mode 100644

index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#

+# This software is a part of ISAR.

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:23 AM2/10/22

to isar-...@googlegroups.com

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
}

dpkg_runbuild_prepend() {
- export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"

+ # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used

+ # for compatibility with gbp version froms debian-stretch. In newer distros
+ # it's possible to use a subcommand `export-orig --pristine-tar`
}
--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:24 AM2/10/22

to isar-...@googlegroups.com

Many of recipes often use shell exports done in dpkg_run_prepend.

While sbuild is performed in isolated environment, we need a way to pass

these exported variables to it. This is done by storing environment just

before dpkg_runbuild (after prepare_build) and finding just before

the actual build what has been changed or added.

This patch allows using shell exports, but warning will be produced.
It's recommended to modify the code (in downstreams) and replace shell
exports by other methods, like using templates.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5511de64..82a722df 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,12 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

addtask devshell after do_install_builddeps

+DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"
+
+do_prepare_build_append() {
+ env > ${DPKG_PREBUILD_ENV_FILE}
+}
+
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"

@@ -40,6 +46,27 @@ dpkg_runbuild() {
export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"

export PARALLEL_MAKE="${PARALLEL_MAKE}"

+ env | while read -r line; do
+ # Filter the same lines
+ grep -q "^${line}" ${DPKG_PREBUILD_ENV_FILE} && continue
+ # Filter some standard variables
+ echo ${line} | grep -q "^HOME=" && continue
+ echo ${line} | grep -q "^PWD=" && continue
+
+ var=$(echo "${line}" | cut -d '=' -f1)
+ value=$(echo "${line}" | cut -d '=' -f2-)
+ sbuild_export $var "$value"
+

+ # Don't warn some variables
+ [ "${var}" = "PARALLEL_MAKE" ] && continue
+ [ "${var}" = "CCACHE_DIR" ] && continue
+ [ "${var}" = "PATH_PREPEND" ] && continue
+ [ "${var}" = "DEB_BUILD_OPTIONS" ] && continue
+ [ "${var}" = "DEB_BUILD_PROFILES" ] && continue
+
+ bbwarn "Export of '${line}' detected, please migrate to templates"

+ done
+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

profiles="${profiles} cross nocheck"

--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:24 AM2/10/22

to isar-...@googlegroups.com

In Isar packages may use their own profiles. For example, linux-custom.inc
introduces 'nolibcdev' profile that should be passed to dpkg-buildpackage
in both cross and native builds.

By default, sbuild for cross-building uses passes "cross,nocheck" options
to dpkg-buildpackage and ignores DEB_BUILD_PROFILES environment variable.

This change makes sbuild use custom profiles even in cross-build mode.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 848ffad1..5511de64 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -40,10 +40,18 @@ dpkg_runbuild() {

export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

+ profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ profiles="${profiles} cross nocheck"
+ fi
+ if [ ! -z "$profiles" ]; then
+ profiles=$(echo --profiles="$profiles" | sed -e 's/ \+/,/g')
+ fi
+
export SBUILD_CONFIG="${SBUILD_CONFIG}"

sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

- --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \

+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:24 AM2/10/22

to isar-...@googlegroups.com

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
`cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 160 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 2 +
2 files changed, 162 insertions(+)
create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..e4341a6f
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,160 @@

+# This software is a part of ISAR.

+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+ import pwd
+ d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+ d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+ d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ dep = "sbuild-chroot-target:do_build"
+ else:
+ d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ dep = "sbuild-chroot-host:do_build"
+ d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {

+ sudo -s <<'EOSUDO'
+ set -e
+

+ cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+ cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+ mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+ cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+ # Prepare mount points
+ cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+

+ if [ -d ${DL_DIR} ]; then

+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {

+ sudo -s <<'EOSUDO'
+ set -e

+ if [ -d "${SBUILD_CONF_DIR}" ]; then
+ rm -rf "${SBUILD_CONF_DIR}"
+ fi
+ rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${SCHROOT_CONF_FILE}"
+ rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+sbuild_export() {
+ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+ VAR=${1}; shift
+ VAR_LINE="'${VAR}' => '${@}',"
+ if [ -s "${SBUILD_CONFIG}" ]; then
+ sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+ else
+ echo "\$build_environment = {" > ${SBUILD_CONFIG}
+ echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ echo "};" >> ${SBUILD_CONFIG}

+ fi
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"

+}
+
+schroot_install() {
+ schroot_create_configs
+ APTS="$1"
+
+ distro="${DISTRO}"

+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends --download-only ${APTS}
+ deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends ${APTS}
+ schroot_delete_configs
+}
+
+insert_mounts() {

+ sudo -s <<'EOSUDO'
+ set -e

+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+remove_mounts() {

+ sudo -s <<'EOSUDO'
+ set -e

+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+schroot_run() {
+ schroot_create_configs
+ insert_mounts
+ schroot $@
+ remove_mounts
+ schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "vmlinuz"
--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:24 AM2/10/22

to isar-...@googlegroups.com

From: Anton Mikanovich <ami...@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 31 ++++++++++++++++---------------
meta/classes/dpkg-gbp.bbclass | 2 ++
meta/classes/dpkg.bbclass | 10 ++++++++--
meta/classes/sbuild.bbclass | 4 ++--
4 files changed, 28 insertions(+), 19 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 550bedfc..31d41754 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT

+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -16,6 +17,8 @@ DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_C
DEB_BUILD_PROFILES ?= ""
DEB_BUILD_OPTIONS ?= ""

+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
python do_adjust_git() {
import subprocess

@@ -173,6 +176,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"
+do_prepare_build[depends] = "${SCHROOT_DEP}"

BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"

@@ -198,6 +202,15 @@ dpkg_undo_mounts() {
sudo rmdir ${BUILDROOT}
}

+do_prepare_build_append() {
+ # Make a local copy of isar-apt repo that is not affected by other parallel builds
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
# Placeholder for actual dpkg_runbuild() implementation
dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"
@@ -220,14 +233,11 @@ def isar_export_build_settings(d):
os.environ['DEB_BUILD_PROFILES'] = isar_deb_build_profiles(d)

python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)

- bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func('schroot_create_configs', d)

try:
bb.build.exec_func("dpkg_runbuild", d)
finally:

- bb.build.exec_func("dpkg_undo_mounts", d)

- bb.utils.unlockfile(lock)

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask dpkg_build
@@ -259,7 +269,7 @@ python do_dpkg_build_setscene() {
addtask dpkg_build_setscene
do_dpkg_build_setscene[dirs] += "${S}/.."

-KEEP_INSTALLED_ON_CLEAN ?= "0"
+do_dpkg_build[depends] = "${SCHROOT_DEP}"

CLEANFUNCS += "deb_clean"

@@ -269,15 +279,6 @@ deb_clean() {
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"

GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {
sh -c "

cd ${WORKDIR}/${PPS}

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 320102ba..848ffad1 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -39,6 +39,12 @@ dpkg_runbuild() {
export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"

export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH}
+

+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+

+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --debbuildopts="--source-option=-I" \
+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index e4341a6f..6fff664f 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -31,6 +31,8 @@ SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"

SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"

SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"

+SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+
schroot_create_configs() {
sudo -s <<'EOSUDO'
set -e
@@ -96,7 +98,6 @@ EOSUDO
}

sbuild_export() {
- SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
VAR=${1}; shift

VAR_LINE="'${VAR}' => '${@}',"

if [ -s "${SBUILD_CONFIG}" ]; then

@@ -106,7 +107,6 @@ sbuild_export() {

echo "${VAR_LINE}" >> ${SBUILD_CONFIG}

echo "};" >> ${SBUILD_CONFIG}
fi
- export SBUILD_CONFIG="${SBUILD_CONFIG}"
}

schroot_install() {
--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:25 AM2/10/22

to isar-...@googlegroups.com

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 23 ++++++++++++++-----
meta/classes/sbuild.bbclass | 3 +++
.../sbuild-chroot/sbuild-chroot.inc | 2 ++

3 files changed, 22 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 31d41754..238a0bbd 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -296,19 +296,30 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

do_deploy_deb[dirs] = "${S}"

python do_devshell() {

- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)

isar_export_proxies(d)
isar_export_ccache(d)
isar_export_build_settings(d)

- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
+ isar_apt = d.getVar('ISAR_APT_REPO')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- # the PATH variable is not forwarded by sudo -E.
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; export PATH=$PATH_PREPEND:$PATH; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)

- bb.build.exec_func('dpkg_undo_mounts', d)

+ install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+ \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+ debian/control"
+
+ termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+ cd {1}; \
+ echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+ apt-get -y -q update; \
+ {3}; \
+ $SHELL -i \
+ '"
+ oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 9896b772..eb358c71 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -78,6 +78,9 @@ EOF

fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"

grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}

+
if [ -d ${DL_DIR} ]; then

fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:25 AM2/10/22

to isar-...@googlegroups.com

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 47 +++++++++++++++++----------------------
1 file changed, 20 insertions(+), 27 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 82a722df..4cf405ad 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,33 +5,6 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-# Install build dependencies for package
-do_install_builddeps() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- export DEB_BUILD_OPTIONS="${@ isar_deb_build_options(d)}"
- export DEB_BUILD_PROFILES="${@ isar_deb_build_profiles(d)}"

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
-
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-
DPKG_PREBUILD_ENV_FILE="${WORKDIR}/dpkg_prebuild.env"

do_prepare_build_append() {
@@ -67,6 +40,21 @@ dpkg_runbuild() {

bbwarn "Export of '${line}' detected, please migrate to templates"

done

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives/"
+ ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+ ( flock 9
+ grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+ ) 9>"${TMPDIR}/sbuildrc.lock"

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

profiles="${profiles} cross nocheck"

@@ -80,6 +68,11 @@ dpkg_runbuild() {

sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--debbuildopts="--source-option=-I" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+
+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
}
--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:25 AM2/10/22

to isar-...@googlegroups.com

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 21 +++++++++++++++++++
.../sbuild-chroot/sbuild-chroot.inc | 1 +
3 files changed, 26 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 4cf405ad..3cd40ba8 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -55,6 +55,10 @@ dpkg_runbuild() {

echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc

) 9>"${TMPDIR}/sbuildrc.lock"

+ if [ ${USE_CCACHE} -eq 1 ]; then
+ schroot_configure_ccache
+ fi

+
profiles=$(grep "DEB_BUILD_PROFILES" ${SBUILD_CONFIG} | tail -n1 | cut -d "'" -f 4)
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
profiles="${profiles} cross nocheck"

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 6fff664f..9896b772 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -158,3 +158,24 @@ schroot_run() {
remove_mounts
schroot_delete_configs
}
+
+schroot_configure_ccache() {

+ sudo -s <<'EOSUDO'
+ set -e
+

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+ cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export PATH="\$PATH_PREPEND:\$PATH"
+exec "\$@"
+END
+ chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+ echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..74811e22 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
build-essential \
debhelper \
+ ccache \
"

SBUILD_CHROOT_PREINSTALL ?= " \
--
2.20.1

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:26 AM2/10/22

to isar-...@googlegroups.com

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 18 ++++++++++++++++++

1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index df73ef89..966e7125 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

@@ -85,6 +85,24 @@ apt install \
pristine-tar
```

+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+

If your host is >= buster, also install the following package.

Uladzimir Bely

unread,

Feb 10, 2022, 1:46:26 AM2/10/22

to isar-...@googlegroups.com

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 25 +++++++++----------------
1 file changed, 9 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 238a0bbd..7220f64f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -101,19 +101,12 @@ python() {
}

do_apt_fetch() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_fetch after do_unpack before do_apt_unpack
@@ -122,13 +115,14 @@ do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
# Add dependency from the correct buildchroot: host or target
do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"

+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] += "${SCHROOT_DEP}"
+
do_apt_unpack() {
rm -rf ${S}

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-

+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -137,8 +131,7 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_unpack after do_apt_fetch before do_patch
--
2.20.1

Uladzimir Bely

unread,

Feb 13, 2022, 2:45:02 AM2/13/22