[PATCH v2 01/24] dpkg: Install raw package files to source root

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To get rid of using absolute paths in debian/rules move D to source
root. This is default location for dh_install, so now there is no need
in mention it in debian/rules.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
meta/classes/dpkg-raw.bbclass | 8 +-------
1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index dd7b761f..73356fa0 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -5,7 +5,7 @@

inherit dpkg

-D = "${WORKDIR}/image"
+D = "${S}"

# Populate folder that will be picked up as package
do_install() {
@@ -22,10 +22,4 @@ do_prepare_build() {
sed 's:^./::' > ${S}/debian/${PN}.install

deb_debianize
-
- cat <<EOF >> ${S}/debian/rules
-
-override_dh_install:
- dh_install --sourcedir=${PP}/image
-EOF
}
--
2.20.1

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and
configured. Current user should be added to sbuild group.

Changes since v1:
- parallel builds with different BUILD_DIR are supported
- parallel multiconfig targets in one build are supported
- per-task schroot configuration in /etc/schroot/ is now used
- patchset now passes Jenkins CI (so patches changes RFC => PATCH)

Current limitations:
- there is an unsolved problem with building foreigh architectures
using kas-docker.

- qemuamd64 images are buildable in kas-docker, but some changes
in kas are reqired (like installing additional packages and adding
'builder' user to 'sbuild group). I used the following changes to
build ghcr.io/siemens/kas/kas-isar:2.6.2-sbuild image based on
current ghcr.io/siemens/kas/kas-isar:2.6.2:
https://github.com/WiseLord/kas/commit/5b7b02aa33

- parallel building of several packages requires more free space
than current buildchroot-based architecture uses. This happens due
the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

- recently merged to 'next' ccache support is not yet implemented
in case of sbuild used.

For experiments, the code can be taken from Isar 'ubely/sbuild' branch.
This is an dev branch, so sometimes it can be force-pushed when some
fixes are applied of rebased to 'next' branch.

Anton Mikanovich (12):
dpkg: Install raw package files to source root
isar-bootstrap: Export bootstrap to schroot config
linux-module: Do not use shell environment
u-boot: Do not use shell environment
trusted-firmware: Do not use shell environment
optee-os: Do not use shell environment
kselftest: Do not use shell environment
dpkg: Build packages with sbuild
sbuild: Introduce environment variables export API
dpkg-gbp: Migrate to schroot
linux-mainline: Move cfg fragment test to debian/rules
linux-custom: Prepare kernel config inside sbuild

Uladzimir Bely (12):
dpkg-gbp: Use separate command to export tarball
sbuild: Add recipes for host and target rootfs to run sbuild
sbuild: Mount base-apt in schroot
sbuild: Add sbuildshell task
dpkg-gbp: Preinstall gbp utils in schroot
dpkg: Remove builddeps install task.
dpkg-base: Switch devshell to use schroot
dpkg-base: Switch apt_fetch and apt_unpack to use schroot
dpkg-base: Cleanup from buildchroot parts.
dpkg-gbp: Use host tools for dsc preparation
doc: Add sbuild-related documentation
sbuild: Replace isar-apt mounting with copying

doc/user_manual.md | 22 ++-
.../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
.../linux/linux-mainline_5.4.70.bb | 16 +-
meta/classes/dpkg-base.bbclass | 80 ++------
meta/classes/dpkg-gbp.bbclass | 25 +--
meta/classes/dpkg-raw.bbclass | 8 +-
meta/classes/dpkg.bbclass | 56 +++---
meta/classes/sbuild.bbclass | 177 ++++++++++++++++++
meta/conf/bitbake.conf | 2 +
.../files/debian/{rules => rules.tmpl} | 4 +-
meta/recipes-bsp/optee-os/optee-os-custom.inc | 10 +-
.../files/debian/{rules => rules.tmpl} | 4 +-
.../trusted-firmware-a-custom.inc | 10 +-
.../u-boot/files/debian/{rules => rules.tmpl} | 2 +-
meta/recipes-bsp/u-boot/u-boot-custom.inc | 10 +-
.../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 +
.../sbuild-chroot/sbuild-chroot.inc | 36 ++++
.../kselftest/files/{rules => rules.tmpl} | 0
meta/recipes-kernel/kselftest/kselftest.inc | 21 +--
.../files/debian/{rules => rules.tmpl} | 14 +-
meta/recipes-kernel/linux-module/module.inc | 23 +--
meta/recipes-kernel/linux/linux-custom.inc | 7 +-
23 files changed, 382 insertions(+), 170 deletions(-)
create mode 100644 meta/classes/sbuild.bbclass
rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)
rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)
rename meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (50%)

--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To use isar-bootstrap with sbuild and schroot tools we need to export
the path to schroot config.

We also need to put isar-apt and downloads into fstab for sbuild
profile to have it mounted inside schroot.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
meta/classes/sbuild.bbclass | 87 +++++++++++++++++++
.../isar-bootstrap/isar-bootstrap.inc | 10 ++-
2 files changed, 96 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..d6e5b926
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,87 @@
+# This software is a part of ISAR.
+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+python __anonymous() {
+ import pwd
+ d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+ d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+ d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ else:
+ d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${DISTRO}-${SBUILD_HOST_ARCH}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+
+SCHROOT_DIR ?= "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${SBUILD_HOST_ARCH}"
+
+schroot_create_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+ mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+ cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+ # Prepare mount points
+ cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+ fstab_isarapt="${DEPLOY_DIR}/isar-apt /isar-apt none rw,bind 0 0"
+ grep -qxF "${fstab_isarapt}" ${sbuild_fstab} || echo "${fstab_isarapt}" >> ${sbuild_fstab}
+
+ if [ -d ${DL_DIR} ]; then
+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+ if [ -d "${SBUILD_CONF_DIR}" ]; then
+ rm -rf "${SBUILD_CONF_DIR}"
+ fi
+ rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index e9f92916..39e9c007 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -25,13 +25,14 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,fakeroot,build-essential,debhelper"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"

inherit deb-dl-dir
+inherit sbuild

python () {
distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
@@ -398,6 +399,13 @@ EOSUDO

addtask bootstrap before do_build after do_generate_keyrings

+do_schroot_config() {
+ schroot_create_configs
+}
+
+addtask schroot_config before do_bootstrap after do_generate_keyrings
+# TODO: remove configs from /etc/schroot
+
CLEANFUNCS = "clean_deploy"
clean_deploy() {
rm -f "${DEPLOY_ISAR_BOOTSTRAP}"
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we
should remove KDIR and PN passing through export call.
KDIR can be prepared during package build. This also will allow not to
rely on previous builddeps install task.
To pass PN variable we can just migrate to template-based debian/rules
file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../files/debian/{rules => rules.tmpl} | 14 ++++++++++-
meta/recipes-kernel/linux-module/module.inc | 23 +++++++------------
2 files changed, 21 insertions(+), 16 deletions(-)

rename meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (50%)

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
similarity index 50%
rename from meta/recipes-kernel/linux-module/files/debian/rules
rename to meta/recipes-kernel/linux-module/files/debian/rules.tmpl
index 59720b37..d8f1d48a 100755
--- a/meta/recipes-kernel/linux-module/files/debian/rules
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -24,14 +24,26 @@ ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
export ARCH=x86
endif

+# Custom kernels contain the build folder directly.
+KDIR := $(shell dpkg -L linux-headers-${KERNEL_NAME} | grep "/lib/modules/.*/build")
+ifeq ($(KDIR),)
+# Debian kernels install that folder indirectly via a dependency.
+KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
+KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
+endif
+
+# With sbuild `dh clean` is called before chroot's apt database updated. So,
+# KDIR is empty at that moment and the override causes error. Will skip it.
+ifneq ($(KDIR),)
override_dh_auto_clean:
$(MAKE) -C $(KDIR) M=$(PWD) clean
+endif

override_dh_auto_build:
$(MAKE) -C $(KDIR) M=$(PWD) modules

override_dh_auto_install:
- $(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/$(PN) modules_install
+ $(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/${PN} modules_install

%:
CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 0515621a..5b086233 100644
--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -22,8 +22,9 @@ AUTOLOAD ?= ""
inherit dpkg

TEMPLATE_FILES = "debian/control.tmpl \
- debian/changelog.tmpl"
-TEMPLATE_VARS += "KERNEL_NAME"
+ debian/changelog.tmpl \
+ debian/rules.tmpl"
+TEMPLATE_VARS += "KERNEL_NAME PN"

do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/
@@ -31,18 +32,10 @@ do_prepare_build() {
for module in "${AUTOLOAD}"; do
echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
done
-}

-dpkg_runbuild_prepend() {
- # Custom kernels contain the build folder directly.
- export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} linux-headers-${KERNEL_NAME} | \
- grep "/lib/modules/.*/build")
- if [ -z "$KDIR" ]; then
- # Debian kernels install that folder indirectly via a dependency.
- KERNEL_DEP=$(dpkg-query -W -f '${Depends}' --admindir=${BUILDCHROOT_DIR}/var/lib/dpkg \
- linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
- export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} ${KERNEL_DEP} | \
- grep "/lib/modules/.*/build")
- fi
- export PN=${PN}
+ # remove templates from the source tree
+ find ${S}/debian -name *.tmpl | xargs rm -f
+
+ # restore execute permissions
+ chmod a+x ${S}/debian/rules
}
--
2.20.1

[Uladzimir Bely]

We don't actually build the package with gbp, but only prepare it for
the building with external dpkg-buildpackage command. In case there is
no need in real build we can perform only export which will produce
upstream tarball to pass for later building.
This allows to use any build tool for later processing.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index d956e8c3..ddf23ca5 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -30,5 +30,9 @@ do_install_builddeps_append() {
}

dpkg_runbuild_prepend() {
- export GBP_PREFIX="gbp buildpackage --git-ignore-new ${GBP_EXTRA_OPTIONS} --git-builder="
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ # NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
+ # for compatibility with gbp version froms debian-stretch. In newer distros
+ # it's possible to use a subcommand `export-orig --pristine-tar`
}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/dpkg.bbclass | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 86b47c28..e021e546 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -2,6 +2,7 @@
# Copyright (C) 2015-2018 ilbers GmbH

inherit dpkg-base
+inherit sbuild

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

@@ -28,10 +29,16 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

addtask devshell after do_install_builddeps

+ISAR_APT_REPO ?= "deb [trusted=yes] file:///isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
+
+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove TF_A_PLATFORM and TF_A_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../files/debian/{rules => rules.tmpl} | 4 ++--
.../trusted-firmware-a/trusted-firmware-a-custom.inc | 10 ++++------
2 files changed, 6 insertions(+), 8 deletions(-)

rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)

diff --git a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
similarity index 77%
rename from meta/recipes-bsp/trusted-firmware-a/files/debian/rules
rename to meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
index 70e1dd66..6dbf44db 100755
--- a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules
+++ b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
endif

override_dh_auto_build:
- CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=$(TF_A_PLATFORM) \
- $(TF_A_EXTRA_BUILDARGS)
+ CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=${TF_A_PLATFORM} \
+ ${TF_A_EXTRA_BUILDARGS}

%:
dh $@
diff --git a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
index 1142bfce..3fe4c415 100644
--- a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
+++ b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= ""

PROVIDES += "trusted-firmware-a-${TF_A_NAME}"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS TF_A_PLATFORM TF_A_EXTRA_BUILDARGS"

do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/

@@ -35,9 +35,7 @@ do_prepare_build() {
echo "build/${TF_A_PLATFORM}/$binary /usr/lib/trusted-firmware-a/${TF_A_NAME}/" >> \
${S}/debian/trusted-firmware-a-${TF_A_NAME}.install
done
-}

-dpkg_runbuild_prepend() {
- export TF_A_PLATFORM="${TF_A_PLATFORM}"
- export TF_A_EXTRA_BUILDARGS="${TF_A_EXTRA_BUILDARGS}"

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove KSELFTEST_ARGS passing through export call. The same
logic can be done by internal recipe parser and then pass variable to

template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../kselftest/files/{rules => rules.tmpl} | 0

meta/recipes-kernel/kselftest/kselftest.inc | 21 +++++++------------
2 files changed, 8 insertions(+), 13 deletions(-)

rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

diff --git a/meta/recipes-kernel/kselftest/files/rules b/meta/recipes-kernel/kselftest/files/rules.tmpl
similarity index 100%
rename from meta/recipes-kernel/kselftest/files/rules
rename to meta/recipes-kernel/kselftest/files/rules.tmpl
diff --git a/meta/recipes-kernel/kselftest/kselftest.inc b/meta/recipes-kernel/kselftest/kselftest.inc
index 2a76028b..6187d8e4 100644
--- a/meta/recipes-kernel/kselftest/kselftest.inc
+++ b/meta/recipes-kernel/kselftest/kselftest.inc
@@ -33,26 +33,21 @@ DEBIAN_BUILD_DEPENDS ?= " \
llvm:native, \
"

-SRC_URI += "file://rules"
+SRC_URI += "file://rules.tmpl"
S = "${WORKDIR}/linux-${PV}"

+TEMPLATE_FILES = "rules.tmpl"
+TEMPLATE_VARS += "KSELFTEST_ARGS"
+
KSELFTEST_TARGETS ?= ""
KSELFTEST_SKIP_TARGETS ?= ""
KSELFTEST_FORCE_TARGETS ?= "0"

+KSELFTEST_ARGS = "${@ "TARGETS=\"${KSELFTEST_TARGETS}\"" if d.getVar('KSELFTEST_TARGETS', True) else ''}"
+KSELFTEST_ARGS_append = "${@ " FORCE_TARGETS=1" if d.getVar('KSELFTEST_FORCE_TARGETS', True) == '1' else ''}"
+KSELFTEST_ARGS_append .= "${@ " SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\"" if d.getVar('KSELFTEST_SKIP_TARGETS', True) else ''}"
+
do_prepare_build[cleandirs] += "${S}/debian"
do_prepare_build() {
deb_debianize
}
-
-dpkg_runbuild_prepend() {
- if [ -n "${KSELFTEST_TARGETS}" ];then
- export KSELFTEST_ARGS="TARGETS=\"${KSELFTEST_TARGETS}\""
- fi
- if [ ${KSELFTEST_FORCE_TARGETS} -eq 1 ];then
- export KSELFTEST_ARGS="${KSELFTEST_ARGS} FORCE_TARGETS=1"
- fi
- if [ -n "${KSELFTEST_SKIP_TARGETS}" ];then
- export KSELFTEST_ARGS="${KSELFTEST_ARGS} SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\""
- fi
-}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove U_BOOT_CONFIG and U_BOOT_BIN passing through export call.
So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../u-boot/files/debian/{rules => rules.tmpl} | 2 +-

meta/recipes-bsp/u-boot/u-boot-custom.inc | 10 ++++------
2 files changed, 5 insertions(+), 7 deletions(-)

rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
similarity index 94%
rename from meta/recipes-bsp/u-boot/files/debian/rules
rename to meta/recipes-bsp/u-boot/files/debian/rules.tmpl
index 3d667620..806b01fe 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
@@ -13,7 +13,7 @@ SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
endif

override_dh_auto_build:
- $(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
+ $(MAKE) $(PARALLEL_MAKE) ${U_BOOT_CONFIG}
$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
$(MAKE) -n u-boot-initial-env >/dev/null 2>&1; if [ $$? -ne 2 ]; then \
$(MAKE) $(PARALLEL_MAKE) u-boot-initial-env; \
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 9984d8cc..819206af 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -26,8 +26,8 @@ python() {

DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS U_BOOT_CONFIG U_BOOT_BIN"

U_BOOT_TOOLS_PACKAGE ?= "0"
U_BOOT_CONFIG_PACKAGE ?= "0"
@@ -70,9 +70,7 @@ u-boot-initial-env /etc
fw_env.config /etc
EOF
fi
-}

-dpkg_runbuild_prepend() {
- export U_BOOT_CONFIG="${U_BOOT_CONFIG}"
- export U_BOOT_BIN="${U_BOOT_BIN}"

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove OPTEE_PLATFORM and OPTEE_EXTRA_BUILDARGS passing through

export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../optee-os/files/debian/{rules => rules.tmpl} | 4 ++--
meta/recipes-bsp/optee-os/optee-os-custom.inc | 10 ++++------
2 files changed, 6 insertions(+), 8 deletions(-)

rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)

diff --git a/meta/recipes-bsp/optee-os/files/debian/rules b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
similarity index 75%
rename from meta/recipes-bsp/optee-os/files/debian/rules
rename to meta/recipes-bsp/optee-os/files/debian/rules.tmpl
index d2e9900f..9ab80dfe 100755
--- a/meta/recipes-bsp/optee-os/files/debian/rules
+++ b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl

@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
endif

override_dh_auto_build:

- CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=$(OPTEE_PLATFORM) \
- $(OPTEE_EXTRA_BUILDARGS)
+ CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=${OPTEE_PLATFORM} \
+ ${OPTEE_EXTRA_BUILDARGS}

%:
dh $@
diff --git a/meta/recipes-bsp/optee-os/optee-os-custom.inc b/meta/recipes-bsp/optee-os/optee-os-custom.inc
index 1bd51969..e9eed54d 100644
--- a/meta/recipes-bsp/optee-os/optee-os-custom.inc
+++ b/meta/recipes-bsp/optee-os/optee-os-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= "python3-pycryptodome:native, python3-pyelftools"

PROVIDES += "optee-os-${OPTEE_NAME}"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS OPTEE_PLATFORM OPTEE_EXTRA_BUILDARGS"

# split strip platform flavor, if any, from the specified platform string
OPTEE_PLATFORM_BASE = "${@d.getVar('OPTEE_PLATFORM').split('-')[0]}"
@@ -38,9 +38,7 @@ do_prepare_build() {
echo "out/arm-plat-${OPTEE_PLATFORM_BASE}/core/$binary /usr/lib/optee-os/${OPTEE_NAME}/" >> \
${S}/debian/optee-os-${OPTEE_NAME}.install
done
-}

-dpkg_runbuild_prepend() {
- export OPTEE_PLATFORM="${OPTEE_PLATFORM}"
- export OPTEE_EXTRA_BUILDARGS="${OPTEE_EXTRA_BUILDARGS}"

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Use schroot instead of chroot for git-buildpackage.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 21 ++++------------
meta/classes/sbuild.bbclass | 45 +++++++++++++++++++++++++++++++++++
2 files changed, 50 insertions(+), 16 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..7ebbf2e0 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -13,25 +13,14 @@ GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

do_install_builddeps_append() {
- dpkg_do_mounts
- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
+ schroot_install "${GBP_DEPENDS}"
}

+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

- sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ schroot_run -d ${PP}/${PPS} -c ${SBUILD_CHROOT} -- \
+ gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}

# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used

# for compatibility with gbp version froms debian-stretch. In newer distros

# it's possible to use a subcommand `export-orig --pristine-tar`

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 5f9f7691..b7a349e6 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -3,6 +3,8 @@

SCHROOT_CONF ?= "/etc/schroot"

+SCHROOT_MOUNTS ?= ""
+
python __anonymous() {
import pwd
d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
@@ -98,3 +100,46 @@ sbuild_export() {
fi
export SBUILD_CONFIG="${SBUILD_CONFIG}"
}
+
+schroot_install() {
+ APTS="$1"
+ #TODO deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends --download-only ${APTS}
+ #TODO deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends ${APTS}
+}
+
+insert_mounts() {
+ sudo -s <<'EOSUDO'
+ ( flock 9
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+ ) 9>'${SCHROOT_LOCKFILE}'
+EOSUDO
+}
+
+remove_mounts() {
+ sudo -s <<'EOSUDO'
+ ( flock 9
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+ done
+ ) 9>'${SCHROOT_LOCKFILE}'
+EOSUDO
+}
+
+schroot_run() {
+ insert_mounts
+ schroot $@
+ remove_mounts
+}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

In case there is no ability to pass variables through template
mechanism we need to keep fallback API to migrate from bare export.
Using it with optee-os-stm32mp15x recipe as example.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
meta/classes/sbuild.bbclass | 13 +++++++++++++
2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
index 0fa2005d..19aa763a 100644
--- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
+++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
@@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
# Set version manually to PV, the tarball does not contain any hint.
# Alternative: pull from git and add git as build dependency.
dpkg_runbuild_prepend() {
- export TEE_IMPL_VERSION=${PV}
+ sbuild_export TEE_IMPL_VERSION=${PV}
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index d6e5b926..5f9f7691 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -85,3 +85,16 @@ schroot_delete_configs() {
rm -f "${SCHROOT_CONF_FILE}"
EOSUDO
}
+
+sbuild_export() {
+ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+ VAR_LINE="'${1%%=*}' => '${1#*=}',"
+ if [ -s "${SBUILD_CONFIG}" ]; then
+ sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+ else
+ echo "\$build_environment = {" > ${SBUILD_CONFIG}
+ echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ echo "};" >> ${SBUILD_CONFIG}
+ fi
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Move prepare code inside sbuild hook to perform all the operations
inside schroot.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 10 +++++-----
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 8 --------
meta/recipes-kernel/linux/linux-custom.inc | 7 +++++--
4 files changed, 14 insertions(+), 15 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index e021e546..6da61185 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -36,9 +36,13 @@ dpkg_runbuild() {

E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

+ schroot_create_configs

+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \

--starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \

--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+
+ schroot_delete_configs
}
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index b7a349e6..a03883f0 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -19,7 +19,7 @@ python __anonymous() {

d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
}

-SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${DISTRO}-${SBUILD_HOST_ARCH}"
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"

SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"

@@ -102,6 +102,7 @@ sbuild_export() {
}

schroot_install() {
+ schroot_create_configs
APTS="$1"

#TODO deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \

@@ -111,35 +112,34 @@ schroot_install() {

schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \

apt install -y -o Debug::pkgProblemResolver=yes \

--no-install-recommends ${APTS}
+ schroot_delete_configs
}

insert_mounts() {
sudo -s <<'EOSUDO'
- ( flock 9
set -e

for mp in ${SCHROOT_MOUNTS}; do

FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"

grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \

echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab

done
- ) 9>'${SCHROOT_LOCKFILE}'
EOSUDO
}

remove_mounts() {
sudo -s <<'EOSUDO'
- ( flock 9
set -e

for mp in ${SCHROOT_MOUNTS}; do

FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"

sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab

done
- ) 9>'${SCHROOT_LOCKFILE}'
EOSUDO
}

schroot_run() {
+ schroot_create_configs
insert_mounts
schroot $@
remove_mounts
+ schroot_delete_configs
}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 39e9c007..1b227028 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -32,7 +32,6 @@ DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else

BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"

inherit deb-dl-dir

-inherit sbuild

python () {
distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()

@@ -399,13 +398,6 @@ EOSUDO

addtask bootstrap before do_build after do_generate_keyrings

-do_schroot_config() {
- schroot_create_configs
-}
-
-addtask schroot_config before do_bootstrap after do_generate_keyrings
-# TODO: remove configs from /etc/schroot
-

CLEANFUNCS = "clean_deploy"
clean_deploy() {
rm -f "${DEPLOY_ISAR_BOOTSTRAP}"

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ed89aa09..21115840 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -125,6 +125,9 @@ do_prepare_build_prepend() {

# remove templates from the source tree

find ${S}/debian -name *.tmpl | xargs rm -f

+ # restore execute permissions
+ chmod a+x ${S}/debian/rules

+
# rename install/remove hooks to match user-specified name for our linux-image package
mv ${S}/debian/linux-image.postinst ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postinst
mv ${S}/debian/linux-image.postrm ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postrm
@@ -164,9 +167,9 @@ dpkg_configure_kernel() {
(cd ${WORKDIR} && cp ${src_frags} ${S}/debian/fragments/)
fi

- sudo -E chroot --userspec=$(id -u):$(id -g) ${BUILDCHROOT_DIR} sh -c " \
+ SBUILD_PREBUILD=" \
export ARCH=${KERNEL_ARCH} && \
- cd ${PP}/${PPS} && \
+ cd %p && \
make O=${KERNEL_BUILD_DIR} ${config_target} && \
./scripts/kconfig/merge_config.sh \
-O ${KERNEL_BUILD_DIR}/ \
--
2.20.1

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 3 ++
meta/classes/dpkg.bbclass | 1 -
meta/classes/sbuild.bbclass | 7 ++--
meta/conf/bitbake.conf | 2 ++
.../isar-bootstrap/isar-bootstrap.inc | 2 +-
.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
8 files changed, 70 insertions(+), 4 deletions(-)

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 8bbc4165..acf967ac 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT

+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -214,6 +215,8 @@ python do_dpkg_build() {

addtask dpkg_build before do_build

+do_dpkg_build[depends] = "${SCHROOT_DEP}"
+
KEEP_INSTALLED_ON_CLEAN ?= "0"

CLEANFUNCS += "deb_clean"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 6da61185..d49648be 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -2,7 +2,6 @@

# Copyright (C) 2015-2018 ilbers GmbH

inherit dpkg-base

-inherit sbuild

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index a03883f0..80f9ef25 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -15,8 +15,13 @@ python __anonymous() {

if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \

(d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):

d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ dep = "sbuild-chroot-target:do_build"
else:

d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))

+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ dep = "sbuild-chroot-host:do_build"
+ d.setVar('SCHROOT_DEP', dep)

}

SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"

@@ -25,8 +30,6 @@ SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"

SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"

SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"

-SCHROOT_DIR ?= "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${SBUILD_HOST_ARCH}"
-
schroot_create_configs() {
sudo -s <<'EOSUDO'
set -e
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index dce467b2..83508ccc 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -57,6 +57,8 @@ DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "vmlinuz"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1b227028..e9f92916 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -25,7 +25,7 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,fakeroot,build-essential,debhelper"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"

DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "${@https_support(d)}"

DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.
+#

+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+ fakeroot \
+ build-essential \
+ debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+ ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Perform all config fragments checking in Debian way.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../linux/linux-mainline_5.4.70.bb | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
index 980e137b..28e51c0d 100644
--- a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
+++ b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
@@ -25,11 +25,13 @@ LINUX_VERSION_EXTENSION = "-isar"

# For testing purposes only
dpkg_configure_kernel_append() {
- if ! grep "# CONFIG_MTD is not set" ${S}/${KERNEL_BUILD_DIR}/.config && \
- ! grep "# CONFIG_MTD_UBI is not set" ${S}/${KERNEL_BUILD_DIR}/.config; then
- grep "# CONFIG_UBIFS_FS is not set" ${S}/${KERNEL_BUILD_DIR}/.config || \
- bbfatal "Self-check failed: CONFIG_UBIFS_FS still enabled"
- fi
- grep "CONFIG_ROOT_NFS=y" ${S}/${KERNEL_BUILD_DIR}/.config || \
- bbfatal "Self-check failed: CONFIG_ROOT_NFS not enabled"
+cat << EOF | sed -i '/^override_dh_auto_build/ r /dev/stdin' ${S}/debian/rules
+ if ! grep "# CONFIG_MTD is not set" \$(O)/.config && \\
+ ! grep "# CONFIG_MTD_UBI is not set" \$(O)/.config; then \\
+ grep "# CONFIG_UBIFS_FS is not set" \$(O)/.config || \\
+ (echo "Self-check failed: CONFIG_UBIFS_FS still enabled" && exit 1); \\
+ fi
+ grep "CONFIG_ROOT_NFS=y" \$(O)/.config || \\
+ (echo "Self-check failed: CONFIG_ROOT_NFS not enabled" && exit 1)
+EOF
}
--
2.20.1

[Uladzimir Bely]

This preinstalls gbp depends like git-buildpackage and pristine-tar
in basic schroot environment.

So, there is no need to do such kind of installation
with every gpb recipe like cowsay.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-gbp.bbclass | 4 ----
meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 2 ++
2 files changed, 2 insertions(+), 4 deletions(-)

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 7ebbf2e0..436fad0f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,10 +12,6 @@ PATCHTOOL ?= "git"

GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

-do_install_builddeps_append() {
- schroot_install "${GBP_DEPENDS}"
-}
-

SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"

dpkg_runbuild_prepend() {
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..b6b73550 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -12,6 +12,8 @@ inherit rootfs

SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
+ git-buildpackage \
+ pristine-tar \
build-essential \
debhelper \
"
--
2.20.1

[Uladzimir Bely]

Similar to devshell for buildchroot, sbuildshell allows
to enter schroot environment with interactive terminal

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 474333e7..dcdd638b 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -155,3 +155,22 @@ schroot_run() {
remove_mounts
schroot_delete_configs
}
+
+python do_sbuildshell() {
+ import sys
+
+ oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib')
+ sys.path.insert(0, oe_lib_path)
+
+ bb.build.exec_func('schroot_create_configs', d)
+
+ buildchroot = d.getVar('SBUILD_CHROOT')
+ rundir = "/"
+ termcmd = "schroot -d / -c {0} -u root -- sh -c 'cd {1}; $SHELL -i'"
+ oe_terminal(termcmd.format(buildchroot, rundir), "Isar devshell", d)
+
+ bb.build.exec_func('schroot_delete_configs', d)
+}
+
+addtask sbuildshell after do_prepare_build
+do_sbuildshell[nostamp] = "1"
--
2.20.1

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 26 ++++++++------------------
1 file changed, 8 insertions(+), 18 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 861b2cb7..f530df58 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -97,34 +97,25 @@ python() {
}

do_apt_fetch() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_fetch after do_unpack before do_apt_unpack
do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

-# Add dependency from the correct buildchroot: host or target
-do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] = "${SCHROOT_DEP}"

do_apt_unpack() {
rm -rf ${S}
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -133,8 +124,7 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_unpack after do_apt_fetch before do_patch
--
2.20.1

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 84c99604..01a7c5be 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -83,6 +83,24 @@ apt install \
pristine-tar
```

+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+
If your host is >= buster, also install the following package.
```
apt install python3-distutils
--
2.20.1

[Uladzimir Bely]

This removes remaining buildchroot-related tasks and variables
from dpkg-base class while there are not used anymore.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 43 +++-------------------------------
meta/classes/dpkg.bbclass | 4 ----
2 files changed, 3 insertions(+), 44 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f530df58..d01ef5e8 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -5,7 +5,6 @@
# SPDX-License-Identifier: MIT

inherit sbuild
-inherit buildchroot
inherit debianize
inherit terminal
inherit repository
@@ -163,29 +162,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"

-BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
-
-dpkg_do_mounts() {
- mkdir -p ${BUILDROOT}
- sudo mount --bind ${WORKDIR} ${BUILDROOT}
-
- buildchroot_do_mounts
-}
-
-dpkg_undo_mounts() {
- i=0
- while ! sudo umount ${BUILDROOT}; do
- sleep 0.1
- if [ `expr $i % 100` -eq 0 ] ; then
- bbwarn "${BUILDROOT}: Couldn't unmount ($i), retrying..."
- fi
- if [ $i -ge 10000 ]; then
- bbfatal "${BUILDROOT}: Couldn't unmount after timeout"
- fi
- i=`expr $i + 1`
- done
- sudo rmdir ${BUILDROOT}
-}
+do_prepare_build[depends] = "${SCHROOT_DEP}"

# Placeholder for actual dpkg_runbuild() implementation
dpkg_runbuild() {
@@ -193,22 +170,17 @@ dpkg_runbuild() {
}

python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)
- bb.build.exec_func("dpkg_do_mounts", d)

+ bb.build.exec_func('schroot_create_configs', d)

try:
bb.build.exec_func("dpkg_runbuild", d)
finally:
- bb.build.exec_func("dpkg_undo_mounts", d)
- bb.utils.unlockfile(lock)

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask dpkg_build before do_build

do_dpkg_build[depends] = "${SCHROOT_DEP}"

-KEEP_INSTALLED_ON_CLEAN ?= "0"
-
CLEANFUNCS += "deb_clean"

deb_clean() {
@@ -217,15 +189,6 @@ deb_clean() {
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d80deeab..355da541 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -12,8 +12,6 @@ dpkg_runbuild() {

E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

- schroot_create_configs
-
distro="${DISTRO}"

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

distro="${HOST_DISTRO}"
@@ -39,6 +37,4 @@ dpkg_runbuild() {

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
-
- schroot_delete_configs
}
--
2.20.1

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 4 +++-
meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 2 --
3 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 535d456a..84c99604 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -78,7 +78,9 @@ apt install \
qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```

If your host is >= buster, also install the following package.

diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 436fad0f..c9cc487e 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -15,8 +15,10 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"

dpkg_runbuild_prepend() {

- schroot_run -d ${PP}/${PPS} -c ${SBUILD_CHROOT} -- \
+ sh -c "
+ cd ${WORKDIR}/${PPS}

gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+ "

# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
# for compatibility with gbp version froms debian-stretch. In newer distros
# it's possible to use a subcommand `export-orig --pristine-tar`

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index b6b73550..177a8a6d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -12,8 +12,6 @@ inherit rootfs

SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
- git-buildpackage \
- pristine-tar \
build-essential \
debhelper \
"
--
2.20.1

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 43 ++++++++++++++++++---------------------
1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index d49648be..d80deeab 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,29 +5,6 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-# Install build dependencies for package
-do_install_builddeps() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi

- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-

ISAR_APT_REPO ?= "deb [trusted=yes] file:///isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"

# Build package from sources using build script

@@ -37,11 +14,31 @@ dpkg_runbuild() {

schroot_create_configs

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+
+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives/"
+ ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+ ( flock 9
+ grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+ ) 9>"${TMPDIR}/sbuildrc.lock"

+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
--starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest \

+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
schroot_delete_configs
}
--
2.20.1

[Uladzimir Bely]

sbuild internally doesn apt-get update and apt-get install <builddeps>.
If isar-apt repo is changed at that time by deploy_deb task for another
package going in parallel, apt won't be able to download dependency
from this repo.
We may copletely lock isar-apt completely for dpkg_build, but this
slows (serializes) the whole build. Instead, it's better to copy
current isar-apt into the package workidir under the lock and use
copied isar-apt repo for building.
While "-l' option for `cp` is used, it won't occupy much additional
space on disk.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 13 ++++++++++++-
meta/classes/sbuild.bbclass | 2 --
2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 355da541..a9a2b3e2 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,7 +5,18 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-ISAR_APT_REPO ?= "deb [trusted=yes] file:///isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
+# Install build dependencies for package
+do_install_isarapt() {
+ # Make local copy of isar-apt not affected by other parallel tasks
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+
+addtask install_isarapt after do_prepare_build before do_dpkg_build
+do_install_isarapt[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

# Build package from sources using build script

dpkg_runbuild() {
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 799f210e..73a5fd6a 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -70,8 +70,6 @@ EOF

cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"

sbuild_fstab="${SBUILD_CONF_DIR}/fstab"

- fstab_isarapt="${DEPLOY_DIR}/isar-apt /isar-apt none rw,bind 0 0"
- grep -qxF "${fstab_isarapt}" ${sbuild_fstab} || echo "${fstab_isarapt}" >> ${sbuild_fstab}
fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

--
2.20.1

[Uladzimir Bely]

This adds mounts for base-apt inside schroot and
adds import/export deb files to/from schroot.

So that it becomes possible to run second `cached` build
from local base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 80f9ef25..474333e7 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -72,6 +72,8 @@ EOF

fstab_isarapt="${DEPLOY_DIR}/isar-apt /isar-apt none rw,bind 0 0"

grep -qxF "${fstab_isarapt}" ${sbuild_fstab} || echo "${fstab_isarapt}" >> ${sbuild_fstab}

+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

if [ -d ${DL_DIR} ]; then

fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

@@ -107,11 +109,18 @@ sbuild_export() {
schroot_install() {
schroot_create_configs
APTS="$1"
- #TODO deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
+

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import ${SCHROOT_DIR} ${distro}

schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
apt install -y -o Debug::pkgProblemResolver=yes \

--no-install-recommends --download-only ${APTS}
- #TODO deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
+ deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+

schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
apt install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends ${APTS}

--
2.20.1

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 10 +++++-----
meta/classes/sbuild.bbclass | 3 +++
2 files changed, 8 insertions(+), 5 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index acf967ac..861b2cb7 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -256,16 +256,16 @@ python do_devshell() {

oe_lib_path = os.path.join(d.getVar('LAYERDIR_core'), 'lib')

sys.path.insert(0, oe_lib_path)

- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)

isar_export_proxies(d)

- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)

+ termcmd = "schroot -d / -c {0} -u root -- sh -c 'cd {1}; $SHELL -i'"

+ oe_terminal(termcmd.format(schroot, pp_pps), "Isar devshell", d)

- bb.build.exec_func('dpkg_undo_mounts', d)

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index dcdd638b..799f210e 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -75,6 +75,9 @@ EOF

fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"

grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}
+

if [ -d ${DL_DIR} ]; then
fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}

--
2.20.1

[Jan Kiszka]

Could we wrap this long line?

> +KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
> +endif
> +
> +# With sbuild `dh clean` is called before chroot's apt database updated. So,
> +# KDIR is empty at that moment and the override causes error. Will skip it.
> +ifneq ($(KDIR),)
> override_dh_auto_clean:
> $(MAKE) -C $(KDIR) M=$(PWD) clean
> +endif

This looks wrong, or at least fragile. Why can we live without a proper
clean on the kernel tree?

Hmm...

> +
> + # restore execute permissions
> + chmod a+x ${S}/debian/rules

The mode should be fixed (aligned to the template) in template.bbclass
to avoid bothering every recipe with such boilerplate code. I'm seeing
this in other changes as well.

With all those "do not use shell env" - how will a downstream user
detect that a recipe is not properly converted? That looks like a nice
trap during migration to sbuild (because of silent / non-obvious errors).

Jan

--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[Jan Kiszka]

Oh, and then you still also add sbuild_export - why not use that for
cases that are ugly to convert (like this one)?

[Henning Schild]

Did not look into any of the patches so far ... but the number 24 makes
it close to impossible to review. Can this q be split up to ease review?

Henning

Am Fri, 19 Nov 2021 13:13:09 +0100
schrieb Uladzimir Bely <ub...@ilbers.de>:

> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
>
> Base schroot image is created with sbuild-chroot-target (or
> sbuild-chroot-host in case of cross-build) recipe. These images are
> based on isar-bootstrap ones, but include some build-related stuff
> preinstalled.
>
> To use this changes you need to have sbuild and schroot installed and
> configured. Current user should be added to sbuild group.
>
> Changes since v1:
> - parallel builds with different BUILD_DIR are supported
> - parallel multiconfig targets in one build are supported
> - per-task schroot configuration in /etc/schroot/ is now used
> - patchset now passes Jenkins CI (so patches changes RFC => PATCH)
>
> Current limitations:
> - there is an unsolved problem with building foreigh architectures
> using kas-docker.
>
> - qemuamd64 images are buildable in kas-docker, but some changes
> in kas are reqired (like installing additional packages and adding
> 'builder' user to 'sbuild group). I used the following changes to
> build ghcr.io/siemens/kas/kas-isar:2.6.2-sbuild image based on
> current ghcr.io/siemens/kas/kas-isar:2.6.2:
> https://github.com/WiseLord/kas/commit/5b7b02aa33
>
> - parallel building of several packages requires more free space
> than current buildchroot-based architecture uses. This happens due
> the sbuild architecture which uses some basic layer (common for all
> packages) and per-package separate layer (where builddeps are
> downloaded and installed).
>
> - recently merged to 'next' ccache support is not yet implemented
> in case of sbuild used.
>
> For experiments, the code can be taken from Isar 'ubely/sbuild'
> branch. This is an dev branch, so sometimes it can be force-pushed
> when some fixes are applied of rebased to 'next' branch.
>
> Anton Mikanovich (12):
> dpkg: Install raw package files to source root
> isar-bootstrap: Export bootstrap to schroot config
> linux-module: Do not use shell environment
> u-boot: Do not use shell environment
> trusted-firmware: Do not use shell environment
> optee-os: Do not use shell environment
> kselftest: Do not use shell environment
> dpkg: Build packages with sbuild
> sbuild: Introduce environment variables export API
> dpkg-gbp: Migrate to schroot
> linux-mainline: Move cfg fragment test to debian/rules
> linux-custom: Prepare kernel config inside sbuild
>
> Uladzimir Bely (12):
> dpkg-gbp: Use separate command to export tarball
> sbuild: Add recipes for host and target rootfs to run sbuild
> sbuild: Mount base-apt in schroot
> sbuild: Add sbuildshell task
> dpkg-gbp: Preinstall gbp utils in schroot
> dpkg: Remove builddeps install task.
> dpkg-base: Switch devshell to use schroot
> dpkg-base: Switch apt_fetch and apt_unpack to use schroot
> dpkg-base: Cleanup from buildchroot parts.
> dpkg-gbp: Use host tools for dsc preparation
> doc: Add sbuild-related documentation
> sbuild: Replace isar-apt mounting with copying
>
> doc/user_manual.md | 22 ++-
> .../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
> .../linux/linux-mainline_5.4.70.bb | 16 +-
> meta/classes/dpkg-base.bbclass | 80 ++------
> meta/classes/dpkg-gbp.bbclass | 25 +--
> meta/classes/dpkg-raw.bbclass | 8 +-
> meta/classes/dpkg.bbclass | 56 +++---
> meta/classes/sbuild.bbclass | 177
> ++++++++++++++++++ meta/conf/bitbake.conf |
> 2 + .../files/debian/{rules => rules.tmpl} | 4 +-
> meta/recipes-bsp/optee-os/optee-os-custom.inc | 10 +-
> .../files/debian/{rules => rules.tmpl} | 4 +-
> .../trusted-firmware-a-custom.inc | 10 +-

> .../u-boot/files/debian/{rules => rules.tmpl} | 2 +-

> meta/recipes-bsp/u-boot/u-boot-custom.inc | 10 +-
> .../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
> .../sbuild-chroot/sbuild-chroot-target.bb | 10 +
> .../sbuild-chroot/sbuild-chroot.inc | 36 ++++

> .../kselftest/files/{rules => rules.tmpl} | 0

> meta/recipes-kernel/kselftest/kselftest.inc | 21 +--
> .../files/debian/{rules => rules.tmpl} | 14 +-
> meta/recipes-kernel/linux-module/module.inc | 23 +--
> meta/recipes-kernel/linux/linux-custom.inc | 7 +-
> 23 files changed, 382 insertions(+), 170 deletions(-)
> create mode 100644 meta/classes/sbuild.bbclass

> rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl}

> (75%) rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules
> => rules.tmpl} (77%) rename

> meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)

> create mode 100644
> meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb create mode
> 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
> create mode 100644

> meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc rename

> meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

[Jan Kiszka]

On 19.11.21 13:13, Uladzimir Bely wrote:

> From: Anton Mikanovich <ami...@ilbers.de>
>
> In case there is no ability to pass variables through template
> mechanism we need to keep fallback API to migrate from bare export.
> Using it with optee-os-stm32mp15x recipe as example.
>
> Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
> ---
> .../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
> meta/classes/sbuild.bbclass | 13 +++++++++++++
> 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> index 0fa2005d..19aa763a 100644
> --- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> +++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
> @@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
> # Set version manually to PV, the tarball does not contain any hint.
> # Alternative: pull from git and add git as build dependency.
> dpkg_runbuild_prepend() {
> - export TEE_IMPL_VERSION=${PV}
> + sbuild_export TEE_IMPL_VERSION=${PV}

build_export - avoids having to rename this once again when we should
switch to something != sbuild.

Again, I think this is the nicer way to make the recipes sbuild
compatible (less invasive, less boilerplate than templating).

Jan

> }
> diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
> index d6e5b926..5f9f7691 100644
> --- a/meta/classes/sbuild.bbclass
> +++ b/meta/classes/sbuild.bbclass
> @@ -85,3 +85,16 @@ schroot_delete_configs() {
> rm -f "${SCHROOT_CONF_FILE}"
> EOSUDO
> }
> +
> +sbuild_export() {
> + SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
> + VAR_LINE="'${1%%=*}' => '${1#*=}',"
> + if [ -s "${SBUILD_CONFIG}" ]; then
> + sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
> + else
> + echo "\$build_environment = {" > ${SBUILD_CONFIG}
> + echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
> + echo "};" >> ${SBUILD_CONFIG}
> + fi
> + export SBUILD_CONFIG="${SBUILD_CONFIG}"
> +}
>

--

[Jan Kiszka]

On 19.11.21 22:48, Henning Schild wrote:
> Did not look into any of the patches so far ... but the number 24 makes
> it close to impossible to review. Can this q be split up to ease review?

I don't think 24 is a show-stopper per se. The preparatory changes to
recipes are split along the recipe types, and that adds a certain
length. In that sense, the series is already split, but I also do not
see that those preparatory patches need to come separately.

Jan

[Uladzimir Bely]

In the email from Friday, 19 Nov 2021. 15:44:12 +03 user Jan Kiszka wrote:
> On 19.11.21 13:13, Uladzimir Bely wrote:
>
> > +KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
> > +endif
> > +
> > +# With sbuild `dh clean` is called before chroot's apt database updated.
> > So, +# KDIR is empty at that moment and the override causes error. Will
> > skip it. +ifneq ($(KDIR),)
> >
> > override_dh_auto_clean:
> > $(MAKE) -C $(KDIR) M=$(PWD) clean
> >
> > +endif
>
> This looks wrong, or at least fragile. Why can we live without a proper
> clean on the kernel tree?
>

> Jan

Actually, this workaround with `ifneq ($(KDIR),)` I added to the patch after
the commit https://github.com/ilbers/isar/commit/248436880eb2 was applied to
`next`.

This change doesn't cancel a proper clean on the kernel. Here are some example
logs:

`next`: https://gist.github.com/WiseLord/0fb98bc036532ec03e4f763b2ba06a41
`sbuild`: https://gist.github.com/WiseLord/3c35d477efc025bcc9cd3c5cbd3dbc5d

On current `next` code dh_clean (and override-related `make ... clean` on
kernel code) is executed by dpkg-buildpackage. At this point KDIR is set to a
proper value, so it works as expected.

With sbuild dh_clean happens to be executed twice:
- 1: immediately after sbuild is run, executed by sbuild itself
- 2: after few stages (like downloading deps and so on), executed by dpkg-
buildpackage.

The issue is that at 1st execution KDIR is not set, so `make ... clean`
(coming with override) produces wrong command.

That's why this workaround (skip `make ... clean` in case of empty KDIR) was
added. Proper cleaning is still performed, but on the second call of dh_clean.

Probably, there is a way to say sbuild not to run this 'generic' dh_clean and
use just one from dpkg-buildpackage later...

--
Uladzimir Bely
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[Uladzimir Bely]

In the email from Friday, 19 Nov 2021 г. 15:13:09 +03 user Uladzimir Bely
wrote:

> This is a patchset showing how sbuild/schroot tools can be integrated
> into Isar build system.
>

> ...

>
> Current limitations:
> - there is an unsolved problem with building foreigh architectures
> using kas-docker.
>
> - qemuamd64 images are buildable in kas-docker, but some changes
> in kas are reqired (like installing additional packages and adding
> 'builder' user to 'sbuild group). I used the following changes to
> build ghcr.io/siemens/kas/kas-isar:2.6.2-sbuild image based on
> current ghcr.io/siemens/kas/kas-isar:2.6.2:
> https://github.com/WiseLord/kas/commit/5b7b02aa33
>

> ...

> meta/recipes-kernel/linux-module/files/debian/{rules => rules.tmpl} (50%)

Here I would like to add some info regarding running the patchset with `kas-
docker` and/or using Gitlab CI.

Initially, the patchset was checked in Jenkins CI (that doesn't use docker)
and sent to the maillist then. But there are some docker-related issues solved
a bit later. So here I'll try to explain them (and this is to be added to user
docs in the following patchset version), so someone could play with sbuild
patchset.

1. Additional packages required.

The patchset requires 'schroot' and 'sbuild' packages to be installed on the
host. Also, they both should have at least "bullseye" version, otherwise some
things won't work properly (like caching .deb via 'sbuild' or foreign
architectures support in 'schroot').

While current (ghcr.io/siemens/kas/kas-isar:2.6.2) image doesn't include the
required packages, I temporary prepared and upuloaded one based on it
(ghcr.io/wiselord/kas-isar:2.6.2-sbuild). Look at Dockerfile.isar.sbuild chunk
in the patch https://github.com/WiseLord/kas/commit/38f4f11f11 for details.

Gitlab

Gitlab uses .gitlab-ci.yml for setup. So, to use modified image in Gitlab CI
user should simply change the first line:

> - image: ghcr.io/siemens/kas/kas-isar:latest
> + image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild

This is required until official kas-isar image have everything included.

2. User should be added to 'sbuild' group.

In case of 'kas-docker' it happend to be a bit tricky, because 'builder' user
is created 'on the fly', when container is run. I'm not an expert in 'kas' so
I simply pathced container-entrypoint to add user to sbuild group. Look at
`container.entrypoint` chunk in the patch for details.

3. Overlayfs restrictions.

Docker uses overlayfs to mount dockerimage rootfs. Schroot uses /var/lib/
schroot/union/{overlay,underlay} directories to keep 'basic' image and
temporary layers on top of it.

So we happen to have 'overlayfs over overlayfs' situation that is not
supported by overlayfs kernel driver.

The solutionis to use an external volume for the /var/lib/schroot/union/. Look
at 'kas-container' chunk in the patch for details.

Gitlab

It seems there is no way to say gitlab using external volume via the
configuration file `.gitlab-ci.yml`. But it can be done by the following
changes in /etc/gitlab/runner/config.toml:

> -volumes = ["/cache"]
> +volumes = ["/m/ws-10/schroot-10a/union:/var/lib/schroot/union", "/cache"]

So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on host.
Also, there should be empty "overlay" and "underlay" directories created in
it.

[Jan Kiszka]

Thanks for the explanation. How about clarifying that no clean is lost
in the commit message?

[Uladzimir Bely]

Regarding restoring permissions: this case can be covered by `[PATCH]
template: Make templates passthrough` (https://groups.google.com/u/0/g/isar-users/c/Pmr2jUjviz8) which automatically keeps debian/<file>.tmpl permissions
when transforming it to debian/<file>. Internally, I've included this patch
and removed all these parts with `chmod` from the recipes and it looks working
(anyway, CI tests are still needed).

And you are right, there might be some way to warn users about non-converted
downstream recipes. It is worth thinking.

[Jan Kiszka]

On 23.11.21 14:05, Uladzimir Bely wrote:

> In the email from Friday, 19 Nov 2021 г. 15:13:09 +03 user Uladzimir Bely
> wrote:
>> This is a patchset showing how sbuild/schroot tools can be integrated
>> into Isar build system.
>>
>> ...
>>
>> Current limitations:
>> - there is an unsolved problem with building foreigh architectures
>> using kas-docker.

What exactly is the problem here?

I've commented on some aspects of you kas patch. It generally looks
feasible, just some smaller details should be improved.

> Gitlab
>
> It seems there is no way to say gitlab using external volume via the
> configuration file `.gitlab-ci.yml`. But it can be done by the following
> changes in /etc/gitlab/runner/config.toml:
>
>> -volumes = ["/cache"]
>> +volumes = ["/m/ws-10/schroot-10a/union:/var/lib/schroot/union", "/cache"]
>
> So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on host.
> Also, there should be empty "overlay" and "underlay" directories created in
> it.

That's also not unrealistic, given that we already need to provide
special runners for the purpose of granting privileges and allowing
binfmt_misc. One target would be
https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci, the backend
for many of our gitlab CI runners that are Isar-compatible.

[Uladzimir Bely]

In mail from Friday, 26 Nov 2021. 09:43:58 +03 user Jan Kiszka wrote:
> On 23.11.21 14:05, Uladzimir Bely wrote:
> >> Current limitations:
> >> - there is an unsolved problem with building foreigh architectures
> >> using kas-docker.
>
> What exactly is the problem here?
>

Actually, the problem was related to UsrMerge. Our Jenkins root is also based
on buster, but /bin, /lib are symlinks to /usr/bin, /usr/lib and so on. While
in buster-slim used in kas-docker /bin, /lib are separate directories. And
schroot version used in buster couldn't work in this case (qemu-*-static
packages couldn't find some libraries).

But when I schroot version from bullseye seems to support both kinds of
rootfs. So, no changes were needed in sbuild patches, I just preinstalled
newer schroot (from bullseye) in kas-docker image (in addition to a newer
sbuild), and it works.

> >> ...

>
> I've commented on some aspects of you kas patch. It generally looks
> feasible, just some smaller details should be improved.
>

Yes, I saw your comments on github. I'm not yet experienced in kas
development, so my implementation is not ideal.

> > ...

> > So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on
> > host. Also, there should be empty "overlay" and "underlay" directories
> > created in it.
>
> That's also not unrealistic, given that we already need to provide
> special runners for the purpose of granting privileges and allowing
> binfmt_misc. One target would be
> https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci, the backend
> for many of our gitlab CI runners that are Isar-compatible.
>
> Jan

Ok, I'll take a look at it.

Currently I'm trying to find/fix possible issues in downstreams. I took meta-
iot2050 for experiments. For now it seems to use quite old docker image and
isar codebase. At first, I made sure it can build with no errors current Isar
'next' branch with using my modified docker image. But for now it is not yet
able to build even partial (preparation patches only) sbuild patchset and I'm
trying to understand and fix the issues.

[Jan Kiszka]

On 26.11.21 09:03, Uladzimir Bely wrote:
> In mail from Friday, 26 Nov 2021. 09:43:58 +03 user Jan Kiszka wrote:
>> On 23.11.21 14:05, Uladzimir Bely wrote:
>>>> Current limitations:
>>>> - there is an unsolved problem with building foreigh architectures
>>>> using kas-docker.
>>
>> What exactly is the problem here?
>>
>
> Actually, the problem was related to UsrMerge. Our Jenkins root is also based
> on buster, but /bin, /lib are symlinks to /usr/bin, /usr/lib and so on. While
> in buster-slim used in kas-docker /bin, /lib are separate directories. And
> schroot version used in buster couldn't work in this case (qemu-*-static
> packages couldn't find some libraries).
>
> But when I schroot version from bullseye seems to support both kinds of
> rootfs. So, no changes were needed in sbuild patches, I just preinstalled
> newer schroot (from bullseye) in kas-docker image (in addition to a newer
> sbuild), and it works.

Ah, ok. Then we will be automatically fine when moving the complete kas
image to bullseye.

>
>>>> ...
>>
>> I've commented on some aspects of you kas patch. It generally looks
>> feasible, just some smaller details should be improved.
>>
>
> Yes, I saw your comments on github. I'm not yet experienced in kas
> development, so my implementation is not ideal.
>

No problem, that can be sorted out. If you have questions, just drop
them on kas mailing list.

>>> ...
>>> So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on
>>> host. Also, there should be empty "overlay" and "underlay" directories
>>> created in it.
>>
>> That's also not unrealistic, given that we already need to provide
>> special runners for the purpose of granting privileges and allowing
>> binfmt_misc. One target would be
>> https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci, the backend
>> for many of our gitlab CI runners that are Isar-compatible.
>>
>> Jan
>
> Ok, I'll take a look at it.
>
> Currently I'm trying to find/fix possible issues in downstreams. I took meta-
> iot2050 for experiments. For now it seems to use quite old docker image and
> isar codebase. At first, I made sure it can build with no errors current Isar
> 'next' branch with using my modified docker image. But for now it is not yet
> able to build even partial (preparation patches only) sbuild patchset and I'm
> trying to understand and fix the issues.
>

Yeah, meta-io2050 dependencies were frozen for an upcoming release. Try
bumping them first and building those with unpatched Isar and unpatched
kas 2.6.2 (that should work, see
https://github.com/siemens/meta-iot2050/pull/205), then modify kas and
finally patch isar. If there are issues already in the first step, let
me know.

[Michael Adler]

Hi Uladzimir,

> It seems there is no way to say gitlab using external volume via the configuration file `.gitlab-ci.yml`

yes, because it is not universally applicable: there are various runner implementations [1] and the "volumes" feature is
not supported by every runner (e.g. SSH-based).

For CIP, we use custom Kubernetes runners. Although it is possible to setup custom volumes [2], that's only half of the
story: the other half is to ensure that these directories actually exist on the host, which in the case of
gitlab-cloud-ci [3] (used by CIP and Siemens) are ephemeral, i.e. they are dynamically created and destroyed based on
on the CI workload. It should be possible to support such volumes but it's definitely extra engineering/maintenance
effort.

> So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on host.

What happens if I choose not to provide such a volume? Is it mandatory for the build? Do I also need this for local
containerized builds?

Kind regards,
Michael

[1] https://docs.gitlab.com/runner/executors/
[2] https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes
[3] https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci

--
Michael Adler

Siemens AG
T RDA IOT SES-DE
Otto-Hahn-Ring 6
81739 München, Deutschland

Siemens Aktiengesellschaft: Vorsitzender des Aufsichtsrats: Jim Hagemann Snabe; Vorstand: Roland Busch, Vorsitzender; Klaus Helmrich, Cedrik Neike, Matthias Rebellius, Ralf P. Thomas, Judith Wiese; Sitz der Gesellschaft: Berlin und München, Deutschland; Registergericht: Berlin-Charlottenburg, HRB 12300, München, HRB 6684; WEEE-Reg.-Nr. DE 23691322

[Uladzimir Bely]

In mail from Friday, 26 Nov 2021. 15:09:21 +03 user Michael Adler wrote:
> Hi Uladzimir,
>
> > It seems there is no way to say gitlab using external volume via the
> > configuration file `.gitlab-ci.yml`
> yes, because it is not universally applicable: there are various runner
> implementations [1] and the "volumes" feature is not supported by every
> runner (e.g. SSH-based).
>
> For CIP, we use custom Kubernetes runners. Although it is possible to setup
> custom volumes [2], that's only half of the story: the other half is to
> ensure that these directories actually exist on the host, which in the case
> of gitlab-cloud-ci [3] (used by CIP and Siemens) are ephemeral, i.e. they
> are dynamically created and destroyed based on on the CI workload. It
> should be possible to support such volumes but it's definitely extra
> engineering/maintenance effort.
>
> > So, gitlab will use external ("/m/ws-10/schroot-10a/union") directory on
> > host.
> What happens if I choose not to provide such a volume? Is it mandatory for
> the build? Do I also need this for local containerized builds?
>

As I wrote, the problem that docker uses overlayfs to produce a working
chroot. So, /var/lib/schroot/union/ happens to be on overlayfs. Sbuild also
uses overlayfs - it mounts rootfs to /var/lib/schroot/union/underlay (as a
lower, read-only layer) and uses /var/lib/schroot/union/overlay/ handle rw
layer.
But, it is not supported by overlayfs (owerdir=/lower,upperdir=/
upper,workdir=/work) when upperdir is on overlayfs (from docker).

>
> Kind regards,
> Michael
>
> [1] https://docs.gitlab.com/runner/executors/
> [2] https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes
> [3] https://gitlab.com/cip-project/cip-testing/gitlab-cloud-ci


--

[Jan Kiszka]

On 26.11.21 13:09, Michael Adler wrote:
> Hi Uladzimir,
>
>> It seems there is no way to say gitlab using external volume via the configuration file `.gitlab-ci.yml`
>
> yes, because it is not universally applicable: there are various runner implementations [1] and the "volumes" feature is
> not supported by every runner (e.g. SSH-based).
>
> For CIP, we use custom Kubernetes runners. Although it is possible to setup custom volumes [2], that's only half of the
> story: the other half is to ensure that these directories actually exist on the host, which in the case of
> gitlab-cloud-ci [3] (used by CIP and Siemens) are ephemeral, i.e. they are dynamically created and destroyed based on
> on the CI workload. It should be possible to support such volumes but it's definitely extra engineering/maintenance
> effort.

We would not need those union mounts to be persist across jobs - just
like the rest the container generated during job execution and that is
not explicitly exported otherwise (artifacts, local cache,
self-uploads). We just need that union to be on something else than an
overlayfs.

[Jan Kiszka]

On 26.11.21 09:50, Jan Kiszka wrote:
> On 26.11.21 09:03, Uladzimir Bely wrote:
>> In mail from Friday, 26 Nov 2021. 09:43:58 +03 user Jan Kiszka wrote:
>>> On 23.11.21 14:05, Uladzimir Bely wrote:
>>>>> Current limitations:
>>>>> - there is an unsolved problem with building foreigh architectures
>>>>> using kas-docker.
>>>
>>> What exactly is the problem here?
>>>
>>
>> Actually, the problem was related to UsrMerge. Our Jenkins root is also based
>> on buster, but /bin, /lib are symlinks to /usr/bin, /usr/lib and so on. While
>> in buster-slim used in kas-docker /bin, /lib are separate directories. And
>> schroot version used in buster couldn't work in this case (qemu-*-static
>> packages couldn't find some libraries).
>>
>> But when I schroot version from bullseye seems to support both kinds of
>> rootfs. So, no changes were needed in sbuild patches, I just preinstalled
>> newer schroot (from bullseye) in kas-docker image (in addition to a newer
>> sbuild), and it works.
>
> Ah, ok. Then we will be automatically fine when moving the complete kas
> image to bullseye.
>

FYI: kas 3.x development has started, 'next' is now on bullseye.

[Uladzimir Bely]

This is a patchset showing how sbuild/schroot tools can be integrated
into Isar build system.

Base schroot image is created with sbuild-chroot-target (or
sbuild-chroot-host in case of cross-build) recipe. These images are based
on isar-bootstrap ones, but include some build-related stuff preinstalled.

To use this changes you need to have sbuild and schroot installed and

configured. Current user should be added to sbuild group.

If 'kas-container' is used, it should be modified to support sbuild:
- /var/lib/schroot/union/overlay should be externally mounted (-v option)
- export KAS_IMAGE_VERSION=2.6.2-sbuild
- export KAS_CONTAINER_IMAGE_PATH=ghcr.io/wiselord
Here I uploaded a docker image that has preinstalled software (sbuild,
schroot, etc) required to run sbuild. Later, the changes might be
included in official kas docker images.

If 'gitlab' is used, the similar changes are required:
- 'image: ghcr.io/wiselord/kas-isar:2.6.2-sbuild' in .gitlab-ci.yml
- external dir for schroot overlay should also be specified in
/etc/gitlab-runner/config.toml:
volumes = ["/path/to/overlay:/var/lib/schroot/union/overlay"]

The patchset was tested with 'meta-iot2050' and 'xenomai-images'
downstreams. Some patches are required for these downstreams,
appropriate pull requests will be created soon.

Changes since v2:
- patches reworked/squashed for easier reading and understanding
- fixed building foreigh architectures with kas-docker
- implemented support of ccache
- fixed devshell and devshell_nodeps

Changes since v1:
- parallel builds with different BUILD_DIR are supported
- parallel multiconfig targets in one build are supported
- per-task schroot configuration in /etc/schroot/ is now used
- patchset now passes Jenkins CI (so patches changes RFC => PATCH)

Current limitations:

- parallel building of several packages requires more free space in
comparison with buildchroot-based architecture. This happens due

the sbuild architecture which uses some basic layer (common for all
packages) and per-package separate layer (where builddeps are downloaded
and installed).

For experiments, the code can be taken from Isar 'ubely/sbuild' branch.

This is a dev branch, so sometimes it can be force-pushed when some

fixes are applied of rebased to 'next' branch.

First 9 patches (1..9) - are just preparation patches. With them, Isar will
continue using buildchroot-based architecture. So, potentially,
they can be merged as a subseries.

Remaining 11 patches (10..20) represent a sbuild-related changes.

Anton Mikanovich (10):
template: Make templates passthrough

linux-module: Do not use shell environment
u-boot: Do not use shell environment
trusted-firmware: Do not use shell environment
optee-os: Do not use shell environment
kselftest: Do not use shell environment

linux-mainline: Move cfg fragment test to debian/rules

dpkg: Build packages with sbuild
sbuild: Introduce environment variables export API

linux-custom: Prepare kernel config inside sbuild

Uladzimir Bely (10):

dpkg-gbp: Use separate command to export tarball

dpkg-gbp: Use host tools for dsc preparation

sbuild: Add recipes for host and target rootfs to run sbuild

sbuild: Introduce a class for another build method

dpkg: Remove builddeps install task.
dpkg-base: Switch devshell to use schroot
dpkg-base: Switch apt_fetch and apt_unpack to use schroot
dpkg-base: Cleanup from buildchroot parts.

doc: Add sbuild-related documentation
sbuild: add ccache support

doc/user_manual.md | 22 ++-
.../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
.../linux/linux-mainline_5.4.70.bb | 16 +-

meta/classes/dpkg-base.bbclass | 98 ++++------
meta/classes/dpkg-gbp.bbclass | 26 +--
meta/classes/dpkg.bbclass | 53 ++---
meta/classes/sbuild.bbclass | 184 ++++++++++++++++++
meta/classes/template.bbclass | 7 +-

meta/conf/bitbake.conf | 2 +
.../files/debian/{rules => rules.tmpl} | 4 +-

meta/recipes-bsp/optee-os/optee-os-custom.inc | 9 +-

.../files/debian/{rules => rules.tmpl} | 4 +-

.../trusted-firmware-a-custom.inc | 9 +-

.../u-boot/files/debian/{rules => rules.tmpl} | 2 +-

meta/recipes-bsp/u-boot/u-boot-custom.inc | 9 +-

.../sbuild-chroot/sbuild-chroot-host.bb | 13 ++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 +

.../sbuild-chroot/sbuild-chroot.inc | 39 ++++

.../kselftest/files/{rules => rules.tmpl} | 0
meta/recipes-kernel/kselftest/kselftest.inc | 21 +-

.../linux-module/files/debian/rules | 37 ----
.../linux-module/files/debian/rules.tmpl | 52 +++++
meta/recipes-kernel/linux-module/module.inc | 19 +-
meta/recipes-kernel/linux/linux-custom.inc | 7 +-
24 files changed, 437 insertions(+), 208 deletions(-)

create mode 100644 meta/classes/sbuild.bbclass
rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)
rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)
rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

delete mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl

--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove TF_A_PLATFORM and TF_A_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../files/debian/{rules => rules.tmpl} | 4 ++--
.../trusted-firmware-a/trusted-firmware-a-custom.inc | 9 ++-------
2 files changed, 4 insertions(+), 9 deletions(-)

rename meta/recipes-bsp/trusted-firmware-a/files/debian/{rules => rules.tmpl} (77%)

diff --git a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
similarity index 77%
rename from meta/recipes-bsp/trusted-firmware-a/files/debian/rules
rename to meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
index 70e1dd66..6dbf44db 100755
--- a/meta/recipes-bsp/trusted-firmware-a/files/debian/rules
+++ b/meta/recipes-bsp/trusted-firmware-a/files/debian/rules.tmpl
@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
endif

override_dh_auto_build:
- CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=$(TF_A_PLATFORM) \
- $(TF_A_EXTRA_BUILDARGS)
+ CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLAT=${TF_A_PLATFORM} \
+ ${TF_A_EXTRA_BUILDARGS}

%:
dh $@
diff --git a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
index 1142bfce..64153c6b 100644
--- a/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
+++ b/meta/recipes-bsp/trusted-firmware-a/trusted-firmware-a-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= ""

PROVIDES += "trusted-firmware-a-${TF_A_NAME}"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "TF_A_NAME DEBIAN_BUILD_DEPENDS TF_A_PLATFORM TF_A_EXTRA_BUILDARGS"

do_prepare_build() {
cp -r ${WORKDIR}/debian ${S}/

@@ -36,8 +36,3 @@ do_prepare_build() {
${S}/debian/trusted-firmware-a-${TF_A_NAME}.install
done
}
-
-dpkg_runbuild_prepend() {
- export TF_A_PLATFORM="${TF_A_PLATFORM}"
- export TF_A_EXTRA_BUILDARGS="${TF_A_EXTRA_BUILDARGS}"
-}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

There is no need to store original template files after the conversion
in real scenarios. To make working folder little cleaner we can remove
them. This can be disabled with TEMPLATE_FILES_KEEP variable.
Moreover output file should have exactly the same flags as input one,
which is usefull for the cases like debian/rules or other executables.
So we should copy this part of metadata after the conversion.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/template.bbclass | 7 ++++++-
meta/recipes-kernel/linux/linux-custom.inc | 3 ---
2 files changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
index fb9d1186..f5760e15 100644
--- a/meta/classes/template.bbclass
+++ b/meta/classes/template.bbclass
@@ -4,11 +4,12 @@
# SPDX-License-Identifier: MIT

TEMPLATE_FILES ?= ""
+TEMPLATE_FILES_KEEP ?= "0"
TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"

do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
python do_transform_template() {
- import subprocess, contextlib
+ import subprocess, contextlib, shutil

workdir = os.path.normpath(d.getVar('WORKDIR', True))

@@ -56,5 +57,9 @@ python do_transform_template() {
stdout=output, env=env))
if process.wait() != 0:
bb.fatal("processing of template failed")
+
+ shutil.copymode(template_file, output_file)
+ if d.getVar('TEMPLATE_FILES_KEEP', True) != '1':
+ os.remove(template_file)
}
addtask do_transform_template after do_unpack
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ed89aa09..57740860 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -122,9 +122,6 @@ do_prepare_build_prepend() {
rm -rf ${S}/debian

cp -r ${WORKDIR}/debian ${S}/

- # remove templates from the source tree
- find ${S}/debian -name *.tmpl | xargs rm -f
-
# rename install/remove hooks to match user-specified name for our linux-image package
mv ${S}/debian/linux-image.postinst ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postinst
mv ${S}/debian/linux-image.postrm ${S}/debian/linux-image-${KERNEL_NAME_PROVIDED}.postrm
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove U_BOOT_CONFIG and U_BOOT_BIN passing through export call.

So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../u-boot/files/debian/{rules => rules.tmpl} | 2 +-

meta/recipes-bsp/u-boot/u-boot-custom.inc | 9 ++-------
2 files changed, 3 insertions(+), 8 deletions(-)

rename meta/recipes-bsp/u-boot/files/debian/{rules => rules.tmpl} (94%)

diff --git a/meta/recipes-bsp/u-boot/files/debian/rules b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
similarity index 94%
rename from meta/recipes-bsp/u-boot/files/debian/rules
rename to meta/recipes-bsp/u-boot/files/debian/rules.tmpl
index 3d667620..806b01fe 100755
--- a/meta/recipes-bsp/u-boot/files/debian/rules
+++ b/meta/recipes-bsp/u-boot/files/debian/rules.tmpl
@@ -13,7 +13,7 @@ SET_CROSS_BUILD_TOOLS=CROSS_BUILD_TOOLS=y
endif

override_dh_auto_build:
- $(MAKE) $(PARALLEL_MAKE) $(U_BOOT_CONFIG)
+ $(MAKE) $(PARALLEL_MAKE) ${U_BOOT_CONFIG}
$(MAKE) $(PARALLEL_MAKE) ${U_BOOT_BIN}
$(MAKE) -n u-boot-initial-env >/dev/null 2>&1; if [ $$? -ne 2 ]; then \
$(MAKE) $(PARALLEL_MAKE) u-boot-initial-env; \
diff --git a/meta/recipes-bsp/u-boot/u-boot-custom.inc b/meta/recipes-bsp/u-boot/u-boot-custom.inc
index 9984d8cc..2af7ada1 100644
--- a/meta/recipes-bsp/u-boot/u-boot-custom.inc
+++ b/meta/recipes-bsp/u-boot/u-boot-custom.inc
@@ -26,8 +26,8 @@ python() {

DEBIAN_BUILD_DEPENDS ?= "bc, bison, flex, device-tree-compiler, git"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "MACHINE DEBIAN_BUILD_DEPENDS U_BOOT_CONFIG U_BOOT_BIN"

U_BOOT_TOOLS_PACKAGE ?= "0"
U_BOOT_CONFIG_PACKAGE ?= "0"
@@ -71,8 +71,3 @@ fw_env.config /etc
EOF
fi
}
-
-dpkg_runbuild_prepend() {
- export U_BOOT_CONFIG="${U_BOOT_CONFIG}"
- export U_BOOT_BIN="${U_BOOT_BIN}"
-}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove KDIR and PN passing through export call.
KDIR can be prepared during package build. This also will allow not to
rely on previous builddeps install task.

To pass PN variable we can just migrate to template-based debian/rules

file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../linux-module/files/debian/rules | 37 -------------
.../linux-module/files/debian/rules.tmpl | 52 +++++++++++++++++++
meta/recipes-kernel/linux-module/module.inc | 19 ++-----
3 files changed, 55 insertions(+), 53 deletions(-)

delete mode 100755 meta/recipes-kernel/linux-module/files/debian/rules
create mode 100755 meta/recipes-kernel/linux-module/files/debian/rules.tmpl

diff --git a/meta/recipes-kernel/linux-module/files/debian/rules b/meta/recipes-kernel/linux-module/files/debian/rules
deleted file mode 100755
index 59720b37..00000000
--- a/meta/recipes-kernel/linux-module/files/debian/rules
+++ /dev/null
@@ -1,37 +0,0 @@
-#!/usr/bin/make -f
-
-# Debian rules for custom kernel module build
-#
-# This software is a part of ISAR.
-# Copyright (c) Siemens AG, 2018
-#
-# SPDX-License-Identifier: MIT
-
-export DEB_BUILD_OPTIONS=parallel=$(shell nproc)
-
-export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
-
-ifeq ($(DEB_HOST_GNU_CPU), arm)
-export ARCH=arm
-endif
-ifeq ($(DEB_HOST_GNU_CPU), aarch64)
-export ARCH=arm64
-endif
-ifeq ($(DEB_HOST_GNU_CPU), riscv64)
-export ARCH=riscv
-endif
-ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
-export ARCH=x86
-endif
-
-override_dh_auto_clean:
- $(MAKE) -C $(KDIR) M=$(PWD) clean
-
-override_dh_auto_build:
- $(MAKE) -C $(KDIR) M=$(PWD) modules
-
-override_dh_auto_install:

- $(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/$(PN) modules_install

-
-%:
- CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/files/debian/rules.tmpl b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
new file mode 100755
index 00000000..52453e5f
--- /dev/null
+++ b/meta/recipes-kernel/linux-module/files/debian/rules.tmpl
@@ -0,0 +1,52 @@
+#!/usr/bin/make -f
+
+# Debian rules for custom kernel module build
+#
+# This software is a part of ISAR.
+# Copyright (c) Siemens AG, 2018
+#
+# SPDX-License-Identifier: MIT
+
+export DEB_BUILD_OPTIONS=parallel=$(shell nproc)
+
+export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
+
+ifeq ($(DEB_HOST_GNU_CPU), arm)
+export ARCH=arm
+endif
+ifeq ($(DEB_HOST_GNU_CPU), aarch64)
+export ARCH=arm64
+endif
+ifeq ($(DEB_HOST_GNU_CPU), riscv64)
+export ARCH=riscv
+endif
+ifneq (,$(findstring 86,$(DEB_HOST_GNU_CPU)))
+export ARCH=x86
+endif
+

+# Custom kernels contain the build folder directly.
+KDIR := $(shell dpkg -L linux-headers-${KERNEL_NAME} | grep "/lib/modules/.*/build")
+ifeq ($(KDIR),)
+# Debian kernels install that folder indirectly via a dependency.
+KERNEL_DEP := $(shell dpkg-query -W -f '$${Depends}' linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')

+KDIR := $(shell dpkg -L $(KERNEL_DEP) | grep "/lib/modules/.*/build")
+endif
+

+# With some build systems like sbuild `dh clean` can be called twice:
+# first, by build system itself before chroot's apt database updated,
+# second, by dpkg-buildpackage during build. So, KDIR may be empty
+# in first case (while no dependencies are yet installed), and a broken
+# 'make ... clean' command is produced. Just skip override in this case.
+ifneq ($(KDIR),)
+override_dh_auto_clean:
+ $(MAKE) -C $(KDIR) M=$(PWD) clean
+endif
+
+override_dh_auto_build:
+ $(MAKE) -C $(KDIR) M=$(PWD) modules
+
+override_dh_auto_install:

+ $(MAKE) -C $(KDIR) M=$(PWD) INSTALL_MOD_PATH=$(PWD)/debian/${PN} modules_install

+
+%:
+ CFLAGS= LDFLAGS= dh $@ --parallel
diff --git a/meta/recipes-kernel/linux-module/module.inc b/meta/recipes-kernel/linux-module/module.inc
index 0515621a..9ae3af9b 100644

--- a/meta/recipes-kernel/linux-module/module.inc
+++ b/meta/recipes-kernel/linux-module/module.inc
@@ -22,8 +22,9 @@ AUTOLOAD ?= ""
inherit dpkg

TEMPLATE_FILES = "debian/control.tmpl \
- debian/changelog.tmpl"
-TEMPLATE_VARS += "KERNEL_NAME"
+ debian/changelog.tmpl \
+ debian/rules.tmpl"
+TEMPLATE_VARS += "KERNEL_NAME PN"

do_prepare_build() {

cp -r ${WORKDIR}/debian ${S}/

@@ -32,17 +33,3 @@ do_prepare_build() {

echo "echo $module >> /etc/modules" >> ${S}/debian/postinst
done
}
-
-dpkg_runbuild_prepend() {
- # Custom kernels contain the build folder directly.
- export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} linux-headers-${KERNEL_NAME} | \
- grep "/lib/modules/.*/build")
- if [ -z "$KDIR" ]; then
- # Debian kernels install that folder indirectly via a dependency.
- KERNEL_DEP=$(dpkg-query -W -f '${Depends}' --admindir=${BUILDCHROOT_DIR}/var/lib/dpkg \
- linux-headers-${KERNEL_NAME} | sed 's/.*\(linux-headers-[[:graph:]]*\).*/\1/')
- export KDIR=$(dpkg -L --root=${BUILDCHROOT_DIR} ${KERNEL_DEP} | \
- grep "/lib/modules/.*/build")
- fi
- export PN=${PN}

-}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Perform all config fragments checking in Debian way.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../linux/linux-mainline_5.4.70.bb | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
index 980e137b..28e51c0d 100644
--- a/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
+++ b/meta-isar/recipes-kernel/linux/linux-mainline_5.4.70.bb
@@ -25,11 +25,13 @@ LINUX_VERSION_EXTENSION = "-isar"

# For testing purposes only
dpkg_configure_kernel_append() {
- if ! grep "# CONFIG_MTD is not set" ${S}/${KERNEL_BUILD_DIR}/.config && \
- ! grep "# CONFIG_MTD_UBI is not set" ${S}/${KERNEL_BUILD_DIR}/.config; then
- grep "# CONFIG_UBIFS_FS is not set" ${S}/${KERNEL_BUILD_DIR}/.config || \
- bbfatal "Self-check failed: CONFIG_UBIFS_FS still enabled"
- fi
- grep "CONFIG_ROOT_NFS=y" ${S}/${KERNEL_BUILD_DIR}/.config || \
- bbfatal "Self-check failed: CONFIG_ROOT_NFS not enabled"
+cat << EOF | sed -i '/^override_dh_auto_build/ r /dev/stdin' ${S}/debian/rules
+ if ! grep "# CONFIG_MTD is not set" \$(O)/.config && \\
+ ! grep "# CONFIG_MTD_UBI is not set" \$(O)/.config; then \\
+ grep "# CONFIG_UBIFS_FS is not set" \$(O)/.config || \\
+ (echo "Self-check failed: CONFIG_UBIFS_FS still enabled" && exit 1); \\
+ fi
+ grep "CONFIG_ROOT_NFS=y" \$(O)/.config || \\
+ (echo "Self-check failed: CONFIG_ROOT_NFS not enabled" && exit 1)
+EOF
}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove KSELFTEST_ARGS passing through export call. The same
logic can be done by internal recipe parser and then pass variable to
template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../kselftest/files/{rules => rules.tmpl} | 0

meta/recipes-kernel/kselftest/kselftest.inc | 21 +++++++------------
2 files changed, 8 insertions(+), 13 deletions(-)

rename meta/recipes-kernel/kselftest/files/{rules => rules.tmpl} (100%)

diff --git a/meta/recipes-kernel/kselftest/files/rules b/meta/recipes-kernel/kselftest/files/rules.tmpl
similarity index 100%
rename from meta/recipes-kernel/kselftest/files/rules
rename to meta/recipes-kernel/kselftest/files/rules.tmpl
diff --git a/meta/recipes-kernel/kselftest/kselftest.inc b/meta/recipes-kernel/kselftest/kselftest.inc
index 2a76028b..6187d8e4 100644
--- a/meta/recipes-kernel/kselftest/kselftest.inc
+++ b/meta/recipes-kernel/kselftest/kselftest.inc
@@ -33,26 +33,21 @@ DEBIAN_BUILD_DEPENDS ?= " \
llvm:native, \
"

-SRC_URI += "file://rules"
+SRC_URI += "file://rules.tmpl"
S = "${WORKDIR}/linux-${PV}"

+TEMPLATE_FILES = "rules.tmpl"
+TEMPLATE_VARS += "KSELFTEST_ARGS"
+
KSELFTEST_TARGETS ?= ""
KSELFTEST_SKIP_TARGETS ?= ""
KSELFTEST_FORCE_TARGETS ?= "0"

+KSELFTEST_ARGS = "${@ "TARGETS=\"${KSELFTEST_TARGETS}\"" if d.getVar('KSELFTEST_TARGETS', True) else ''}"
+KSELFTEST_ARGS_append = "${@ " FORCE_TARGETS=1" if d.getVar('KSELFTEST_FORCE_TARGETS', True) == '1' else ''}"
+KSELFTEST_ARGS_append .= "${@ " SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\"" if d.getVar('KSELFTEST_SKIP_TARGETS', True) else ''}"
+
do_prepare_build[cleandirs] += "${S}/debian"
do_prepare_build() {
deb_debianize
}
-
-dpkg_runbuild_prepend() {
- if [ -n "${KSELFTEST_TARGETS}" ];then
- export KSELFTEST_ARGS="TARGETS=\"${KSELFTEST_TARGETS}\""
- fi
- if [ ${KSELFTEST_FORCE_TARGETS} -eq 1 ];then
- export KSELFTEST_ARGS="${KSELFTEST_ARGS} FORCE_TARGETS=1"
- fi
- if [ -n "${KSELFTEST_SKIP_TARGETS}" ];then
- export KSELFTEST_ARGS="${KSELFTEST_ARGS} SKIP_TARGETS=\"${KSELFTEST_SKIP_TARGETS}\""
- fi
-}
--
2.20.1

[Uladzimir Bely]

Instead of preinstalling gbp-related packages in sbuild chroot
we will use their host versions in order to keep sbuild environment
as clean as possible.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 4 +++-
meta/classes/dpkg-gbp.bbclass | 24 ++++--------------------
2 files changed, 7 insertions(+), 21 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index c81c6e46..9017dd89 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -79,7 +79,9 @@ apt install \
qemu \
qemu-user-static \
reprepro \
- sudo
+ sudo \
+ git-buildpackage \
+ pristine-tar
```

If your host is >= buster, also install the following package.
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index ddf23ca5..2fc37b2f 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -9,29 +9,13 @@ S = "${WORKDIR}/git"

PATCHTOOL ?= "git"

-GBP_DEPENDS ?= "git-buildpackage pristine-tar"
GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

-do_install_builddeps_append() {
- dpkg_do_mounts
- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
- sudo -E chroot ${BUILDCHROOT_DIR} \
- apt-get install -y -o Debug::pkgProblemResolver=yes \
- --no-install-recommends ${GBP_DEPENDS}
- dpkg_undo_mounts
-}
-
dpkg_runbuild_prepend() {
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- sh -c "cd ${PP}/${PPS} && gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}"
+ sh -c "
+ cd ${WORKDIR}/${PPS}
+ gbp buildpackage --git-builder=/bin/true ${GBP_EXTRA_OPTIONS}
+ "
# NOTE: `buildpackage --git-builder=/bin/true --git-pristine-tar` is used
# for compatibility with gbp version froms debian-stretch. In newer distros
# it's possible to use a subcommand `export-orig --pristine-tar`
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

To make package build process independent of the shell environment we

should remove OPTEE_PLATFORM and OPTEE_EXTRA_BUILDARGS passing through
export call. So we migrate to template-based debian/rules file.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../optee-os/files/debian/{rules => rules.tmpl} | 4 ++--
meta/recipes-bsp/optee-os/optee-os-custom.inc | 9 ++-------
2 files changed, 4 insertions(+), 9 deletions(-)

rename meta/recipes-bsp/optee-os/files/debian/{rules => rules.tmpl} (75%)

diff --git a/meta/recipes-bsp/optee-os/files/debian/rules b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl
similarity index 75%
rename from meta/recipes-bsp/optee-os/files/debian/rules
rename to meta/recipes-bsp/optee-os/files/debian/rules.tmpl
index d2e9900f..9ab80dfe 100755
--- a/meta/recipes-bsp/optee-os/files/debian/rules
+++ b/meta/recipes-bsp/optee-os/files/debian/rules.tmpl

@@ -12,8 +12,8 @@ export CROSS_COMPILE=$(DEB_HOST_GNU_TYPE)-
endif

override_dh_auto_build:

- CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=$(OPTEE_PLATFORM) \
- $(OPTEE_EXTRA_BUILDARGS)
+ CFLAGS= LDFLAGS= $(MAKE) $(PARALLEL_MAKE) PLATFORM=${OPTEE_PLATFORM} \
+ ${OPTEE_EXTRA_BUILDARGS}

%:
dh $@
diff --git a/meta/recipes-bsp/optee-os/optee-os-custom.inc b/meta/recipes-bsp/optee-os/optee-os-custom.inc
index 1bd51969..23612d81 100644
--- a/meta/recipes-bsp/optee-os/optee-os-custom.inc
+++ b/meta/recipes-bsp/optee-os/optee-os-custom.inc
@@ -22,8 +22,8 @@ DEBIAN_BUILD_DEPENDS ?= "python3-pycryptodome:native, python3-pyelftools"

PROVIDES += "optee-os-${OPTEE_NAME}"

-TEMPLATE_FILES = "debian/control.tmpl"
-TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS"
+TEMPLATE_FILES = "debian/control.tmpl debian/rules.tmpl"
+TEMPLATE_VARS += "OPTEE_NAME DEBIAN_BUILD_DEPENDS OPTEE_PLATFORM OPTEE_EXTRA_BUILDARGS"

# split strip platform flavor, if any, from the specified platform string
OPTEE_PLATFORM_BASE = "${@d.getVar('OPTEE_PLATFORM').split('-')[0]}"
@@ -39,8 +39,3 @@ do_prepare_build() {
${S}/debian/optee-os-${OPTEE_NAME}.install
done
}
-
-dpkg_runbuild_prepend() {
- export OPTEE_PLATFORM="${OPTEE_PLATFORM}"
- export OPTEE_EXTRA_BUILDARGS="${OPTEE_EXTRA_BUILDARGS}"
-}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Use previously exported to schroot rootfs to build the package with
sbuild tool. Provide isar-apt as extra repository to be automatically
add to apt sources.

Also added /home/.git-downloads mount while it's used
as git alternates location.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 13 +++++++++++++
meta/classes/dpkg-gbp.bbclass | 2 ++
meta/classes/dpkg.bbclass | 14 ++++++++++++--
3 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 6704385b..18c80411 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -4,6 +4,7 @@
#
# SPDX-License-Identifier: MIT

+inherit sbuild
inherit buildchroot
inherit debianize
inherit terminal
@@ -14,6 +15,8 @@ DEPENDS ?= ""

DEPENDS_append_riscv64 = "${@' crossbuild-essential-riscv64' if d.getVar('ISAR_CROSS_COMPILE', True) == '1' and d.getVar('PN') != 'crossbuild-essential-riscv64' else ''}"

+ISAR_APT_REPO ?= "deb [trusted=yes] file:///home/builder/${PN}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO} ${DEBDISTRONAME} main"
+
python do_adjust_git() {
import subprocess

@@ -196,6 +199,14 @@ dpkg_undo_mounts() {
sudo rmdir ${BUILDROOT}
}

+do_prepare_build_append() {
+ # Make a local copy of isar-apt repo that is not affected by other parallel builds
+ mkdir -p ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+ rm -rf ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/*
+ cp -Rl ${REPO_ISAR_DIR} ${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}
+}
+do_prepare_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
+
# Placeholder for actual dpkg_runbuild() implementation
dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"
@@ -239,6 +250,8 @@ python do_dpkg_build_setscene() {
addtask dpkg_build_setscene
do_dpkg_build_setscene[dirs] += "${S}/.."

+do_dpkg_build[depends] = "${SCHROOT_DEP}"
+
KEEP_INSTALLED_ON_CLEAN ?= "0"

CLEANFUNCS += "deb_clean"
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index 2fc37b2f..7eda9b03 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -11,6 +11,8 @@ PATCHTOOL ?= "git"

GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

+SCHROOT_MOUNTS = "${WORKDIR}:${PP} ${GITDIR}:/home/.git-downloads"
+
dpkg_runbuild_prepend() {
sh -c "

cd ${WORKDIR}/${PPS}

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 7da73341..6cd0f7d1 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -29,10 +29,20 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

addtask devshell after do_install_builddeps

+
# Build package from sources using build script
dpkg_runbuild() {
E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

- /isar/build.sh ${PP}/${PPS} ${PACKAGE_ARCH} ${USE_CCACHE}
+
+ schroot_create_configs
+
+ sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
+ --host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
+ --starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \
+ --no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --debbuildopts="-d --source-option=-I" \
+ --build-dir=${WORKDIR} ${WORKDIR}/${PPS}
+
+ schroot_delete_configs
}
--
2.20.1

[Uladzimir Bely]

Similar to buildchroot, we need a separate rootfs to be used
for schroot.

It's based on bootstrapped rootfs, but includes several
common build-related packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../sbuild-chroot/sbuild-chroot-host.bb | 13 +++++++
.../sbuild-chroot/sbuild-chroot-target.bb | 10 ++++++
.../sbuild-chroot/sbuild-chroot.inc | 36 +++++++++++++++++++
3 files changed, 59 insertions(+)

create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
create mode 100644 meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
new file mode 100644
index 00000000..aa82846b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -0,0 +1,13 @@
+# Root filesystem for packages building

+#
+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for host"
+
+SBUILD_VARIANT = "host"
+
+require sbuild-chroot.inc
+
+ROOTFS_ARCH = "${HOST_ARCH}"
+ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
new file mode 100644
index 00000000..d75d783b
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-target.bb
@@ -0,0 +1,10 @@
+# Root filesystem for packages building

+#
+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+DESCRIPTION = "Isar sbuild/schroot filesystem for target"
+
+SBUILD_VARIANT = "target"
+
+require sbuild-chroot.inc
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
new file mode 100644
index 00000000..177a8a6d
--- /dev/null
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -0,0 +1,36 @@
+# Common part for build chroot filesystem.

+#
+# This software is a part of ISAR.

+# Copyright (C) 2015-2021 ilbers GmbH
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+PV = "1.0"
+
+inherit rootfs
+
+SBUILD_CHROOT_PREINSTALL_COMMON = " \
+ fakeroot \
+ build-essential \
+ debhelper \
+"
+
+SBUILD_CHROOT_PREINSTALL ?= " \
+ ${SBUILD_CHROOT_PREINSTALL_COMMON} \
+"
+
+SBUILD_CHROOT_DIR = "${WORKDIR}/rootfs"
+ROOTFSDIR = "${SBUILD_CHROOT_DIR}"
+ROOTFS_PACKAGES = "${SBUILD_CHROOT_PREINSTALL}"
+
+# We don't need /etc/apt/sources.list.d/isar-apt.list' while it's handled by sbuild
+ROOTFS_CONFIGURE_COMMAND_remove = "rootfs_configure_isar_apt"
+
+DEPLOY_SCHROOT = "${@d.getVar('SCHROOT_' + d.getVar('SBUILD_VARIANT').upper() + '_DIR')}"
+
+do_sbuildchroot_deploy[dirs] = "${DEPLOY_DIR}/schroot-${SBUILD_VARIANT}"
+do_sbuildchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_SCHROOT}"
+}
+addtask sbuildchroot_deploy before do_build after do_rootfs
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

In case there is no ability to pass variables through template
mechanism we need to keep fallback API to migrate from bare export.
Using it with optee-os-stm32mp15x recipe as example.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../optee-os/optee-os-stm32mp15x_3.11.0.bb | 2 +-
meta/classes/sbuild.bbclass | 13 +++++++++++++
2 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
index 0fa2005d..19aa763a 100644
--- a/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
+++ b/meta-isar/recipes-bsp/optee-os/optee-os-stm32mp15x_3.11.0.bb
@@ -21,5 +21,5 @@ OPTEE_BINARIES = "tee-header_v2.stm32 tee-pageable_v2.stm32 tee-pager_v2.stm32"
# Set version manually to PV, the tarball does not contain any hint.
# Alternative: pull from git and add git as build dependency.
dpkg_runbuild_prepend() {
- export TEE_IMPL_VERSION=${PV}
+ sbuild_export TEE_IMPL_VERSION=${PV}
}

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 208b00e1..3471595a 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -95,6 +95,19 @@ schroot_delete_configs() {
EOSUDO

}

+sbuild_export() {
+ SBUILD_CONFIG="${WORKDIR}/sbuild.conf"
+ VAR_LINE="'${1%%=*}' => '${1#*=}',"
+ if [ -s "${SBUILD_CONFIG}" ]; then
+ sed -i -e "\$i\\" -e "${VAR_LINE}" ${SBUILD_CONFIG}
+ else
+ echo "\$build_environment = {" > ${SBUILD_CONFIG}
+ echo "${VAR_LINE}" >> ${SBUILD_CONFIG}
+ echo "};" >> ${SBUILD_CONFIG}
+ fi
+ export SBUILD_CONFIG="${SBUILD_CONFIG}"
+}

+
schroot_install() {
schroot_create_configs
APTS="$1"
--
2.20.1

[Uladzimir Bely]

This also adds mounts for base-apt inside schroot and adds import/export
deb files to/from schroot. So that it becomes possible to run second
`cached` build from local base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/sbuild.bbclass | 146 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 2 +
2 files changed, 148 insertions(+)
create mode 100644 meta/classes/sbuild.bbclass

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
new file mode 100644
index 00000000..208b00e1
--- /dev/null
+++ b/meta/classes/sbuild.bbclass
@@ -0,0 +1,146 @@

+# This software is a part of ISAR.

+# Copyright (C) 2021 ilbers GmbH
+
+SCHROOT_CONF ?= "/etc/schroot"
+
+SCHROOT_MOUNTS ?= ""
+
+python __anonymous() {
+ import pwd
+ d.setVar('SCHROOT_USER', pwd.getpwuid(os.geteuid()).pw_name)
+ d.setVar('SCHROOT_USER_HOME', pwd.getpwuid(os.geteuid()).pw_dir)
+
+ mode = d.getVar('ISAR_CROSS_COMPILE', True)
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
+ d.setVar('SBUILD_HOST_ARCH', distro_arch)
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_TARGET_DIR'))
+ dep = "sbuild-chroot-target:do_build"
+ else:
+ d.setVar('SBUILD_HOST_ARCH', d.getVar('HOST_ARCH'))
+ d.setVar('SCHROOT_DIR', d.getVar('SCHROOT_HOST_DIR'))
+ dep = "sbuild-chroot-host:do_build"
+ d.setVar('SCHROOT_DEP', dep)
+}
+
+SBUILD_CHROOT ?= "${DEBDISTRONAME}-${SCHROOT_USER}-${@os.getpid()}"
+SBUILD_CHROOT_RW ?= "${SBUILD_CHROOT}-rw"
+
+SBUILD_CONF_DIR ?= "${SCHROOT_CONF}/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}"
+SCHROOT_CONF_FILE_RW ?= "${SCHROOT_CONF}/chroot.d/${SBUILD_CHROOT}-rw"
+
+schroot_create_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+
+ cat << EOF > "${SCHROOT_CONF_FILE}"
+[${SBUILD_CHROOT}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+source-root-users=${SCHROOT_USER}
+source-root-groups=root,sbuild
+union-type=overlay
+preserve-environment=true
+EOF
+
+ cat << EOF > "${SCHROOT_CONF_FILE_RW}"
+[${SBUILD_CHROOT_RW}]
+type=directory
+directory=${SCHROOT_DIR}
+profile=${SBUILD_CHROOT}
+users=${SCHROOT_USER}
+groups=root,sbuild
+root-users=${SCHROOT_USER}
+root-groups=root,sbuild
+preserve-environment=true
+EOF
+
+ mkdir -p "${SCHROOT_DIR}/etc/apt/preferences.d"
+ cat << EOF > "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+Package: *
+Pin: release n=${DEBDISTRONAME}
+Pin-Priority: 1000
+EOF
+
+ # Prepare mount points
+ cp -rf "${SCHROOT_CONF}/sbuild" "${SBUILD_CONF_DIR}"
+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+
+ fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"
+ grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}
+
+ if [ -d ${DL_DIR} ]; then
+ fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"
+ grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}
+ fi
+EOSUDO
+}
+
+schroot_delete_configs() {
+ sudo -s <<'EOSUDO'
+ set -e
+ if [ -d "${SBUILD_CONF_DIR}" ]; then
+ rm -rf "${SBUILD_CONF_DIR}"
+ fi
+ rm -f "${SCHROOT_DIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${SCHROOT_CONF_FILE}"
+ rm -f "${SCHROOT_CONF_FILE_RW}"
+EOSUDO
+}
+
+schroot_install() {
+ schroot_create_configs
+ APTS="$1"
+
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+
+ deb_dl_dir_import ${SCHROOT_DIR} ${distro}
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends --download-only ${APTS}
+ deb_dl_dir_export ${SCHROOT_DIR} ${distro}
+
+ schroot -d / -c ${SBUILD_CHROOT_RW} -u root -- \
+ apt install -y -o Debug::pkgProblemResolver=yes \
+ --no-install-recommends ${APTS}
+ schroot_delete_configs
+}
+
+insert_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ grep -qxF "${FSTAB_LINE}" ${SBUILD_CONF_DIR}/fstab || \
+ echo "${FSTAB_LINE}" >> ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+remove_mounts() {
+ sudo -s <<'EOSUDO'
+ set -e
+ for mp in ${SCHROOT_MOUNTS}; do
+ FSTAB_LINE="${mp%%:*} ${mp#*:} none rw,bind 0 0"
+ sed -i "\|${FSTAB_LINE}|d" ${SBUILD_CONF_DIR}/fstab
+ done
+EOSUDO
+}
+
+schroot_run() {
+ schroot_create_configs
+ insert_mounts
+ schroot $@
+ remove_mounts
+ schroot_delete_configs
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index cd1c4a64..c1458f05 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -58,6 +58,8 @@ SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
SSTATE_MANIFESTS = "${TMPDIR}/sstate-control/${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_HOST_DIR = "${DEPLOY_DIR}/schroot-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+SCHROOT_TARGET_DIR = "${DEPLOY_DIR}/schroot-target/${DISTRO}-${DISTRO_ARCH}"
SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"
KERNEL_FILE ?= "vmlinuz"
--
2.20.1

[Uladzimir Bely]

While builddeps are now handled interlnally by sbuild, we don't need
a task to preinstall them in buildchroot.

But for the second local (base-apt) build we need a way to keep
all dependencies in form of .deb files in DL_DIR. This is done
by executing additional commands in sbuild that copy them to/from
an externally mounted folder.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 43 ++++++++++++++++++---------------------
1 file changed, 20 insertions(+), 23 deletions(-)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 6cd0f7d1..fe5a6011 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -5,30 +5,7 @@ inherit dpkg-base

PACKAGE_ARCH ?= "${DISTRO_ARCH}"

-# Install build dependencies for package
-do_install_builddeps() {
- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- distro="${DISTRO}"
- if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
- distro="${HOST_DISTRO}"
- fi
- deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH} --download-only

- deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"

- sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
- ${PP}/${PPS} ${PACKAGE_ARCH}
- dpkg_undo_mounts
-}
-
-addtask install_builddeps after do_prepare_build before do_dpkg_build
do_install_builddeps[depends] += "isar-apt:do_cache_config"
-# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-
-addtask devshell after do_install_builddeps
-

# Build package from sources using build script
dpkg_runbuild() {

@@ -37,12 +14,32 @@ dpkg_runbuild() {

schroot_create_configs

+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+

+ deb_dl_dir_import "${WORKDIR}/rootfs" "${distro}"
+
+ deb_dir="/var/cache/apt/archives/"
+ ext_deb_dir="/home/builder/${PN}/rootfs/${deb_dir}"
+
+ ( flock 9
+ grep -qxF '$apt_keep_downloaded_packages = 1;' ${SCHROOT_USER_HOME}/.sbuildrc ||
+ echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc
+ ) 9>"${TMPDIR}/sbuildrc.lock"

+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \

--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \

--starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \

--no-run-lintian --no-run-piuparts --no-run-autopkgtest \
+ --chroot-setup-commands="cp -n --no-preserve=owner ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
+ --finished-build-commands="cp -n --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
--debbuildopts="-d --source-option=-I" \

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

+ deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"
+
schroot_delete_configs
}
--
2.20.1

[Uladzimir Bely]

From: Anton Mikanovich <ami...@ilbers.de>

Move prepare code inside sbuild hook to perform all the operations
inside schroot.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/recipes-kernel/linux/linux-custom.inc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 57740860..92c5e4a7 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -161,9 +161,9 @@ dpkg_configure_kernel() {
(cd ${WORKDIR} && cp ${src_frags} ${S}/debian/fragments/)
fi

- sudo -E chroot --userspec=$(id -u):$(id -g) ${BUILDCHROOT_DIR} sh -c " \
+ SBUILD_PREBUILD=" \
export ARCH=${KERNEL_ARCH} && \
- cd ${PP}/${PPS} && \
+ cd %p && \
make O=${KERNEL_BUILD_DIR} ${config_target} && \
./scripts/kconfig/merge_config.sh \
-O ${KERNEL_BUILD_DIR}/ \
--
2.20.1

[Uladzimir Bely]

While packages are now build in schroot with sbuild, devshell
task should make terminal open inside sbuild environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 22 ++++++++++++++-----
meta/classes/sbuild.bbclass | 3 +++
.../sbuild-chroot/sbuild-chroot.inc | 2 ++
3 files changed, 22 insertions(+), 5 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 18c80411..34f6e7ab 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -288,16 +288,28 @@ do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[dirs] = "${S}"

python do_devshell() {
- bb.build.exec_func('dpkg_do_mounts', d)
+ bb.build.exec_func('schroot_create_configs', d)

isar_export_proxies(d)

- buildchroot = d.getVar('BUILDCHROOT_DIR')
+ schroot = d.getVar('SBUILD_CHROOT')
+ isar_apt = d.getVar('ISAR_APT_REPO')
pp_pps = os.path.join(d.getVar('PP'), d.getVar('PPS'))
- termcmd = "sudo -E chroot {0} sh -c 'cd {1}; $SHELL -i'"
- oe_terminal(termcmd.format(buildchroot, pp_pps), "Isar devshell", d)

- bb.build.exec_func('dpkg_undo_mounts', d)
+ install_deps = ":" if d.getVar('BB_CURRENTTASK') == "devshell_nodeps" else "mk-build-deps -i -t \
+ \"apt-get -y -q -o Debug::pkgProblemResolver=yes --no-install-recommends --allow-downgrades\" \
+ debian/control"
+
+ termcmd = "schroot -d / -c {0} -u root -- sh -c ' \
+ cd {1}; \
+ echo {2} > /etc/apt/sources.list.d/isar_apt.list; \
+ apt-get -y -q update; \
+ {3}; \
+ $SHELL -i \
+ '"
+ oe_terminal(termcmd.format(schroot, pp_pps, isar_apt, install_deps), "Isar devshell", d)
+
+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask devshell after do_prepare_build
diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 3471595a..0df8cb90 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -76,6 +76,9 @@ EOF

fstab_baseapt="${REPO_BASE_DIR} /base-apt none rw,bind 0 0"

grep -qxF "${fstab_baseapt}" ${sbuild_fstab} || echo "${fstab_baseapt}" >> ${sbuild_fstab}

+ fstab_pkgdir="${WORKDIR} /home/builder/${PN} none rw,bind 0 0"
+ grep -qxF "${fstab_pkgdir}" ${sbuild_fstab} || echo "${fstab_pkgdir}" >> ${sbuild_fstab}

+
if [ -d ${DL_DIR} ]; then

fstab_downloads="${DL_DIR} /downloads none rw,bind 0 0"

grep -qxF "${fstab_downloads}" ${sbuild_fstab} || echo "${fstab_downloads}" >> ${sbuild_fstab}

diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 177a8a6d..7b41d11d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,8 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
build-essential \
debhelper \
+ devscripts \
+ equivs \
"

SBUILD_CHROOT_PREINSTALL ?= " \
--
2.20.1

[Uladzimir Bely]

This removes remaining buildchroot-related tasks and variables
from dpkg-base class while there are not used anymore.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 43 +++-------------------------------
meta/classes/dpkg.bbclass | 4 ----
2 files changed, 3 insertions(+), 44 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index f10f424f..e7acace6 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -5,7 +5,6 @@
# SPDX-License-Identifier: MIT

inherit sbuild
-inherit buildchroot
inherit debianize
inherit terminal
inherit repository
@@ -165,29 +164,7 @@ addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"

-BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
-
-dpkg_do_mounts() {
- mkdir -p ${BUILDROOT}
- sudo mount --bind ${WORKDIR} ${BUILDROOT}
-
- buildchroot_do_mounts
-}
-
-dpkg_undo_mounts() {
- i=0
- while ! sudo umount ${BUILDROOT}; do
- sleep 0.1
- if [ `expr $i % 100` -eq 0 ] ; then
- bbwarn "${BUILDROOT}: Couldn't unmount ($i), retrying..."
- fi
- if [ $i -ge 10000 ]; then
- bbfatal "${BUILDROOT}: Couldn't unmount after timeout"
- fi
- i=`expr $i + 1`
- done
- sudo rmdir ${BUILDROOT}
-}
+do_prepare_build[depends] = "${SCHROOT_DEP}"

do_prepare_build_append() {

# Make a local copy of isar-apt repo that is not affected by other parallel builds

@@ -203,14 +180,11 @@ dpkg_runbuild() {
}

python do_dpkg_build() {
- lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
- shared=True)

- bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func('schroot_create_configs', d)

try:
bb.build.exec_func("dpkg_runbuild", d)
finally:

- bb.build.exec_func("dpkg_undo_mounts", d)

- bb.utils.unlockfile(lock)

+ bb.build.exec_func('schroot_delete_configs', d)
}

addtask dpkg_build
@@ -242,8 +216,6 @@ do_dpkg_build_setscene[dirs] += "${S}/.."

do_dpkg_build[depends] = "${SCHROOT_DEP}"

-KEEP_INSTALLED_ON_CLEAN ?= "0"
-
CLEANFUNCS += "deb_clean"

deb_clean() {
@@ -252,15 +224,6 @@ deb_clean() {
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
- if [ "${KEEP_INSTALLED_ON_CLEAN}" = "1" ]; then
- continue;
- fi
- package=$(basename "${d}")
- package_remove="/usr/bin/apt-get remove -y ${package%%_*}"
- sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
- if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
- sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
- fi
done
fi
}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index fe5a6011..85bb1300 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -12,8 +12,6 @@ dpkg_runbuild() {

E="${@ isar_export_proxies(d)}"
export PARALLEL_MAKE="${PARALLEL_MAKE}"

- schroot_create_configs
-
distro="${DISTRO}"

if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then

distro="${HOST_DISTRO}"
@@ -40,6 +38,4 @@ dpkg_runbuild() {

--build-dir=${WORKDIR} ${WORKDIR}/${PPS}

deb_dl_dir_export "${WORKDIR}/rootfs" "${distro}"

-
- schroot_delete_configs
}
--
2.20.1

[Uladzimir Bely]

This moves downloading and unpacking deb-src to schroot environment.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 26 ++++++++------------------
1 file changed, 8 insertions(+), 18 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 34f6e7ab..f10f424f 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -99,34 +99,25 @@ python() {
}

do_apt_fetch() {

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"

- sudo -E chroot ${BUILDCHROOT_DIR} /usr/bin/apt-get update \
- -o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
- -o Dir::Etc::SourceParts="-" \
- -o APT::Get::List-Cleanup="0"
-
+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_fetch after do_unpack before do_apt_unpack
do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

-# Add dependency from the correct buildchroot: host or target
-do_apt_fetch[depends] = "${BUILDCHROOT_DEP}"
+# Add dependency from the correct schroot: host or target
+do_apt_fetch[depends] = "${SCHROOT_DEP}"

do_apt_unpack() {
rm -rf ${S}

- dpkg_do_mounts
- E="${@ isar_export_proxies(d)}"
-

+ schroot_create_configs
for uri in "${SRC_APT}"; do

- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \

+ schroot -d / -c ${SBUILD_CHROOT} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -135,8 +126,7 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${DISTRO}" "${uri}"
done
-
- dpkg_undo_mounts
+ schroot_delete_configs
}

addtask apt_unpack after do_apt_fetch before do_patch
--
2.20.1

[Uladzimir Bely]

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

doc/user_manual.md | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 9017dd89..088cbe18 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -84,6 +84,24 @@ apt install \
pristine-tar
```

+Additional setup is required since `sbuild` is now used for package build.
+Install the following packages:
+```
+apt install \
+ sbuild \
+ schroot
+```
+Also, user who runs isar should be added to `sbuild` group.
+
+**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
+`$apt_keep_downloaded_packages` option which is required in Isar for
+populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
+upgraded to >=0.81.2 version from Debian Bullseye.
+
+```
+sudo gpasswd -a <username> sbuild
+```
+

If your host is >= buster, also install the following package.

```
apt install python3-distutils
--
2.20.1

[Uladzimir Bely]

This adds ccache support for custom packages in the same manner
as it was done previously for buildchroot-based build, by using
USE_CCACHE variable in local.conf or per-recipe.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg.bbclass | 4 ++++
meta/classes/sbuild.bbclass | 22 +++++++++++++++++++
.../sbuild-chroot/sbuild-chroot.inc | 1 +
3 files changed, 27 insertions(+)

diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 85bb1300..e68605ef 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -27,6 +27,10 @@ dpkg_runbuild() {

echo '$apt_keep_downloaded_packages = 1;' >> ${SCHROOT_USER_HOME}/.sbuildrc

) 9>"${TMPDIR}/sbuildrc.lock"

+ if [ ${USE_CCACHE} -eq 1 ]; then
+ schroot_configure_ccache
+ fi

+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${SBUILD_HOST_ARCH} \
--starting-build-commands="runuser -u ${SCHROOT_USER} -- sh -c \"${SBUILD_PREBUILD:-:}\"" \

diff --git a/meta/classes/sbuild.bbclass b/meta/classes/sbuild.bbclass
index 0df8cb90..54ae8550 100644
--- a/meta/classes/sbuild.bbclass
+++ b/meta/classes/sbuild.bbclass
@@ -160,3 +160,25 @@ schroot_run() {
remove_mounts
schroot_delete_configs
}
+
+schroot_configure_ccache() {

+ sudo -s <<'EOSUDO'
+ set -e
+

+ sbuild_fstab="${SBUILD_CONF_DIR}/fstab"
+

+ install --group=sbuild --mode=2775 -d ${CCACHE_DIR}
+ fstab_ccachedir="${CCACHE_DIR} /ccache none rw,bind 0 0"
+ grep -qxF "${fstab_ccachedir}" ${sbuild_fstab} || echo "${fstab_ccachedir}" >> ${sbuild_fstab}
+
+ cat << END > ${CCACHE_DIR}/sbuild-setup
+#!/bin/sh
+export CCACHE_DIR=/ccache
+export PATH="/usr/lib/ccache:\$PATH"
+exec "\$@"
+END
+ chmod a+rx ${CCACHE_DIR}/sbuild-setup
+
+ echo "command-prefix=/ccache/sbuild-setup" >> "${SCHROOT_CONF_FILE}"
+EOSUDO
+}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index 7b41d11d..4688da1d 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -14,6 +14,7 @@ SBUILD_CHROOT_PREINSTALL_COMMON = " \
fakeroot \
build-essential \
debhelper \
+ ccache \
devscripts \
equivs \
"
--
2.20.1

[Uladzimir Bely]

Just a question: Does 'xenomai-images' and 'meta-iot2050' have their own
maillists where I could send the changes to work with isar-based sbuild? I
just found a maillist for 'kas', but not for these projects.

[Jan Kiszka]

For xenomai-images, we handle changes via xen...@xenomai.org (please
additional tag related patches with "[xenomai-images]").

meta-iot2050 takes pull requests on github.

For both projects, changes that will work with current Isar can already
be submitted. Changes that depend on your pending series could be
proposed along with an Isar revision bump once 'next' picked up your
queue. Or send them earlier with a clear remark that those are for
reference and future use. But it is probably better to wait for the
'next' merge.

Thanks,