[PATCH v5 0/7] Migrate to mmdebstrap

[Anton Mikanovich]

Switch Isar from debootstrap to mmdebstrap.

It makes isar-bootstrap target ~10..30% faster on both cross and native
rootfs types. Some measurements on qemuarm64-bookworm (required debs are
predownloaded and used during deb-dl-import):

- 20s vs 28s for "isar-bootstrap-host"
- 1m32s vs 1m48s for "isar-bootstrap-target

Also mmdebstrap support using SOURCE_DATE_EPOCH for images reproducibly
out of the box. But the main advantage is an ability to remove sudo for
debootstrapping later on (this needs "uidmap" package installed).

This change breaks Debian Buster support as host distro.

Changes since v4:
- Rebase on next.
- Migrate from apt-key to gpg.
- Fix Ubuntu and RPi targets.
- Cleanup debootstrap mentions.
- Fix commit messages.

Changes since v3:
- Rebase on latest next.
- Remove temporary patch 7 since kas:4.0 already includes mmdebstrap.

Changes since v2:
- Rebase on next.
- Fix commit messages.
- Update host requirements.

Changes since v1:
- Move mmdebstrap tmpdir to workdir.
- Fix kas building.
- Improve mounts cleanup.

Anton Mikanovich (6):
meta: Preinstall apt for Ubuntu and RPi targets
isar-bootstrap: Move preparations to hooks
isar-bootstrap: Use tar output instead of directory
user_manual.md: Replace debootstrap with mmdebstrap
user_manual.md: Update minimal host requirements
isar-bootstrap: Remove unused code

Uladzimir Bely (1):
meta: Switch to mmdebstrap

doc/user_manual.md | 41 +--
meta-isar/conf/distro/ubuntu-common.inc | 4 -
meta/classes/rootfs.bbclass | 11 +-
meta/conf/bitbake.conf | 2 +-
.../isar-bootstrap/isar-bootstrap.inc | 284 ++++++++----------
5 files changed, 153 insertions(+), 189 deletions(-)

--
2.34.1

[Anton Mikanovich]

From: Uladzimir Bely <ub...@ilbers.de>

Instead of debootstrap, use mmdebstrap alternative.

Internally, it uses apt and allows to debootstrap the distro from
multiple repositories.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---
meta-isar/conf/distro/ubuntu-common.inc | 4 --
.../isar-bootstrap/isar-bootstrap.inc | 64 ++++++++++---------
2 files changed, 33 insertions(+), 35 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 6a990a89..afe96b73 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -18,10 +18,6 @@ HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}"

# kernel package name is linux-image-generic in Ubuntu
KERNEL_NAME = "generic"
-
-# that is what debootstrap_1.0.118ubuntu1 does anyways
-DISTRO_DEBOOTSTRAP_SCRIPT = "/usr/share/debootstrap/scripts/gutsy"
-
DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"

DISTRO_KERNELS ?= " \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3477c2fb..9f1142ba 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
@@ -23,12 +22,14 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir"
FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir
@@ -39,7 +40,7 @@ python () {
topdir = d.getVar("TOPDIR")

# The cached repo key can be both for bootstrapping and apt package
- # installation afterwards. However, debootstrap will include the key into
+ # installation afterwards. However, bootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.

if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
@@ -267,7 +268,7 @@ do_bootstrap[vardeps] += " \
ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

@@ -279,31 +280,35 @@ do_bootstrap() {
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
- debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
+ bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
- debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
- fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
- debootstrap_args="$debootstrap_args --no-check-gpg"
+ bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
fi
E="${@ isar_export_proxies(d)}"
- export BOOTSTRAP_FOR_HOST debootstrap_args E
+ export BOOTSTRAP_FOR_HOST

- sudo rm -rf --one-file-system "${ROOTFSDIR}"
deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ sudo rm -rf --one-file-system "${ROOTFSDIR}"
+ mkdir -p "${ROOTFSDIR}"
+
+ if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then
+ export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+ fi
+
+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+
+ sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
+ $arch_param \
+ --mode=unshare \
+ ${@get_distro_components_argument(d)} \
+ "${@get_distro_suite(d)}" \
+ "${WORKDIR}/rootfs.tar.zst" \
+ "${@get_distro_source(d)}"

sudo -E -s <<'EOSUDO'
set -e
- if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
- fi
- ${DEBOOTSTRAP} $debootstrap_args \
- $arch_param \
- ${@get_distro_components_argument(d)} \
- "${@get_distro_suite(d)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
+
+ tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
@@ -389,7 +394,7 @@ do_bootstrap() {
umount -l "${ROOTFSDIR}/sys"
umount -l "${ROOTFSDIR}/base-apt" || true

- # Finalize debootstrap by setting the link in deploy
+ # Finalize bootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
@@ -406,19 +411,16 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically
- lopts="--one-file-system --exclude=var/cache/apt/archives"
- sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")
- sudo chown $(id -u):$(id -g) bootstrap.tar
+ sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR
- # - after building the bootstrap, the tar won't be there, but we also don't need to unpack
- # - after restoring from cache, there will be a tar which we unpack and then delete
- if [ -f bootstrap.tar ]; then
- sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar
- sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
- rm bootstrap.tar
+ if [ -f bootstrap.tar.zst ]; then
+ mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
+ sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
fi
}

--
2.34.1

[Anton Mikanovich]

Move all bootstrap rootfs prepare logic into mmdebstrap hooks.
Also migrate from apt-key to gpg.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 198 +++++++++---------
1 file changed, 103 insertions(+), 95 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 22fb398f..8fa22e52 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -213,24 +213,21 @@ def get_distro_components_argument(d):
else:
return ""

-APT_KEYS_DIR = "${WORKDIR}/aptkeys"
DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"

-do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
-do_generate_keyrings[dirs] = "${DL_DIR}"
+do_generate_keyrings[cleandirs] = "${WORKDIR}/trusted.gpg.d"
+do_generate_keyrings[dirs] = "${DEBDIR}"
do_generate_keyrings[vardeps] += "DISTRO_BOOTSTRAP_KEYS THIRD_PARTY_APT_KEYS"
do_generate_keyrings[network] = "${TASK_USE_SUDO}"
do_generate_keyrings() {
if [ -n "${@d.getVar("THIRD_PARTY_APT_KEYFILES") or ""}" ]; then
- chmod 777 "${APT_KEYS_DIR}"
for keyfile in ${@d.getVar("THIRD_PARTY_APT_KEYFILES")}; do
- cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
+ gpg --no-default-keyring --keyring "${DISTRO_BOOTSTRAP_KEYRING}" --import $keyfile
done
fi
if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES") or ""}" ]; then
for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES")}; do
- sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
- cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
+ gpg --no-default-keyring --keyring "${DISTRO_BOOTSTRAP_KEYRING}" --import $keyfile
done
fi
}
@@ -271,12 +268,14 @@ do_bootstrap[vardeps] += " \
ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"

do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

inherit compat

+DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
@@ -286,124 +285,133 @@ do_bootstrap() {

bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then

bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
+ cp "${DISTRO_BOOTSTRAP_KEYRING}" "${WORKDIR}/trusted.gpg.d/"

fi
E="${@ isar_export_proxies(d)}"
- export BOOTSTRAP_FOR_HOST

- deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${ROOTFSDIR}"

mkdir -p "${ROOTFSDIR}"

+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
+ bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"
+ line="copy://$base_apt_tmp/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+ if [ -z "${BASE_REPO_KEY}" ]; then
+ line="[trusted=yes] ${line}"
+ fi
+ echo "deb ${line}" > "${WORKDIR}/sources.list.d/base-apt.list"
+ line="copy://$base_apt_tmp/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
+ if [ -z "${BASE_REPO_KEY}" ]; then
+ line="[trusted=yes] ${line}"
+ fi
+ echo "deb-src ${line}" >> "${WORKDIR}/sources.list.d/base-apt.list"
+
+ # no need to sync /var/cache/apt/archives if base-apt used
+ syncin='echo skip sync-in'
+ syncout='echo skip sync-out'
+ extra_setup="mount --bind '${REPO_BASE_DIR}' $base_apt_tmp"
+ extra_extract="$syncout"
+ # save mmdebstrap tempdir for cleanup
+ extra_essential="mkdir -p \$1/$base_apt_tmp && \
+ echo \$1 > ${WORKDIR}/mmtmpdir && \
+ mount --bind '${REPO_BASE_DIR}' \$1/$base_apt_tmp"
+ # replace base-apt mount in tmp with /base-apt mount
+ extra_customize="sed -i \"s|copy://$base_apt_tmp|file:///base-apt|g\" \
+ \$1/etc/apt/sources.list.d/*.list && \
+ mkdir -p \$1/base-apt && \
+ mount --bind '${REPO_BASE_DIR}' \$1/base-apt && \
+ chroot \$1 apt-get update -y \
+ -o APT::Update::Error-Mode=any && \
+ umount \$1/base-apt && \
+ umount \$1/$base_apt_tmp && rm ${WORKDIR}/mmtmpdir && \
+ umount $base_apt_tmp && rm -rf --one-file-system $base_apt_tmp"
+ else
+ deb_dl_dir_import "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+
+ bootstrap_list="${WORKDIR}/sources.list.d/bootstrap.list"
+ install -v -m644 "${APTSRCS}" \
+ "${WORKDIR}/sources.list.d/bootstrap.list"
+
+ syncin='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"*.deb \
+ "$1/var/cache/apt/archives/" || true'
+ syncout='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "$1/var/cache/apt/archives/"*.deb \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"'
+ extra_setup="$syncin"
+ extra_extract="$syncout"
+ # prefetch apt debs because mmdebstrap will clean them on next stage
+ extra_essential='apt-get install apt -y -d \
+ -o Dir::State="$1/var/lib/apt" \
+ -o Dir::Etc="$1/etc/apt" \
+ -o Dir::Cache="$1/var/cache/apt" \
+ -o Apt::Architecture="${BOOTSTRAP_DISTRO_ARCH}"'
+ extra_essential="$extra_essential && $syncout"
+ extra_customize="$syncout"
+ fi

+
if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then

export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
fi

arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"

+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
+ fi
+
+ # Cleanup mounts if fails
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \
+ && rm "${WORKDIR}/mmtmpdir"; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/$base_apt_tmp \
+ && sudo umount $tmpdir/$base_apt_tmp; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/base-apt \
+ && sudo umount $tmpdir/base-apt; \
+ [ -d "$tmpdir" ] && sudo rm -rf --one-file-system $tmpdir; \
+ [ -n "$base_apt_tmp" ] && mountpoint -q $base_apt_tmp \
+ && sudo umount $base_apt_tmp \
+ && rm -rf --one-file-system $base_apt_tmp' EXIT

sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \

$arch_param \
--mode=unshare \
+ --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \
+ --setup-hook="$extra_setup" \
+ --setup-hook='upload "${APTPREFS}" /etc/apt/preferences.d/bootstrap' \
+ --setup-hook='upload "${APTSRCS_INIT}" /etc/apt/sources-list' \
+ --setup-hook='upload "${WORKDIR}/locale" /etc/locale' \
+ --setup-hook='mkdir -p "$1/etc/apt/trusted.gpg.d"' \
+ --setup-hook='sync-in "${WORKDIR}/trusted.gpg.d" /etc/apt/trusted.gpg.d' \
+ --setup-hook='install -v -m755 "${WORKDIR}/chroot-setup.sh" "$1/chroot-setup.sh"' \
+ --extract-hook="$extra_extract" \
+ --essential-hook="$extra_essential" \
+ --customize-hook="$extra_customize" \
+ --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' \
+ --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
+ --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
+ --skip=cleanup/apt \
+ --skip=download/empty \
${@get_distro_components_argument(d)} \
"${@get_distro_suite(d)}" \
"${WORKDIR}/rootfs.tar.zst" \
- "${@get_distro_source(d)}"
+ "$bootstrap_list"

sudo -E -s <<'EOSUDO'
set -e

tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

- # Install apt config
- mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
- install -v -m644 "${APTPREFS}" \
- "${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
- mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
- line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
- if [ -z "${BASE_REPO_KEY}" ]; then
- line="[trusted=yes] ${line}"
- fi
- echo "deb ${line}" > "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
- line="file:///base-apt/${BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
- if [ -z "${BASE_REPO_KEY}" ]; then
- line="[trusted=yes] ${line}"
- fi
- echo "deb-src ${line}" >> "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
- mkdir -p ${ROOTFSDIR}/base-apt
- mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt
- else
- install -v -m644 "${APTSRCS}" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
- fi
- install -v -m644 "${APTSRCS_INIT}" "${ROOTFSDIR}/etc/apt/sources-list"
- rm -f "${ROOTFSDIR}/etc/apt/sources.list"
- rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
- find ${APT_KEYS_DIR}/ -type f | while read keyfile
- do
- MY_GPGHOME="$(chroot "${ROOTFSDIR}" mktemp -d /tmp/gpghomeXXXXXXXXXX)"
- echo "Created temporary directory ${MY_GPGHOME} for gpg-agent"
- export GNUPGHOME="${MY_GPGHOME}"
- APT_KEY_APPEND="--homedir ${MY_GPGHOME}"
-
- kfn="$(basename $keyfile)"
- cp $keyfile "${ROOTFSDIR}/tmp/$kfn"
- chroot "${ROOTFSDIR}" /usr/bin/gpg-agent --daemon -- /usr/bin/apt-key \
- --keyring ${THIRD_PARTY_APT_KEYRING} ${APT_KEY_APPEND} add "/tmp/$kfn"
- rm "${ROOTFSDIR}/tmp/$kfn"
-
- echo "Removing ${MY_GPGHOME}"
- rm -rf "${ROOTFSDIR}${MY_GPGHOME}"
- done
-
- # Set locale
- install -v -m644 "${WORKDIR}/locale" "${ROOTFSDIR}/etc/locale"
-
- sed -i '/en_US.UTF-8 UTF-8/s/^#//g' "${ROOTFSDIR}/etc/locale.gen"
- chroot "${ROOTFSDIR}" /usr/sbin/locale-gen
-
# setup chroot
install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
"${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"

- # update APT
- mount -o bind,private /dev ${ROOTFSDIR}/dev
- mount --bind /dev/pts ${ROOTFSDIR}/dev/pts
- mount -t tmpfs none "${ROOTFSDIR}/dev/shm"
- mount -t proc none ${ROOTFSDIR}/proc
- mount --rbind /sys ${ROOTFSDIR}/sys
- mount --make-rslave ${ROOTFSDIR}/sys
-
- export DEBIAN_FRONTEND=noninteractive
-
- if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
- chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
- fi
-
- if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
- chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
- fi
-
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
- -o APT::Update::Error-Mode=any
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
-
- umount -l "${ROOTFSDIR}/dev/shm"
- umount -l "${ROOTFSDIR}/dev/pts"
- umount -l "${ROOTFSDIR}/dev"
- umount -l "${ROOTFSDIR}/proc"
- umount -l "${ROOTFSDIR}/sys"
- umount -l "${ROOTFSDIR}/base-apt" || true
-

# Finalize bootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO

- deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
-
- # Cleanup apt cache
- sudo -Es chroot "${ROOTFSDIR}" /usr/bin/apt-get -y clean
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then
+ deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"
+ fi
}

addtask bootstrap before do_build after do_generate_keyrings
--
2.34.1

[Anton Mikanovich]

Apt package is not in required packages in Ubuntu and Raspberry repos.
To have it installed during bootstrap introduce BASE_DISTRO override
and append apt to DISTRO_BOOTSTRAP_BASE_PACKAGES for this distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/conf/bitbake.conf | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 80dc01c7..22fdc0e7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -70,7 +70,7 @@ KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"

MACHINEOVERRIDES ?= "${MACHINE}"
-DISTROOVERRIDES ?= "${DISTRO}"
+DISTROOVERRIDES ?= "${DISTRO}:${BASE_DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 9f1142ba..22fb398f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -25,6 +25,9 @@ DEPLOY_ISAR_BOOTSTRAP ?= ""

DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"

+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"

DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"

--
2.34.1

[Anton Mikanovich]

Do not unpack tar with rootfs content inside bootstrap recipe.
This change can allow to remove sudo usage during the bootstrap later.
Using tar output requires moving chroot prepare step to the later
unpack step.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/classes/rootfs.bbclass | 11 +++++----
.../isar-bootstrap/isar-bootstrap.inc | 23 ++++---------------
2 files changed, 12 insertions(+), 22 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 65574a6c..98e17c3c 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -73,12 +73,15 @@ rootfs_do_qemu() {
fi
}

-BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}"
-BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}"
+BOOTSTRAP_SRC = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-host_${DISTRO}-${DISTRO_ARCH}.tar.zst"
+BOOTSTRAP_SRC:${ROOTFS_ARCH} = "${DEPLOY_DIR_BOOTSTRAP}/${ROOTFS_DISTRO}-${ROOTFS_ARCH}.tar.zst"

rootfs_prepare[weight] = "25"
rootfs_prepare(){
- sudo cp -Trpfx --reflink=auto '${BOOTSTRAP_SRC}/' '${ROOTFSDIR}'
+ sudo tar -xf "${BOOTSTRAP_SRC}" -C "${ROOTFSDIR}" --exclude="./dev/console"
+
+ # setup chroot
+ sudo "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
}

ROOTFS_CONFIGURE_COMMAND += "rootfs_configure_isar_apt"
@@ -224,7 +227,7 @@ cache_deb_src() {
# Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
# relies on that through out the build. Copy that state information instead of apt-get update
# which generates a new state from upstream.
- sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
+ sudo tar -xf "${BOOTSTRAP_SRC}" ./var/lib/apt/lists --one-top-level="${ROOTFSDIR}"

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 8fa22e52..dee6cc31 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-ROOTFSDIR = "${WORKDIR}/rootfs"

APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"

APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

@@ -289,9 +288,6 @@ do_bootstrap() {

fi
E="${@ isar_export_proxies(d)}"

- sudo rm -rf --one-file-system "${ROOTFSDIR}"

- mkdir -p "${ROOTFSDIR}"

-
if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"
@@ -396,18 +392,9 @@ do_bootstrap() {
"${WORKDIR}/rootfs.tar.zst" \
"$bootstrap_list"

- sudo -E -s <<'EOSUDO'
- set -e
-
- tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
-
- # setup chroot
- install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
- "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+ # Finalize bootstrap by setting the link in deploy
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"

- # Finalize bootstrap by setting the link in deploy

- ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-EOSUDO

if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then

deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"

@@ -422,15 +409,15 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically

- sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst

sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR

if [ -f bootstrap.tar.zst ]; then
- mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
- sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst"
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \

[Anton Mikanovich]

Update documentation in user manual.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index ecb987f4..6cbff8cf 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -67,7 +67,9 @@ Install the following packages:
apt install \
binfmt-support \
bzip2 \
- debootstrap \
+ mmdebstrap \
+ arch-test \
+ apt-utils \
dpkg-dev \
gettext-base \
git \
@@ -127,7 +129,7 @@ apt install qemu

### Setup Sudo

-Isar requires `sudo` rights without password to work with `chroot` and `debootstrap`. To add them, use the following steps:
+Isar requires `sudo` rights without password to work with `chroot`. To add them, use the following steps:
```
# visudo
```
--
2.34.1

[Anton Mikanovich]

Using mmdebstrap requires at least Debian Bullseye on host.
Update user manual with the new minimal host distro.
Also remove all Buster-related notes.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 35 ++++++++++++-----------------------
1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 6cbff8cf..512721c9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -48,7 +48,7 @@ For demonstration purposes, Isar provides support for the following
configurations:

- QEMU ARM with Debian Buster
- - QEMU ARM64 with Debian Buster (for host >= buster)
+ - QEMU ARM64 with Debian Buster
- QEMU i386 with Debian Buster
- QEMU amd64 with Debian Buster
- Raspberry Pi various models with Raspberry OS Bullseye
@@ -60,7 +60,7 @@ The steps below describe how to build the images provided by default.

### Install Host Tools

-The supported host system is >= buster.
+The minimal supported host system is Debian Bullseye.

Install the following packages:
```
@@ -74,6 +74,7 @@ apt install \
gettext-base \
git \
python3 \
+ python3-distutils \
quilt \
qemu-user-static \
reprepro \
@@ -87,28 +88,19 @@ apt install \
zstd
```

-If your host is >= buster, also install the following package.
-```
-apt install python3-distutils
-```
-
-**NOTE:** sbuild version (<=0.78.1) packaged in Debian Buster doesn't support
-`$apt_keep_downloaded_packages` option which is required in Isar for
-populating `${DL_DIR}/deb`. So, host `sbuild` in this case should be manually
-upgraded to >=0.81.2 version from Debian Bullseye.
-
Next, the user who should run Isar needs to be added to `sbuild` group.
```
sudo gpasswd -a <username> sbuild
```

If you want to generate containerized SDKs, also install the following
-packages: `umoci` and `skopeo`.
-Umoci is provided by Debian Buster and can be installed with
-`apt install umoci`, Skopeo is provided by Debian Bullseye/Unstable and has to
-be installed either manually downloading the DEB and installing it (no other
-packages required) or with `apt install -t bullseye skopeo` (if
-unstable/bullseye included in `/etc/apt/sources.list[.d]`).
+packages:
+
+```
+apt install \
+ umoci \
+ skopeo
+```

Notes:

@@ -234,9 +226,6 @@ Variables may be used in `.wks.in` files; Isar will expand them and generate a r
In order to run the EFI images with `qemu`, an EFI firmware is required and available at the following address:
https://github.com/tianocore/edk2/tree/3858b4a1ff09d3243fea8d07bd135478237cb8f7

-Note that the `ovmf` package in Debian Buster contains a pre-compiled firmware, but doesn't seem to be recent
-enough to allow images to be testable under `qemu`.
-
```
# AMD64 image, EFI
qemu-system-x86_64 -m 256M -nographic -bios edk2/Build/OvmfX64/RELEASE_*/FV/OVMF.fd -hda tmp/deploy/images/qemuamd64/isar-image-base-debian-buster-qemuamd64.wic
@@ -963,8 +952,8 @@ put into Isar apt.
Debian cross-compilation works out of the box. Currently the following build configurations are supported in Isar:

- buster armhf
- - buster arm64 (for host >= buster)
- - buster mipsel (for host >= buster)
+ - buster arm64
+ - buster mipsel
- bullseye armhf
- bullseye arm64
- bullseye mipsel
--
2.34.1

[Anton Mikanovich]

Some external logic like installing gpg and ca-certificates is now
doing by mmdebstrap inself. So remove duplications.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 26 -------------------
1 file changed, 26 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index dee6cc31..33bdf6b9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -22,8 +22,6 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""

DEPLOY_ISAR_BOOTSTRAP ?= ""
DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"

-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"
@@ -178,30 +176,6 @@ def get_distro_primary_source_entry(d):
return source[2:]
bb.fatal('Invalid apt sources list')

-def get_distro_have_https_source(d):
- return any(source[2].startswith("https://") for source in generate_distro_sources(d))
-
-def get_distro_needs_https_support(d):
- if get_distro_have_https_source(d):
- return "https-support"
- else:
- return ""
-
-OVERRIDES:append = ":${@get_distro_needs_https_support(d)}"
-
-def get_distro_needs_gpg_support(d):
- if d.getVar("DISTRO_BOOTSTRAP_KEYS") or \
- d.getVar("THIRD_PARTY_APT_KEYS") or \
- d.getVar("BASE_REPO_KEY"):
- return "gnupg"
- else:
- return ""
-
-OVERRIDES:append = ":${@get_distro_needs_gpg_support(d)}"
-
-def get_distro_source(d):
- return get_distro_primary_source_entry(d)[0]
-
def get_distro_suite(d):
return get_distro_primary_source_entry(d)[1]

--
2.34.1

[Anton Mikanovich]

Hello,

Just some statistics to proof zst compression choosing.
Building mc:qemuarm64-bookworm:isar-image-base with 3 types of bootstrap tar
compression gives the following measurements:

      | bootstrap, sec | chroot1, sec | chroot2, sec
 gz |          73           |         80        |         88
 xz |          76           |         82        |         89
zst |          68           |         80        |         88

where:
bootstrap - do_bootstrap task of isar-bootstrap-target
chroot1 - do_rootfs_install task of sbuild-chroot-target
chroot2 - do_rootfs_install task of sbuild-chroot-target-db2m
(chroot1 and chroot2 are performing in parallel)

[Anton Mikanovich]

Switch Isar from debootstrap to mmdebstrap.

It makes isar-bootstrap target ~10..30% faster on both cross and native
rootfs types. Some measurements on qemuarm64-bookworm (required debs are
predownloaded and used during deb-dl-import):

- 20s vs 28s for "isar-bootstrap-host"
- 1m32s vs 1m48s for "isar-bootstrap-target

Also mmdebstrap support using SOURCE_DATE_EPOCH for images reproducibly
out of the box. But the main advantage is an ability to remove sudo for
debootstrapping later on (this needs "uidmap" package installed).

This change breaks Debian Buster support as host distro.

Changes since v5:
- Improve keyring handling.
- Fix mmdebstrap hooks.

Changes since v4:
- Rebase on next.
- Migrate from apt-key to gpg.
- Fix Ubuntu and RPi targets.
- Cleanup debootstrap mentions.
- Fix commit messages.

Changes since v3:
- Rebase on latest next.
- Remove temporary patch 7 since kas:4.0 already includes mmdebstrap.

Changes since v2:
- Rebase on next.
- Fix commit messages.
- Update host requirements.

Changes since v1:
- Move mmdebstrap tmpdir to workdir.
- Fix kas building.
- Improve mounts cleanup.

Anton Mikanovich (7):

meta: Preinstall apt for Ubuntu and RPi targets
isar-bootstrap: Move preparations to hooks
isar-bootstrap: Use tar output instead of directory
user_manual.md: Replace debootstrap with mmdebstrap
user_manual.md: Update minimal host requirements
isar-bootstrap: Remove unused code

meta: Fix missing dpkg available

Uladzimir Bely (1):
meta: Switch to mmdebstrap

doc/user_manual.md | 41 +--
meta-isar/conf/distro/ubuntu-common.inc | 4 -
meta/classes/rootfs.bbclass | 11 +-
meta/conf/bitbake.conf | 2 +-

.../isar-bootstrap/isar-bootstrap.inc | 302 +++++++++---------
5 files changed, 171 insertions(+), 189 deletions(-)

--
2.34.1

[Anton Mikanovich]

From: Uladzimir Bely <ub...@ilbers.de>

Instead of debootstrap, use mmdebstrap alternative.

Internally, it uses apt and allows to debootstrap the distro from
multiple repositories.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta-isar/conf/distro/ubuntu-common.inc | 4 --
.../isar-bootstrap/isar-bootstrap.inc | 64 ++++++++++---------
2 files changed, 33 insertions(+), 35 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 6a990a89..afe96b73 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -18,10 +18,6 @@ HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}"

# kernel package name is linux-image-generic in Ubuntu
KERNEL_NAME = "generic"
-
-# that is what debootstrap_1.0.118ubuntu1 does anyways
-DISTRO_DEBOOTSTRAP_SCRIPT = "/usr/share/debootstrap/scripts/gutsy"
-
DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"

DISTRO_KERNELS ?= " \

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3477c2fb..9f1142ba 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-DEBOOTSTRAP ?= "qemu-debootstrap"

ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"

@@ -23,12 +22,14 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"

DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"

+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir"
FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir
@@ -39,7 +40,7 @@ python () {
topdir = d.getVar("TOPDIR")

# The cached repo key can be both for bootstrapping and apt package
- # installation afterwards. However, debootstrap will include the key into
+ # installation afterwards. However, bootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.

if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):

@@ -267,7 +268,7 @@ do_bootstrap[vardeps] += " \

ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"

+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"

do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

@@ -279,31 +280,35 @@ do_bootstrap() {
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
- debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"

+ bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"

if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then

- debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
- fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
- debootstrap_args="$debootstrap_args --no-check-gpg"
+ bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"

fi
E="${@ isar_export_proxies(d)}"

- export BOOTSTRAP_FOR_HOST debootstrap_args E
+ export BOOTSTRAP_FOR_HOST

- sudo rm -rf --one-file-system "${ROOTFSDIR}"

deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

+ sudo rm -rf --one-file-system "${ROOTFSDIR}"
+ mkdir -p "${ROOTFSDIR}"
+

+ if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then

+ export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+ fi
+

+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+

+ sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
+ $arch_param \
+ --mode=unshare \
+ ${@get_distro_components_argument(d)} \
+ "${@get_distro_suite(d)}" \
+ "${WORKDIR}/rootfs.tar.zst" \
+ "${@get_distro_source(d)}"

sudo -E -s <<'EOSUDO'
set -e

- if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
- fi
- ${DEBOOTSTRAP} $debootstrap_args \
- $arch_param \
- ${@get_distro_components_argument(d)} \
- "${@get_distro_suite(d)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
+

+ tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

# Install apt config

mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"

@@ -389,7 +394,7 @@ do_bootstrap() {

umount -l "${ROOTFSDIR}/sys"

umount -l "${ROOTFSDIR}/base-apt" || true

- # Finalize debootstrap by setting the link in deploy

+ # Finalize bootstrap by setting the link in deploy

ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"

EOSUDO

deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

@@ -406,19 +411,16 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically

- lopts="--one-file-system --exclude=var/cache/apt/archives"
- sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")

- sudo chown $(id -u):$(id -g) bootstrap.tar
+ sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo chown $(id -u):$(id -g) bootstrap.tar.zst

}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR

- # - after building the bootstrap, the tar won't be there, but we also don't need to unpack
- # - after restoring from cache, there will be a tar which we unpack and then delete
- if [ -f bootstrap.tar ]; then
- sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar
- sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
- rm bootstrap.tar
+ if [ -f bootstrap.tar.zst ]; then

+ mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
+ sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
fi
}

--
2.34.1

[Anton Mikanovich]

Apt package is not in required packages in Ubuntu and Raspberry repos.
To have it installed during bootstrap introduce BASE_DISTRO override
and append apt to DISTRO_BOOTSTRAP_BASE_PACKAGES for this distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/conf/bitbake.conf | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 80dc01c7..22fdc0e7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -70,7 +70,7 @@ KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"

MACHINEOVERRIDES ?= "${MACHINE}"
-DISTROOVERRIDES ?= "${DISTRO}"
+DISTROOVERRIDES ?= "${DISTRO}:${BASE_DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 9f1142ba..22fb398f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -25,6 +25,9 @@ DEPLOY_ISAR_BOOTSTRAP ?= ""

DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"

+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"

DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"

--
2.34.1

[Anton Mikanovich]

Move all bootstrap rootfs prepare logic into mmdebstrap hooks.
Also migrate from apt-key to gpg.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 204 ++++++++++--------
1 file changed, 109 insertions(+), 95 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 22fb398f..d7fc8b01 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -213,26 +213,29 @@ def get_distro_components_argument(d):

else:
return ""

-APT_KEYS_DIR = "${WORKDIR}/aptkeys"
DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"

-do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
-do_generate_keyrings[dirs] = "${DL_DIR}"
+do_generate_keyrings[cleandirs] = "${WORKDIR}/trusted.gpg.d"
+do_generate_keyrings[dirs] = "${DEBDIR}"
do_generate_keyrings[vardeps] += "DISTRO_BOOTSTRAP_KEYS THIRD_PARTY_APT_KEYS"
do_generate_keyrings[network] = "${TASK_USE_SUDO}"
do_generate_keyrings() {

+ export GNUPGHOME="$(mktemp -td gpghomeXXXXXXXXXX)"

if [ -n "${@d.getVar("THIRD_PARTY_APT_KEYFILES") or ""}" ]; then
- chmod 777 "${APT_KEYS_DIR}"
for keyfile in ${@d.getVar("THIRD_PARTY_APT_KEYFILES")}; do
- cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"

+ gpg --no-autostart --keyring "gnupg-ring:${DISTRO_BOOTSTRAP_KEYRING}" \
+ --no-default-keyring --import $keyfile

done
fi
if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES") or ""}" ]; then
for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES")}; do
- sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
- cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"

+ gpg --no-autostart --keyring "gnupg-ring:${DISTRO_BOOTSTRAP_KEYRING}" \
+ --no-default-keyring --import $keyfile
done
fi
+ if [ -r "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
+ chmod o+r "${DISTRO_BOOTSTRAP_KEYRING}"
+ fi
}
addtask generate_keyrings before do_build after do_unpack

@@ -271,12 +274,14 @@ do_bootstrap[vardeps] += " \
ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"

-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"

do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

inherit compat

+DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then

@@ -286,124 +291,133 @@ do_bootstrap() {

bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then

bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
+ cp "${DISTRO_BOOTSTRAP_KEYRING}" "${WORKDIR}/trusted.gpg.d/"

fi
E="${@ isar_export_proxies(d)}"
- export BOOTSTRAP_FOR_HOST

- deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${ROOTFSDIR}"

mkdir -p "${ROOTFSDIR}"

+ deb_dl_dir_import "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+

+ bootstrap_list="${WORKDIR}/sources.list.d/bootstrap.list"
+ install -v -m644 "${APTSRCS}" \
+ "${WORKDIR}/sources.list.d/bootstrap.list"
+
+ syncin='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"*.deb \
+ "$1/var/cache/apt/archives/" || true'
+ syncout='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "$1/var/cache/apt/archives/"*.deb \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"'
+ extra_setup="$syncin"
+ extra_extract="$syncout"
+ # prefetch apt debs because mmdebstrap will clean them on next stage
+ extra_essential='apt-get install apt -y -d \
+ -o Dir::State="$1/var/lib/apt" \
+ -o Dir::Etc="$1/etc/apt" \
+ -o Dir::Cache="$1/var/cache/apt" \
+ -o Apt::Architecture="${BOOTSTRAP_DISTRO_ARCH}"'
+ extra_essential="$extra_essential && $syncout"
+ extra_customize="$syncout"
+ fi

+
if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then

export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
fi

arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"

+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
+ fi
+
+ # Cleanup mounts if fails
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \
+ && rm "${WORKDIR}/mmtmpdir"; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/$base_apt_tmp \
+ && sudo umount $tmpdir/$base_apt_tmp; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/base-apt \
+ && sudo umount $tmpdir/base-apt; \
+ [ -d "$tmpdir" ] && sudo rm -rf --one-file-system $tmpdir; \
+ [ -n "$base_apt_tmp" ] && mountpoint -q $base_apt_tmp \
+ && sudo umount $base_apt_tmp \
+ && rm -rf --one-file-system $base_apt_tmp' EXIT

sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \

$arch_param \
--mode=unshare \
+ --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \
+ --setup-hook="$extra_setup" \
+ --setup-hook='upload "${APTPREFS}" /etc/apt/preferences.d/bootstrap' \
+ --setup-hook='upload "${APTSRCS_INIT}" /etc/apt/sources-list' \
+ --setup-hook='upload "${WORKDIR}/locale" /etc/locale' \
+ --setup-hook='mkdir -p "$1/etc/apt/trusted.gpg.d"' \
+ --setup-hook='sync-in "${WORKDIR}/trusted.gpg.d" /etc/apt/trusted.gpg.d' \
+ --setup-hook='install -v -m755 "${WORKDIR}/chroot-setup.sh" "$1/chroot-setup.sh"' \
+ --extract-hook="$extra_extract" \
+ --essential-hook="$extra_essential" \
+ --customize-hook="$extra_customize" \
+ --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' \
+ --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
+ --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
+ --skip=cleanup/apt \
+ --skip=download/empty \
${@get_distro_components_argument(d)} \
"${@get_distro_suite(d)}" \
"${WORKDIR}/rootfs.tar.zst" \
- "${@get_distro_source(d)}"
+ "$bootstrap_list"

sudo -E -s <<'EOSUDO'
set -e

tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"

"${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"

- umount -l "${ROOTFSDIR}/sys"
- umount -l "${ROOTFSDIR}/base-apt" || true
-

# Finalize bootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO

[Anton Mikanovich]

Do not unpack tar with rootfs content inside bootstrap recipe.
This change can allow to remove sudo usage during the bootstrap later.
Using tar output requires moving chroot prepare step to the later
unpack step.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d7fc8b01..0cc8ee7f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-ROOTFSDIR = "${WORKDIR}/rootfs"

APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"

APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

@@ -295,9 +294,6 @@ do_bootstrap() {

fi
E="${@ isar_export_proxies(d)}"

- sudo rm -rf --one-file-system "${ROOTFSDIR}"

- mkdir -p "${ROOTFSDIR}"

-
if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"

@@ -402,18 +398,9 @@ do_bootstrap() {

"${WORKDIR}/rootfs.tar.zst" \
"$bootstrap_list"

- sudo -E -s <<'EOSUDO'
- set -e
-

- tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
-

- # setup chroot
- install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
- "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+ # Finalize bootstrap by setting the link in deploy
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"

- # Finalize bootstrap by setting the link in deploy

- ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-EOSUDO

if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then

deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"

@@ -428,15 +415,15 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically

- sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst

sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR

if [ -f bootstrap.tar.zst ]; then
- mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
- sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst"
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \

[Anton Mikanovich]

Update documentation in user manual.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index ecb987f4..6cbff8cf 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

[Anton Mikanovich]

Using mmdebstrap requires at least Debian Bullseye on host.
Update user manual with the new minimal host distro.
Also remove all Buster-related notes.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 35 ++++++++++++-----------------------
1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 6cbff8cf..512721c9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

[Anton Mikanovich]

Some external logic like installing gpg and ca-certificates is now
doing by mmdebstrap inself. So remove duplications.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 26 -------------------
1 file changed, 26 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0cc8ee7f..98345755 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -22,8 +22,6 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""

DEPLOY_ISAR_BOOTSTRAP ?= ""
DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"

[Anton Mikanovich]

In some distros /var/lib/dpkg/available is required for
do_rootfs_install task.
The fix for it is present in jessie-or-older mmdebstrap hook in
Bookworm and later, but maybe-jessie-or-older wrapper do not handle
distro version check correctly.
That's why we need to check and apply hook manually or create
/var/lib/dpkg/available by hand for older distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++++
1 file changed, 12 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 98345755..03237c6c 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -34,6 +34,17 @@ FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir

+# Fix for /var/lib/apt/available while maybe-jessie-or-older hook do not work
+MM_HOOK_JESSIE = "/usr/share/mmdebstrap/hooks/jessie-or-older"
+DPKG_HOOKS ?= "${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \
+ if os.path.isdir(d.getVar('MM_HOOK_JESSIE')) \
+ else '--customize-hook=\'touch /var/lib/dpkg/available\''}"
+MMHOOKS:ubuntu-focal ?= "${DPKG_HOOKS}"
+MMHOOKS:debian-buster ?= "${DPKG_HOOKS}"
+
+def get_distro_suite(d):
+ return get_distro_primary_source_entry(d)[1]
+
python () {
distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
third_party_apt_keys = (d.getVar("THIRD_PARTY_APT_KEYS") or "").split()
@@ -351,6 +362,7 @@ do_bootstrap() {

sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
$arch_param \
--mode=unshare \

+ ${MMHOOKS} \
--setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \
--setup-hook="$extra_setup" \

--setup-hook='upload "${APTPREFS}" /etc/apt/preferences.d/bootstrap' \

--
2.34.1

[Jan Kiszka]

On 24.10.23 08:23, Anton Mikanovich wrote:
> Switch Isar from debootstrap to mmdebstrap.
>
> It makes isar-bootstrap target ~10..30% faster on both cross and native
> rootfs types. Some measurements on qemuarm64-bookworm (required debs are
> predownloaded and used during deb-dl-import):
>
> - 20s vs 28s for "isar-bootstrap-host"
> - 1m32s vs 1m48s for "isar-bootstrap-target
>
> Also mmdebstrap support using SOURCE_DATE_EPOCH for images reproducibly
> out of the box. But the main advantage is an ability to remove sudo for
> debootstrapping later on (this needs "uidmap" package installed).
>
> This change breaks Debian Buster support as host distro.

No other regressions known? Or do we need an option to select the
bootstrap variant, at least for some transition period?

BTW, the series seems to be a bit misordered, given that it first
switched to mmdebstrap and only after that fixes related issues. Is that
unavoidable?

Jan

Siemens AG, Technology
Linux Expert Center

[Anton Mikanovich]

27/10/2023 18:36, Jan Kiszka wrote:
> No other regressions known? Or do we need an option to select the
> bootstrap variant, at least for some transition period?

There are no other regressions found so far, so I think loosing
oldoldstrable
build host support is not a big issue.
Bootstrap variant selection will require a lot of effort and can cause
issues
by itself. And it probably can be done only with huge code duplication
just like
we had it with buildchroot + sbuild prevoiusly.

> BTW, the series seems to be a bit misordered, given that it first
> switched to mmdebstrap and only after that fixes related issues. Is that
> unavoidable?

Right, it looks not really nice. Will rearrange some patches and resend.
> Jan

[Anton Mikanovich]

Switch Isar from debootstrap to mmdebstrap.

It makes isar-bootstrap target ~10..30% faster on both cross and native
rootfs types. Some measurements on qemuarm64-bookworm (required debs are
predownloaded and used during deb-dl-import):

- 20s vs 28s for "isar-bootstrap-host"
- 1m32s vs 1m48s for "isar-bootstrap-target

Also mmdebstrap support using SOURCE_DATE_EPOCH for images reproducibly
out of the box. But the main advantage is an ability to remove sudo for
debootstrapping later on (this needs "uidmap" package installed).

This change breaks Debian Buster support as host distro.

Changes since v6:
- Reorder patches.

Changes since v5:
- Improve keyring handling.
- Fix mmdebstrap hooks.

Changes since v4:
- Rebase on next.
- Migrate from apt-key to gpg.
- Fix Ubuntu and RPi targets.
- Cleanup debootstrap mentions.
- Fix commit messages.

Changes since v3:
- Rebase on latest next.
- Remove temporary patch 7 since kas:4.0 already includes mmdebstrap.

Changes since v2:
- Rebase on next.
- Fix commit messages.
- Update host requirements.

Changes since v1:
- Move mmdebstrap tmpdir to workdir.
- Fix kas building.
- Improve mounts cleanup.

Anton Mikanovich (7):
meta: Preinstall apt for Ubuntu and RPi targets

meta: Fix missing dpkg available
isar-bootstrap: Move preparations to hooks
isar-bootstrap: Use tar output instead of directory
user_manual.md: Replace debootstrap with mmdebstrap
user_manual.md: Update minimal host requirements
isar-bootstrap: Remove unused code

Uladzimir Bely (1):
meta: Switch to mmdebstrap

doc/user_manual.md | 41 +--
meta-isar/conf/distro/ubuntu-common.inc | 4 -
meta/classes/rootfs.bbclass | 11 +-
meta/conf/bitbake.conf | 2 +-
.../isar-bootstrap/isar-bootstrap.inc | 302 +++++++++---------
5 files changed, 171 insertions(+), 189 deletions(-)

--

2.34.1

[Anton Mikanovich]

Apt package is not in required packages in Ubuntu and Raspberry repos.
To have it installed during bootstrap introduce BASE_DISTRO override

and append apt to DISTRO_BOOTSTRAP_BASE_PACKAGES for this distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/conf/bitbake.conf | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 3 +++
2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 80dc01c7..22fdc0e7 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -70,7 +70,7 @@ KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"

MACHINEOVERRIDES ?= "${MACHINE}"
-DISTROOVERRIDES ?= "${DISTRO}"
+DISTROOVERRIDES ?= "${DISTRO}:${BASE_DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3477c2fb..334898fd 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -26,6 +26,9 @@ DEPLOY_ISAR_BOOTSTRAP ?= ""
DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"

[Anton Mikanovich]

From: Uladzimir Bely <ub...@ilbers.de>

Instead of debootstrap, use mmdebstrap alternative.

Internally, it uses apt and allows to debootstrap the distro from
multiple repositories.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta-isar/conf/distro/ubuntu-common.inc | 4 --
.../isar-bootstrap/isar-bootstrap.inc | 64 ++++++++++---------
2 files changed, 33 insertions(+), 35 deletions(-)

diff --git a/meta-isar/conf/distro/ubuntu-common.inc b/meta-isar/conf/distro/ubuntu-common.inc
index 6a990a89..afe96b73 100644
--- a/meta-isar/conf/distro/ubuntu-common.inc
+++ b/meta-isar/conf/distro/ubuntu-common.inc
@@ -18,10 +18,6 @@ HOST_DISTRO_BOOTSTRAP_KEYS += "${BOOTSTRAP_KEY}"

# kernel package name is linux-image-generic in Ubuntu
KERNEL_NAME = "generic"
-
-# that is what debootstrap_1.0.118ubuntu1 does anyways
-DISTRO_DEBOOTSTRAP_SCRIPT = "/usr/share/debootstrap/scripts/gutsy"
-
DISTRO_CONFIG_SCRIPT ?= "debian-configscript.sh"

DISTRO_KERNELS ?= " \

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 334898fd..22fb398f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-DEBOOTSTRAP ?= "qemu-debootstrap"

ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"

@@ -23,7 +22,7 @@ APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"
DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"

@@ -32,6 +31,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"

DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"

+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+BOOTSTRAP_TMPDIR = "${WORKDIR}/tempdir"

FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir

@@ -42,7 +43,7 @@ python () {

topdir = d.getVar("TOPDIR")

# The cached repo key can be both for bootstrapping and apt package
- # installation afterwards. However, debootstrap will include the key into
+ # installation afterwards. However, bootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.

if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):

@@ -270,7 +271,7 @@ do_bootstrap[vardeps] += " \

ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"
-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"

+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"

do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

@@ -282,31 +283,35 @@ do_bootstrap() {

bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
- debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"

+ bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"

if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then

- debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
- fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
- debootstrap_args="$debootstrap_args --no-check-gpg"
+ bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"

fi
E="${@ isar_export_proxies(d)}"

- export BOOTSTRAP_FOR_HOST debootstrap_args E
+ export BOOTSTRAP_FOR_HOST

- sudo rm -rf --one-file-system "${ROOTFSDIR}"

deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

+ sudo rm -rf --one-file-system "${ROOTFSDIR}"
+ mkdir -p "${ROOTFSDIR}"
+

+ if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then

+ export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
+ fi
+

+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"
+

+ sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
+ $arch_param \
+ --mode=unshare \
+ ${@get_distro_components_argument(d)} \
+ "${@get_distro_suite(d)}" \
+ "${WORKDIR}/rootfs.tar.zst" \
+ "${@get_distro_source(d)}"

sudo -E -s <<'EOSUDO'
set -e

- if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
- fi
- ${DEBOOTSTRAP} $debootstrap_args \
- $arch_param \
- ${@get_distro_components_argument(d)} \
- "${@get_distro_suite(d)}" \
- "${ROOTFSDIR}" \
- "${@get_distro_source(d)}" \
- ${DISTRO_DEBOOTSTRAP_SCRIPT}
+

+ tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

# Install apt config

mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"

@@ -392,7 +397,7 @@ do_bootstrap() {

umount -l "${ROOTFSDIR}/sys"

umount -l "${ROOTFSDIR}/base-apt" || true

- # Finalize debootstrap by setting the link in deploy

+ # Finalize bootstrap by setting the link in deploy

ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"

EOSUDO

deb_dl_dir_export "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

@@ -409,19 +414,16 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically

- lopts="--one-file-system --exclude=var/cache/apt/archives"
- sudo tar -C $(dirname "${ROOTFSDIR}") -cpSf bootstrap.tar $lopts $(basename "${ROOTFSDIR}")

- sudo chown $(id -u):$(id -g) bootstrap.tar
+ sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo chown $(id -u):$(id -g) bootstrap.tar.zst

}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR

- # - after building the bootstrap, the tar won't be there, but we also don't need to unpack
- # - after restoring from cache, there will be a tar which we unpack and then delete
- if [ -f bootstrap.tar ]; then
- sudo tar -C $(dirname "${ROOTFSDIR}") -xpf bootstrap.tar
- sudo ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
- rm bootstrap.tar
+ if [ -f bootstrap.tar.zst ]; then

+ mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
+ sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"
fi
}

--
2.34.1

[Anton Mikanovich]

In some distros /var/lib/dpkg/available is required for
do_rootfs_install task.
The fix for it is present in jessie-or-older mmdebstrap hook in
Bookworm and later, but maybe-jessie-or-older wrapper do not handle
distro version check correctly.
That's why we need to check and apply hook manually or create
/var/lib/dpkg/available by hand for older distros.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++++
1 file changed, 12 insertions(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 22fb398f..8048521f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -37,6 +37,17 @@ FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir

+# Fix for /var/lib/apt/available while maybe-jessie-or-older hook do not work
+MM_HOOK_JESSIE = "/usr/share/mmdebstrap/hooks/jessie-or-older"
+DPKG_HOOKS ?= "${@'--hook-dir='+d.getVar('MM_HOOK_JESSIE') \
+ if os.path.isdir(d.getVar('MM_HOOK_JESSIE')) \
+ else '--customize-hook=\'touch /var/lib/dpkg/available\''}"
+MMHOOKS:ubuntu-focal ?= "${DPKG_HOOKS}"
+MMHOOKS:debian-buster ?= "${DPKG_HOOKS}"
+
+def get_distro_suite(d):
+ return get_distro_primary_source_entry(d)[1]
+
python () {
distro_bootstrap_keys = (d.getVar("DISTRO_BOOTSTRAP_KEYS") or "").split()
third_party_apt_keys = (d.getVar("THIRD_PARTY_APT_KEYS") or "").split()

@@ -303,6 +314,7 @@ do_bootstrap() {

sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \

$arch_param \
--mode=unshare \
+ ${MMHOOKS} \

${@get_distro_components_argument(d)} \
"${@get_distro_suite(d)}" \
"${WORKDIR}/rootfs.tar.zst" \

--
2.34.1

[Anton Mikanovich]

Move all bootstrap rootfs prepare logic into mmdebstrap hooks.
Also migrate from apt-key to gpg.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 204 ++++++++++--------
1 file changed, 109 insertions(+), 95 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 8048521f..d89ad56d 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -224,26 +224,29 @@ def get_distro_components_argument(d):

@@ -282,12 +285,14 @@ do_bootstrap[vardeps] += " \
ISAR_ENABLE_COMPAT_ARCH \
${DISTRO_VARS_PREFIX}DISTRO_APT_SOURCES \
"

-do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR}"
+do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP} ${BOOTSTRAP_TMPDIR} ${WORKDIR}/trusted.gpg.d ${WORKDIR}/sources.list.d"

do_bootstrap[depends] = "base-apt:do_cache isar-apt:do_cache_config"
do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

inherit compat

+DEB_DL_LOCK ?= "${DEBDIR}/${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}.lock"
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then

@@ -297,125 +302,134 @@ do_bootstrap() {

bootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then

bootstrap_args="$bootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
+ cp "${DISTRO_BOOTSTRAP_KEYRING}" "${WORKDIR}/trusted.gpg.d/"

fi
E="${@ isar_export_proxies(d)}"
- export BOOTSTRAP_FOR_HOST

- deb_dl_dir_import "${ROOTFSDIR}" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${ROOTFSDIR}"

mkdir -p "${ROOTFSDIR}"

+ deb_dl_dir_import "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+

+ bootstrap_list="${WORKDIR}/sources.list.d/bootstrap.list"
+ install -v -m644 "${APTSRCS}" \
+ "${WORKDIR}/sources.list.d/bootstrap.list"
+
+ syncin='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"*.deb \
+ "$1/var/cache/apt/archives/" || true'
+ syncout='flock -s ${DEB_DL_LOCK} cp -n --no-preserve=owner \
+ "$1/var/cache/apt/archives/"*.deb \
+ "${WORKDIR}/dl_dir/var/cache/apt/archives/"'
+ extra_setup="$syncin"
+ extra_extract="$syncout"
+ # prefetch apt debs because mmdebstrap will clean them on next stage
+ extra_essential='apt-get install apt -y -d \
+ -o Dir::State="$1/var/lib/apt" \
+ -o Dir::Etc="$1/etc/apt" \
+ -o Dir::Cache="$1/var/cache/apt" \
+ -o Apt::Architecture="${BOOTSTRAP_DISTRO_ARCH}"'
+ extra_essential="$extra_essential && $syncout"
+ extra_customize="$syncout"
+ fi

+
if [ ! -z "${SOURCE_DATE_EPOCH}" ]; then

export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH}"
fi

arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH},${DISTRO_ARCH}"

+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ arch_param="$arch_param,${COMPAT_DISTRO_ARCH}"
+ fi
+
+ # Cleanup mounts if fails
+ trap 'exit 1' INT HUP QUIT TERM ALRM USR1
+ trap '[ -r "${WORKDIR}/mmtmpdir" ] && tmpdir=$(cat "${WORKDIR}/mmtmpdir") \
+ && rm "${WORKDIR}/mmtmpdir"; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/$base_apt_tmp \
+ && sudo umount $tmpdir/$base_apt_tmp; \
+ [ -d "$tmpdir" ] && mountpoint -q $tmpdir/base-apt \
+ && sudo umount $tmpdir/base-apt; \
+ [ -d "$tmpdir" ] && sudo rm -rf --one-file-system $tmpdir; \
+ [ -n "$base_apt_tmp" ] && mountpoint -q $base_apt_tmp \
+ && sudo umount $base_apt_tmp \
+ && rm -rf --one-file-system $base_apt_tmp' EXIT

sudo TMPDIR="${BOOTSTRAP_TMPDIR}" mmdebstrap $bootstrap_args \
$arch_param \
--mode=unshare \

${MMHOOKS} \

+ --setup-hook='mkdir -p "$1/var/cache/apt/archives/"' \
+ --setup-hook="$extra_setup" \
+ --setup-hook='upload "${APTPREFS}" /etc/apt/preferences.d/bootstrap' \
+ --setup-hook='upload "${APTSRCS_INIT}" /etc/apt/sources-list' \
+ --setup-hook='upload "${WORKDIR}/locale" /etc/locale' \
+ --setup-hook='mkdir -p "$1/etc/apt/trusted.gpg.d"' \
+ --setup-hook='sync-in "${WORKDIR}/trusted.gpg.d" /etc/apt/trusted.gpg.d' \
+ --setup-hook='install -v -m755 "${WORKDIR}/chroot-setup.sh" "$1/chroot-setup.sh"' \
+ --extract-hook="$extra_extract" \
+ --essential-hook="$extra_essential" \
+ --customize-hook="$extra_customize" \
+ --customize-hook='sed -i "/en_US.UTF-8 UTF-8/s/^#//g" "$1/etc/locale.gen"' \
+ --customize-hook='chroot "$1" /usr/sbin/locale-gen' \
+ --customize-hook='chroot "$1" /usr/bin/apt-get -y clean' \
+ --skip=cleanup/apt \
+ --skip=download/empty \

${@get_distro_components_argument(d)} \
"${@get_distro_suite(d)}" \
"${WORKDIR}/rootfs.tar.zst" \

- "${@get_distro_source(d)}"
+ "$bootstrap_list"

sudo -E -s <<'EOSUDO'
set -e

tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"

install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"

"${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"

- umount -l "${ROOTFSDIR}/sys"
- umount -l "${ROOTFSDIR}/base-apt" || true
-

# Finalize bootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO

[Anton Mikanovich]

Do not unpack tar with rootfs content inside bootstrap recipe.
This change can allow to remove sudo usage during the bootstrap later.
Using tar output requires moving chroot prepare step to the later
unpack step.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d89ad56d..3048e0e0 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

@@ -15,7 +15,6 @@ PV = "1.0"

BOOTSTRAP_FOR_HOST ?= "0"

-ROOTFSDIR = "${WORKDIR}/rootfs"

APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"

APTSRCS_INIT = "${WORKDIR}/apt-sources-init"

@@ -306,9 +305,6 @@ do_bootstrap() {

fi
E="${@ isar_export_proxies(d)}"

- sudo rm -rf --one-file-system "${ROOTFSDIR}"

- mkdir -p "${ROOTFSDIR}"

-
if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

base_apt_tmp="$(mktemp -d /tmp/isar-base-aptXXXXXXXXXX)"
bootstrap_list="${WORKDIR}/sources.list.d/base-apt.list"

@@ -414,18 +410,9 @@ do_bootstrap() {

"${WORKDIR}/rootfs.tar.zst" \
"$bootstrap_list"

- sudo -E -s <<'EOSUDO'
- set -e
-

- tar -xf "${WORKDIR}/rootfs.tar.zst" -C "${ROOTFSDIR}" --exclude="./dev/console"
-

- # setup chroot
- install -v -m755 "${WORKDIR}/chroot-setup.sh" "${ROOTFSDIR}/chroot-setup.sh"
- "${ROOTFSDIR}/chroot-setup.sh" "setup" "${ROOTFSDIR}"
+ # Finalize bootstrap by setting the link in deploy
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" "${DEPLOY_ISAR_BOOTSTRAP}.tar.zst"

- # Finalize bootstrap by setting the link in deploy

- ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
-EOSUDO

if [ "${ISAR_USE_CACHED_BASE_REPO}" != "1" ]; then

deb_dl_dir_export "${WORKDIR}/dl_dir" "${BOOTSTRAP_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

sudo rm -rf --one-file-system "${WORKDIR}/dl_dir"

@@ -440,15 +427,15 @@ SSTATEPOSTINSTFUNCS += "bootstrap_sstate_finalize"

bootstrap_sstate_prepare() {
# this runs in SSTATE_BUILDDIR, which will be deleted automatically

- sudo cp -a "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" ./bootstrap.tar.zst
+ sudo cp -a "${WORKDIR}/rootfs.tar.zst" ./bootstrap.tar.zst

sudo chown $(id -u):$(id -g) bootstrap.tar.zst
}

bootstrap_sstate_finalize() {
# this runs in SSTATE_INSTDIR

if [ -f bootstrap.tar.zst ]; then
- mv bootstrap.tar.zst "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst"
- sudo ln -Tfsr "$(dirname "${ROOTFSDIR}")/rootfs.tar.zst" \
+ mv bootstrap.tar.zst "${WORKDIR}/rootfs.tar.zst"
+ sudo ln -Tfsr "${WORKDIR}/rootfs.tar.zst" \

[Anton Mikanovich]

Update documentation in user manual.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index ecb987f4..6cbff8cf 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

[Anton Mikanovich]

Using mmdebstrap requires at least Debian Bullseye on host.
Update user manual with the new minimal host distro.
Also remove all Buster-related notes.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

doc/user_manual.md | 35 ++++++++++++-----------------------
1 file changed, 12 insertions(+), 23 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 6cbff8cf..512721c9 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md

[Anton Mikanovich]

Some external logic like installing gpg and ca-certificates is now
doing by mmdebstrap inself. So remove duplications.

Signed-off-by: Anton Mikanovich <ami...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 26 -------------------
1 file changed, 26 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 3048e0e0..03237c6c 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -22,8 +22,6 @@ DISTRO_BOOTSTRAP_KEYFILES = ""
THIRD_PARTY_APT_KEYFILES = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales,usrmerge"
-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:gnupg = ",gnupg"

-DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:ubuntu = ",apt"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios = ",apt"

DISTRO_BOOTSTRAP_BASE_PACKAGES:append:raspios64 = ",apt"

@@ -189,30 +187,6 @@ def get_distro_primary_source_entry(d):