[PATCH v5 00/12] Improving base-apt usage PoC

[Uladzimir Bely]

Currently, base-apt is used in the following way:

- After the first build, all .deb files that took part in the process
are is cached in the ${DL_DIR}/deb/. All the packages are downloaded
from the remote repositories.

- At the second build, `base-apt` repo is first created from the
previously downloaded packages (if ISAR_USE_CACHED_BASE_REPO is set).
Futher debootstrap and installing packages are done from this local repo

The idea of this patchset is using local `base-apt` repo even at
the first build. Before any build step that requires additional
packages, these packages are predownloaded with `debrepo` script to
the `base-apt` repo and the debootstrapping / installation are always
done from it.

Potentialy, such an approach allows to stop using deb-del-dir import
and export functionality in favour of debian-like repositories.

Currently, separate `debrepo` script is used for prefetching packages
to the local 'base-apt' repo. It requires python3-apt to be installed
on the build host. Potentially, the functionality it provides could be
directly integrated to the `debrepo.bbclass`.

Changes since v4:
- rebased on latest next;
- rearranged patches since some of them are already in next;
- added possibility to select between new "prefetch" base-apt mode and
old behaviour when it's populated on 2nd build with packages downloaded
during 1st build. New behaviour is disabled by default, but enabled in
local.conf.example for testing purposes.
- code passes both full and fast CI in both "old" and "new" modes.

Changes since v3:
- rebased on latest next;
- cross-building for raspberry supported;
- code passes both full and fast CI.

Changes since v2:
- populate base-apt before using at all steps of native build.

Changes since v1:
- rebased on latest next;
- updated patchset description.

Uladzimir Bely (12):
meta: move base-apt from deploy directory to the top
ci_build.sh: Install python3-apt if not installed
scripts: Add debrepo python script handling base-apt
meta: Add debrepo bbclass handling base-apt prefetching
meta: Always use base-apt repo in local mode
meta: Use cached base-apt repo to debootstrap
meta: Consider global debrepo context
base-apt: Predownload packages to base-apt before install
meta: Add cache-deb-src functionality in base-apt mode
Set ISAR_PREFETCH_BASE_APT by default
meta: Specify grub-efi packages arch
Disable deb-dl-dir in base-apt prefetch mode

RECIPE-API-CHANGELOG.md | 9 +
meta-isar/conf/local.conf.sample | 4 +
meta/classes/buildchroot.bbclass | 5 +-
meta/classes/deb-dl-dir.bbclass | 25 +
meta/classes/debrepo.bbclass | 86 ++++
meta/classes/dpkg-base.bbclass | 47 +-
meta/classes/dpkg.bbclass | 8 +
meta/classes/image-locales-extension.bbclass | 5 +
meta/classes/image-tools-extension.bbclass | 9 +
meta/classes/rootfs.bbclass | 13 +-
meta/conf/bitbake.conf | 9 +-
meta/conf/distro/debian-common.conf | 8 +-
.../isar-bootstrap/isar-bootstrap.inc | 111 ++++-
meta/recipes-devtools/base-apt/base-apt.bb | 21 +-
.../buildchroot/buildchroot.inc | 11 +
.../sbuild-chroot/sbuild-chroot.inc | 11 +
scripts/ci_build.sh | 8 +-
scripts/debrepo | 443 ++++++++++++++++++
testsuite/citest.py | 3 +
19 files changed, 804 insertions(+), 32 deletions(-)
create mode 100644 meta/classes/debrepo.bbclass
create mode 100755 scripts/debrepo

--
2.20.1

[Uladzimir Bely]

This is the main utility responsible for prefetching packages
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").

Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.

In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
scripts/debrepo | 443 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 443 insertions(+)
create mode 100755 scripts/debrepo

diff --git a/scripts/debrepo b/scripts/debrepo
new file mode 100755
index 00000000..2a5e5b51
--- /dev/null
+++ b/scripts/debrepo
@@ -0,0 +1,443 @@
+#!/usr/bin/env python3
+
+# This software is a part of ISAR.
+# Copyright (C) 2022 ilbers GmbH
+
+import os
+import sys
+import fcntl
+
+import shutil
+import subprocess
+import getopt
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+class DebRepo(object):
+ def __init__(self, workdir, cmdline_opts):
+ self.workdir = workdir
+ self.optsfile = self.workdir + "/repo.opts"
+
+ # Set default values
+ self.distro = "debian"
+
+ self.repo = self.workdir + "/repo/apt" + "/" + self.distro
+ self.repodb = self.workdir + "/repo/db" + "/" + self.distro
+ self.mirror = "http://deb.debian.org/debian"
+ self.arch = "amd64"
+ self.codename = "bullseye"
+ self.keydir = "/etc/apt/trusted.gpg.d"
+ self.check_gpg = True
+ self.isaraptdir = ""
+
+ # Load stored opts
+ opts = self.load_opts()
+ print("stored opts: " + str(opts))
+
+ # Overwrite opts by cmdline_opts
+ for opt, arg in cmdline_opts.items():
+ opts[opt] = arg
+
+ print("all opts: " + str(opts))
+
+ # Replace by values passed in commandline
+ for opt, arg in opts.items():
+ if opt == "mirror":
+ self.mirror = arg
+ if opt == "arch":
+ self.arch = arg
+ if opt == "distro":
+ self.distro = arg
+ if opt == "codename":
+ self.codename = arg
+ if opt == "keydir":
+ self.keydir = arg
+ if opt == "isaraptdir":
+ self.isaraptdir = arg
+ if opt == "check_gpg":
+ self.check_gpg = arg
+
+ self.crossarch = self.arch
+ for opt, arg in opts.items():
+ if opt == "crossarch":
+ self.crossarch = arg
+
+ self.compatarch = ""
+ for opt, arg in opts.items():
+ if opt == "compatarch":
+ self.compatarch = arg
+
+ for opt, arg in opts.items():
+ if opt == "repodir":
+ self.repo = arg + "/" + self.distro
+ if opt == "repodbdir":
+ self.repodb = arg + "/" + self.distro
+
+ self.save_opts(opts)
+
+ print("workdir: " + str(self.workdir))
+ print("repo: " + str(self.repo))
+ print("repodb: " + str(self.repodb))
+ print("mirror: " + str(self.mirror))
+ print("arch: " + str(self.arch))
+ print("crossarch: " + str(self.crossarch))
+ if self.compatarch:
+ print("compatarch: " + str(self.compatarch))
+ print("distro: " + str(self.distro))
+ print("codename: " + str(self.codename))
+ print("keydir: " + str(self.keydir))
+ print("isaraptdir: " + str(self.isaraptdir))
+ print("check_gpg: " + str(self.check_gpg))
+
+ self.cache = None
+ self.depcache = None
+ self.sr = None
+
+ def create_rootfs(self, aptsrcsfile):
+ if not os.path.exists(self.workdir + "/var/lib/dpkg"):
+ os.makedirs(self.workdir + "/var/lib/dpkg")
+ with open(self.workdir + "/var/lib/dpkg" + "/status", "w"):
+ pass
+
+ if not os.path.exists(self.workdir + "/etc/apt/sources.list.d"):
+ os.makedirs(self.workdir + "/etc/apt/sources.list.d")
+ if os.path.exists(aptsrcsfile):
+ shutil.copy(aptsrcsfile, self.workdir + "/etc/apt/sources.list.d/bootstrap.list")
+
+ if self.isaraptdir:
+ with open(self.workdir + "/etc/apt/sources.list.d/isar-apt.list", "w") as f:
+ f.write("deb [trusted=yes] file://" + self.isaraptdir + " isar main\n")
+
+ if not os.path.exists(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial"):
+ os.makedirs(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + "/archives/partial")
+
+ if not os.path.exists(self.workdir + "/tmp"):
+ os.makedirs(self.workdir + "/tmp")
+
+ def apt_config(self, init, crossbuild):
+ # Configure apt to work with empty directory
+ if not init and self.arch != self.crossarch:
+ apt_pkg.config["APT::Architectures::"] = self.crossarch
+ apt_pkg.config["APT::Architectures::"] = self.arch
+
+ if not init and self.compatarch:
+ apt_pkg.config["APT::Architectures::"] = self.compatarch
+
+ apt_pkg.config.set("APT::Architecture", self.arch)
+
+ apt_pkg.config.set("Dir", self.workdir)
+ apt_pkg.config.set("Dir::Cache", self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename)
+ apt_pkg.config.set("Dir::State::status", self.workdir + "/var/lib/dpkg/status")
+
+ apt_pkg.config.set("APT::Install-Recommends", "0")
+ apt_pkg.config.set("APT::Install-Suggests", "0")
+
+ # Use host keys for authentification
+ # apt_pkg.config.set("Dir::Etc::Trusted", "/etc/apt/trusted.gpg")
+ # apt_pkg.config.set("Dir::Etc::TrustedParts", "/etc/apt/trusted.gpg.d")
+ apt_pkg.config.set("Dir::Etc::TrustedParts", self.keydir)
+
+ # Allow using repositories without keys
+ if not self.check_gpg:
+ apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+ def mark_essential(self):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.arch:
+ if pkg.essential:
+ self.depcache.mark_install(pkg)
+
+ def mark_by_prio(self, priority):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.arch:
+ ver = self.depcache.get_candidate_ver(pkg)
+ if ver and ver.priority <= priority:
+ self.depcache.mark_install(pkg)
+
+ def mark_pkg(self, pkgname, crossbuild):
+ if pkgname in self.cache:
+ pkg = self.cache[pkgname]
+
+ if not crossbuild or ':' in pkgname or len(pkg.version_list) == 0:
+ if pkg.has_provides and not pkg.has_versions:
+ print("pkgname is virtual package, selecting best provide")
+ # Select first provide
+ pkg_provide = pkg.provides_list[0][2]
+ # Find better provide with higher version
+ for provide in pkg.provides_list:
+ if apt_pkg.version_compare(provide[2].ver_str, pkg_provide.ver_str) > 0:
+ pkg_provide = provide[2]
+ self.depcache.mark_install(pkg_provide.parent_pkg)
+ else:
+ self.depcache.mark_install(pkg)
+ else:
+ version = pkg.version_list[0]
+ if version.arch == "all":
+ self.depcache.mark_install(pkg)
+ else:
+ if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+ if (pkgname, self.arch) in self.cache:
+ nativepkg = self.cache[pkgname, self.arch]
+ self.depcache.mark_install(nativepkg)
+ else:
+ if (pkgname, self.crossarch) in self.cache:
+ crosspkg = self.cache[pkgname, self.crossarch]
+ self.depcache.mark_install(crosspkg)
+
+ def mark_list(self, pkglist, crossbuild):
+ if pkglist:
+ for pkgname in pkglist:
+ self.mark_pkg(pkgname, crossbuild)
+
+ def handle_deb(self, item):
+ lockfd = open(self.repo + '/../repo.lock', 'w')
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", self.repodb,
+ "--outdir", self.repo,
+ "--confdir", self.repo + "/conf",
+ "-C", "main",
+ "includedeb",
+ self.codename,
+ item.destfile
+ ])
+ lockfd.close()
+
+ def handle_repo(self, fetcher):
+ lockfd = open(self.workdir + "/../apt_cache/" + self.distro + "-" + self.codename + ".lock", "w")
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ fetcher.run()
+ lockfd.close()
+ for item in fetcher.items:
+ if item.status == item.STAT_ERROR:
+ print("Some error ocured: '%s'" % item.error_text)
+ pass
+ else:
+ self.handle_deb(item)
+
+ def get_filename(self, uri):
+ path = urllib.parse.urlparse(uri).path
+ unquoted_path = urllib.parse.unquote(path)
+ basename = os.path.basename(unquoted_path)
+ return basename
+
+ def download_file(self, uri):
+ filename = self.get_filename(uri)
+ subprocess.run([
+ "wget",
+ "-H",
+ "--timeout=30",
+ "--tries=3",
+ "-q",
+ uri,
+ "-O",
+ self.workdir + "/tmp/" + filename
+ ])
+
+ def handle_dsc(self, uri):
+ filename = self.get_filename(uri)
+ lockfd = open(self.repo + '/../repo.lock', 'w')
+ fcntl.flock(lockfd,fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", self.repodb,
+ "--outdir", self.repo,
+ "--confdir", self.repo + "/conf",
+ "-C", "main",
+ "-S", "-", "-P" "source",
+ "--delete",
+ "includedsc",
+ self.codename,
+ os.path.realpath(self.workdir + "/tmp/" + filename)
+ ])
+ lockfd.close()
+
+ def handle_src_list(self, pkgs):
+ if pkgs:
+ for pkg in pkgs:
+ pkgname=pkg
+ pkgver=""
+ if '=' in pkg:
+ pkgname = pkg.split("=")[0]
+ pkgver = pkg.split("=")[1]
+
+ self.sr.restart()
+ while self.sr.lookup(pkgname):
+ if pkgver and pkgver != self.sr.version:
+ continue
+
+ for sr_file in self.sr.files:
+ print(self.sr.index.archive_uri(sr_file[2]))
+ self.download_file(self.sr.index.archive_uri(sr_file[2]))
+
+ dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2])
+ self.handle_dsc(dsc_uri)
+ break
+
+ def apt_run(self, init, srcmode, pkgs, controlfile, crossbuild):
+ apt_pkg.init()
+
+ sources = apt_pkg.SourceList()
+ sources.read_main_list()
+
+ progress = apt.progress.text.AcquireProgress()
+
+ self.cache = apt_pkg.Cache()
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+
+ self.depcache = apt_pkg.DepCache(self.cache)
+ self.sr = apt_pkg.SourceRecords()
+
+ if init:
+ self.mark_essential()
+ # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+ self.mark_by_prio(1)
+
+ if srcmode:
+ self.handle_src_list(pkgs)
+ else:
+ self.mark_list(pkgs, crossbuild)
+
+ if controlfile:
+ fobj = open(controlfile, "r")
+
+ try:
+ tagfile = apt_pkg.TagFile(fobj)
+ while tagfile.step() == 1:
+ deps = tagfile.section.get("Build-Depends", "")
+ # Remove extra commas and spaces - apt_pkg.parse_depends doesnt like
+ # lines like ", device-tree-compiler"
+ deps = ', '.join([s.strip() for s in deps.split(',') if s.strip()])
+ print("parsed deps: " + str(deps))
+ for item in apt_pkg.parse_depends(deps, False):
+ pkgname = item[0][0]
+ self.mark_pkg(pkgname, crossbuild)
+
+ finally:
+ fobj.close()
+
+ if init or not srcmode:
+ fetcher = apt_pkg.Acquire(progress)
+ pm = apt_pkg.PackageManager(self.depcache)
+
+ recs = apt_pkg.PackageRecords(self.cache)
+ pm.get_archives(fetcher, sources, recs)
+
+ self.handle_repo(fetcher)
+
+ def load_opts(self):
+ params = {}
+ if os.path.isfile(self.optsfile):
+ with open(self.optsfile, 'rb') as file:
+ data = file.read()
+ if data:
+ params = pickle.loads(data)
+
+ return params
+
+ def save_opts(self, opts):
+ file = open(self.optsfile, 'wb')
+ pickle.dump(opts, file)
+ file.close()
+
+
+class DebRepoArgs(object):
+ def __init__(self):
+ self.workdir = ""
+ self.init = False
+ self.srcmode = False
+ self.controlfile = ""
+ self.aptsrcsfile = ""
+ self.crossbuild = False
+
+ self.opts = {}
+ self.pkgs = []
+
+ try:
+ opts, args = getopt.getopt(sys.argv[1:], "", [
+ "init",
+ "srcmode",
+ "workdir=",
+ "repodir=",
+ "repodbdir=",
+ "mirror=",
+ "arch=",
+ "crossarch=",
+ "compatarch=",
+ "distro=",
+ "codename=",
+ "keydir=",
+ "isaraptdir=",
+ "no-check-gpg",
+ "controlfile=",
+ "aptsrcsfile=",
+ "crossbuild"
+ ])
+ except getopt.GetoptError as msg:
+ print("Error: " + str(msg))
+ sys.exit(1)
+
+ for opt, arg in opts:
+ if opt in ("--workdir"):
+ self.workdir = arg
+ if opt in ("--init"):
+ self.init = True
+ if opt in ("--srcmode"):
+ self.srcmode = True
+ if opt in ("--controlfile"):
+ self.controlfile = arg
+ if opt in ("--aptsrcsfile"):
+ self.aptsrcsfile = arg
+ if opt in ("--crossbuild"):
+ self.crossbuild = True
+
+ if opt in ("--repodir",
+ "--repodbdir",
+ "--mirror",
+ "--arch",
+ "--crossarch",
+ "--compatarch",
+ "--distro",
+ "--codename",
+ "--keydir",
+ "--isaraptdir",
+ "--controlfile",
+ ):
+ self.opts[opt[2:]] = arg
+ if opt in ("--no-check-gpg"):
+ self.opts['check_gpg'] = False
+
+ if not self.workdir:
+ print("Error: workdir is not specified")
+ sys.exit(1)
+
+ self.pkgs = args
+
+
+def main():
+ args = DebRepoArgs()
+
+ if not (args.init or args.pkgs or args.controlfile):
+ print("Nothing to do")
+ sys.exit(0)
+
+ if not os.path.exists(args.workdir):
+ os.makedirs(args.workdir)
+
+ debrepo = DebRepo(args.workdir, args.opts)
+
+ if (args.init):
+ debrepo.create_rootfs(args.aptsrcsfile)
+
+ debrepo.apt_config(args.init, args.crossbuild)
+ debrepo.apt_run(args.init, args.srcmode, args.pkgs, args.controlfile, args.crossbuild)
+
+
+if __name__ == "__main__":
+ main()
--
2.20.1

[Uladzimir Bely]

This is mostly related to gitlab CI that migth use an image without
preinstalled python3-apt.

Also, make system python packages available in virtualenv.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

scripts/ci_build.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 2fa5cccd..ff7c114a 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
sudo apt-get update -qq
sudo apt-get install -y virtualenv
rm -rf /tmp/avocado_venv
- virtualenv --python python3 /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv --system-site-packages
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install avocado-framework==100.1
@@ -127,6 +127,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
fi
fi

+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+ sudo apt-get update -qq
+ sudo apt-get install -y python3-apt
+fi
+
# Provide working path
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
--
2.20.1

[Uladzimir Bely]

This class uses 'scripts/debrepo' python script to prefetch given
packages or sources to local base-apt repository.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/debrepo.bbclass | 86 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 5 +++
2 files changed, 91 insertions(+)
create mode 100644 meta/classes/debrepo.bbclass

diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..8f2a3215
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,86 @@
+DEBREPO_WORKDIR ?= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ chroot_dir=${1}
+ apt_list=${2}
+
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+ -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+ -o Dir::Etc::SourceParts=\"-\" \
+ -o APT::Get::List-Cleanup=\"0\"
+ "
+}
+
+debrepo_add_packages() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ return
+ fi
+
+ args=""
+ if [ "${1}" = "--srcmode" ]; then
+ args="${args} --srcmode"
+ shift
+ fi
+
+ workdir="${1}"
+ pkgs="${2}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${workdir}/repo.lock" -c "
+ ${SCRIPTSDIR}/debrepo \
+ ${args} \
+ --workdir=\"${workdir}\" \
+ ${pkgs}
+ "
+}
+
+debrepo_handle_controlfile() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ return
+ fi
+
+ control_file="${1}"
+
+ args=""
+ pkgs=""
+
+ build_arch=${DISTRO_ARCH}
+ if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+ build_arch=${HOST_ARCH}
+ fi
+ if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+ args="--crossbuild"
+ pkgs="${pkgs} crossbuild-essential-${PACKAGE_ARCH}:${build_arch}"
+ pkgs="${pkgs} dose-distcheck:${build_arch}"
+ pkgs="${pkgs} libc-dev:${PACKAGE_ARCH}"
+ pkgs="${pkgs} libstdc++-dev:${PACKAGE_ARCH}"
+ fi
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "
+ ${SCRIPTSDIR}/debrepo \
+ --workdir=\"${DEBREPO_WORKDIR}\" \
+ --controlfile=\"${control_file}\" \
+ ${args} \
+ ${pkgs}
+ "
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 89b3e6a3..91f81bd4 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -71,6 +71,11 @@ KERNEL_FILE:mipsel ?= "vmlinux"
KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"

+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
OVERRIDES = "${PACKAGE_ARCH}:${MACHINE}:${DISTRO}:${BASE_DISTRO_CODENAME}:forcevariable"
FILESOVERRIDES = "${PACKAGE_ARCH}:${MACHINE}"

--
2.20.1

[Uladzimir Bely]

This means only local URLs in apt sources.list* are present during
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/buildchroot.bbclass | 5 +++--
meta/classes/rootfs.bbclass | 5 +++--
.../isar-bootstrap/isar-bootstrap.inc | 12 +++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++-------
4 files changed, 28 insertions(+), 15 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index e4a7a571..947529b0 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -38,8 +38,9 @@ buildchroot_do_mounts() {
mount --rbind /sys '${BUILDCHROOT_DIR}/sys'
mount --make-rslave '${BUILDCHROOT_DIR}/sys'

- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${BUILDCHROOT_DIR}/base-apt'
mountpoint -q '${BUILDCHROOT_DIR}/base-apt' || \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 22449d71..cedde239 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -54,8 +54,9 @@ rootfs_do_mounts() {
mount --bind '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
fi

- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${ROOTFSDIR}/base-apt'
mountpoint -q '${ROOTFSDIR}/base-apt' || \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index b94ae0bd..0cfe82ab 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -42,7 +42,8 @@ python () {
# installation afterwards. However, debootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.

- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
own_pub_key = d.getVar("BASE_REPO_KEY")
if own_pub_key:
distro_bootstrap_keys += own_pub_key.split()
@@ -110,7 +111,8 @@ def parse_aptsources_list_line(source_list_line):
def get_apt_source_mirror(d, aptsources_entry_list):
import re

- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
else:
premirrors = d.getVar('DISTRO_APT_PREMIRRORS') or ""
@@ -286,7 +288,8 @@ do_bootstrap() {
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
debootstrap_args="$debootstrap_args --no-check-gpg"
fi
E="${@ isar_export_proxies(d)}"
@@ -313,7 +316,8 @@ do_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
if [ -z "${BASE_REPO_KEY}" ]; then
line="[trusted=yes] ${line}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index ea885fe6..e62958d1 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -57,9 +57,12 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ fi

if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -67,14 +70,18 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${HOST_BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${HOST_BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ fi
fi
}

python do_cache() {
- if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+ not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
return 0

for key in d.getVar('BASE_REPO_KEY').split():
--
2.20.1

[Uladzimir Bely]

This patch makes local base-apt repo to be created before
debootstrap task. So, debootstrap is then done from it.

The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.

For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.

The goal is have workable base-apt repo before first build completed.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

.../isar-bootstrap/isar-bootstrap.inc | 99 +++++++++++++++++--
1 file changed, 91 insertions(+), 8 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 0cfe82ab..e1ddac42 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -29,6 +29,8 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+
FILESEXTRAPATHS:append = ":${BBPATH}"

inherit deb-dl-dir
@@ -278,12 +280,73 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"

inherit compat

+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+ if [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ return
+ fi
+
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

+ return
+ fi
+

+ debrepo_args=""
+ if [ "${BASE_DISTRO}" != "debian" ]; then
+ if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ else
+ if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ fi
+ #if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ debrepo_args="$debrepo_args --no-check-gpg"
+ #fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+ fi
+
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+ debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
+ else
+ debrepo_args="$debrepo_args --isaraptdir=${REPO_ISAR_DIR}/${DISTRO}"
+ fi
+ mkdir -p "${DEBREPO_WORKDIR}"

+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+
+ flock -x "${DEBREPO_WORKDIR}/repo.lock" -c "

+ ${SCRIPTSDIR}/debrepo --init \
+ --workdir=\"${debrepo_workdir}\" \
+ --aptsrcsfile=\"${APTSRCS_INIT}\" \
+ --repodir=\"${REPO_BASE_DIR}\" \
+ --repodbdir=\"${REPO_BASE_DB_DIR}\" \
+ --mirror=\"${@get_distro_source(d)}\" \
+ --arch=\"${BOOTSTRAP_DISTRO_ARCH}\" \
+ --distro=\"${BOOTSTRAP_BASE_DISTRO}\" \
+ --codename=\"${BASE_DISTRO_CODENAME}\" \
+ ${debrepo_args} \
+ gnupg locales usr-is-merged
+ "
+}
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
+
+ debrepo_bootstrap_prepare
+
debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"

if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"

@@ -301,8 +364,19 @@ do_bootstrap() {
sudo -E -s <<'EOSUDO'
set -e
if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}"
fi
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then

+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "

+ ${DEBOOTSTRAP} $debootstrap_args \
+ $arch_param \
+ ${@get_distro_components_argument(d)} \
+ ${@get_distro_suite(d)} \
+ ${ROOTFSDIR} \
+ file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ "
+ else
${DEBOOTSTRAP} $debootstrap_args \
$arch_param \
${@get_distro_components_argument(d)} \
@@ -310,7 +384,7 @@ do_bootstrap() {
"${ROOTFSDIR}" \
"${@get_distro_source(d)}" \
${DISTRO_DEBOOTSTRAP_SCRIPT}
-
+ fi
# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"

install -v -m644 "${APTPREFS}" \

@@ -382,12 +456,21 @@ do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
fi
-
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
- -o APT::Update::Error-Mode=any
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then

+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "

+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ "
+ else
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ fi

umount -l "${ROOTFSDIR}/dev/shm"
umount -l "${ROOTFSDIR}/dev/pts"
--
2.20.1

[Uladzimir Bely]

For host/target and cross/native modes different debrepo context
(e.g., working dir) should be used.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 11 +++++++++++
meta/recipes-devtools/buildchroot/buildchroot.inc | 11 +++++++++++
meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc | 11 +++++++++++
3 files changed, 33 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index e3412c82..c3e47a69 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -12,6 +12,17 @@ inherit repository
inherit deb-dl-dir
inherit essential

+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ cross = d.getVar('ISAR_CROSS_COMPILE', True)
+
+ if cross == "0" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index f74896fb..320df55a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -13,6 +13,17 @@ SRC_URI = "file://configscript.sh \
file://deps.sh"
PV = "1.0"

+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ variant = d.getVar('BUILDCHROOT_VARIANT', True)
+
+ if variant == "target" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
inherit rootfs

BUILDCHROOT_DIR = "${WORKDIR}/rootfs"
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
index fd8bb648..03f812c9 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot.inc
@@ -8,6 +8,17 @@ LIC_FILES_CHKSUM = "file://${LAYERDIR_core}/licenses/COPYING.GPLv2;md5=751419260

PV = "1.0"

+python __anonymous() {
+ distro_arch = d.getVar('DISTRO_ARCH')
+ host_arch = d.getVar('HOST_ARCH')
+ variant = d.getVar('SBUILD_VARIANT', True)
+
+ if variant == "target" or host_arch == distro_arch:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR', True))
+ else:
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
+}
+
inherit rootfs
inherit compat

--
2.20.1

[Uladzimir Bely]

This makes Isar use `base-apt` repo in different way. Any package
installation is done from `base-apt` repo, which is prepopulated
from the external mirrors.

This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

RECIPE-API-CHANGELOG.md | 9 +++++++++
meta-isar/conf/local.conf.sample | 4 ++++
2 files changed, 13 insertions(+)

diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 7a16b5c2..ae845674 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -492,3 +492,12 @@ Recipes inheriting dpkg-base now automatically have a bitbake target
exists for the current `DISTRO_ARCH`.
In that case the compat package can be built by adding `<foo>-compat`
to `DEPENDS` or `IMAGE_INSTALL`.
+
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before any step they are requested.
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 7e0184e4..2d3c9c1e 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -215,6 +215,10 @@ ISAR_CROSS_COMPILE ?= "0"
# does not access the network
#BB_NO_NETWORK ?= "1"

+# Comment out for switching to old base-apt behaviour when it's populated
+# at the beginnig of the second build
+ISAR_PREFETCH_BASE_APT ?= "1"
+
# Set root password to 'root'
# Password was encrypted using following command:
# mkpasswd -m sha512crypt -R 10000
--
2.20.1

[Uladzimir Bely]

Fill base-apt repo with source packages.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/deb-dl-dir.bbclass | 17 +++++++++++++++++
meta/classes/rootfs.bbclass | 4 ++++
2 files changed, 21 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index f0ab9714..a7a8dcf7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -43,6 +43,23 @@ debsrc_undo_mounts() {
EOSUDO
}

+debsrc_fill_base_apt() {
+ export rootfs="$1"
+
+ local srcpkgs=""
+ for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do
+ is_not_part_of_current_build "${package}" && continue
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+ local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
+ [ -n "$dscfile" ] && continue
+
+ srcpkgs="${srcpkgs} ${src}=${version}"
+ done
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}"
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 47f07129..0ada4721 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -230,8 +230,12 @@ cache_deb_src() {
sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ debsrc_fill_base_apt ${ROOTFSDIR}
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}

+
sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ] ||
[ -h "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
--
2.20.1

[Uladzimir Bely]

When using python-apt, it can't mark it for downloading when used
without ":<arch>" field, so we miss the package in base-apt repo.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/conf/distro/debian-common.conf | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/meta/conf/distro/debian-common.conf b/meta/conf/distro/debian-common.conf
index 5610dcd4..ee84b7dd 100644
--- a/meta/conf/distro/debian-common.conf
+++ b/meta/conf/distro/debian-common.conf
@@ -23,10 +23,10 @@ WIC_IMAGER_INSTALL = "parted \
tar \
fdisk"

-GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin"
-GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin"
-GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin"
-GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin"
+GRUB_BOOTLOADER_INSTALL:amd64 = "grub-efi-amd64-bin:amd64"
+GRUB_BOOTLOADER_INSTALL:i386 = "grub-efi-ia32-bin:i386"
+GRUB_BOOTLOADER_INSTALL:armhf = "grub-efi-arm-bin:armhf"
+GRUB_BOOTLOADER_INSTALL:arm64 = "grub-efi-arm64-bin:arm64"

GRUB_DEBIAN_SB_CHAIN:amd64 = "grub-efi-amd64-signed shim-signed"
GRUB_DEBIAN_SB_MOK:amd64 = "shim-helpers-amd64-signed"
--
2.20.1

[Uladzimir Bely]

This patch uses debrepo script to predownload packages to base-apt
repository before they are installed in buildchroot/rootfs.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/dpkg-base.bbclass | 36 ++++++++++++++++++--
meta/classes/dpkg.bbclass | 8 +++++
meta/classes/image-locales-extension.bbclass | 5 +++
meta/classes/image-tools-extension.bbclass | 9 +++++
meta/classes/rootfs.bbclass | 4 +++
5 files changed, 60 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c3e47a69..faf2ec77 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -23,6 +23,8 @@ python __anonymous() {

d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR', True))
}

+inherit debrepo

+
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"

@@ -126,10 +128,25 @@ do_apt_fetch() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT

+ # Begin chroot session
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}"

+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "

+ schroot -d / -r -c ${session_id} -u root -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
for uri in "${SRC_APT}"; do
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ schroot -d / -r -c ${session_id} -- \
sh -c 'mkdir -p /downloads/deb-src/"$1"/"$2" && cd /downloads/deb-src/"$1"/"$2" && apt-get -y --download-only --only-source source "$2"' my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
done
+
+ # End chroot session
+ schroot -e -c ${session_id}
+
schroot_delete_configs
}

@@ -139,6 +156,8 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"

# Add dependency from the correct schroot: host or target
do_apt_fetch[depends] += "${SCHROOT_DEP}"
+# Debrepo context is created by target bootstrap, need this dependency too
+do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap"

do_apt_unpack() {
rm -rf ${S}
@@ -150,8 +169,18 @@ do_apt_unpack() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT

+ # Begin chroot session
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+

+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "

+ schroot -d / -r -c ${session_id} -u root -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
for uri in "${SRC_APT}"; do
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ schroot -d / -r -c ${session_id} -- \
sh -c ' \
set -e
dscfile="$(apt-get -y -qq --print-uris --only-source source "${2}" | cut -d " " -f2 | grep -E "*.dsc")"
@@ -160,6 +189,9 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"' \
my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${uri}"
done
+
+ # End chroot session
+ schroot -e -c ${session_id}
schroot_delete_configs
}
do_apt_unpack[network] = "${TASK_USE_SUDO}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 5c8996d6..ea26b530 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -32,6 +32,7 @@ dpkg_runbuild() {
export PARALLEL_MAKE="${PARALLEL_MAKE}"

rm -f ${SBUILD_CONFIG}
+ debrepo_handle_controlfile "${WORKDIR}/${PPS}/debian/control"

env | while read -r line; do
# Filter the same lines
@@ -101,6 +102,11 @@ dpkg_runbuild() {
sh -c "cd ${WORKDIR}; dpkg-source -q -b ${PPS}"
DSC_FILE=$(find ${WORKDIR} -name "${DEB_SOURCE_NAME}*.dsc" -print)

+ locked_update_cmd=":"

+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then

+ locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'"
+ fi
+
sbuild -A -n -c ${SBUILD_CHROOT} --extra-repository="${ISAR_APT_REPO}" \
--host=${PACKAGE_ARCH} --build=${BUILD_HOST_ARCH} ${profiles} \
--no-run-lintian --no-run-piuparts --no-run-autopkgtest --resolve-alternatives \
@@ -109,8 +115,10 @@ dpkg_runbuild() {
--chroot-setup-commands="echo \"APT::Get::allow-downgrades 1;\" > /etc/apt/apt.conf.d/50isar-apt" \
--chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="ln -sf ${ext_deb_dir}/*.deb -t ${deb_dir}/ || :" \
+ --chroot-setup-commands="${locked_update_cmd}" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="cp -Ln --no-preserve=owner ${deb_dir}/*.deb -t ${ext_deb_dir}/ || :" \
+ --finished-build-commands="mkdir -p ${ext_root}" \
--finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--debbuildopts="--source-option=-I" \
--build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index b48bcec8..aa4874c1 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
# This class extends the image.bbclass for setting locales and purging unneeded
# ones.

+inherit debrepo
+
LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
en_US ISO-8859-1\n"
LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
image_install_localepurge_download[weight] = "40"
image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
image_install_localepurge_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"

+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"

+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
}
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 7c84505f..f98cd759 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -6,6 +6,7 @@
# This file extends the image.bbclass to supply tools for futher imager functions

inherit buildchroot
+inherit debrepo

IMAGER_INSTALL ??= ""
IMAGER_BUILD_DEPS ??= ""
@@ -21,14 +22,22 @@ do_install_imager_deps() {
fi

distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"

+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi

fi

buildchroot_do_mounts

E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${BUILDCHROOT_DIR} ${distro}
+
+ debrepo_add_packages "${debrepo_workdir}" "${IMAGER_INSTALL}"
+ debrepo_update_apt_source_list "${BUILDCHROOT_DIR}" "base-apt"
+
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get update \
-o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cedde239..47f07129 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -2,6 +2,7 @@
# Copyright (c) Siemens AG, 2020

inherit deb-dl-dir
+inherit debrepo

ROOTFS_ARCH ?= "${DISTRO_ARCH}"
ROOTFS_DISTRO ?= "${DISTRO}"
@@ -148,6 +149,9 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
rootfs_install_pkgs_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"

+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"

+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}
--
2.20.1

[Uladzimir Bely]

Since all packages and source packages are placed to base-apt repo
during build, there is no need to have them in one more place.

Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---

meta/classes/deb-dl-dir.bbclass | 8 ++++++++
1 file changed, 8 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index a7a8dcf7..54110cdb 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -87,6 +87,10 @@ debsrc_download() {
}

deb_dl_dir_import() {

+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then

+ return
+ fi
+
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -104,6 +108,10 @@ EOSUDO
}

deb_dl_dir_export() {

+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then

+ return
+ fi
+
export pc="${DEBDIR}/${2}"
export rootfs="${1}"
export owner=$(id -u):$(id -g)
--
2.20.1