[PATCH v7 00/10] Improving base-apt usage
2 views
Skip to first unread message
Uladzimir Bely
Jul 25, 2024, 11:10:15 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
`base-apt` is a local apt repository containing all upstream (Debian, Raspberry Pi OS, Ubuntu...) packages needed for a particular build. This series implements upfront repository downloading. This is the first step towards local partial mirror management.
The current approach in `next`:
- On the first build, debootstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories.
- On the next build:
- Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`.
- Use `base-apt` for bootstrapping, building and image creation.
Some issues with the current approach:
1. Different policies must be followed for the first and the subsequent builds.
2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`.
3. We rely on internal implementation of `debootstrap` and `sbuild` for saving and reusing the packages. Changing to e.g. `mmdebstrap` breaks the unrelated `base-apt` functionality.
4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible.
5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case).
6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary.
The new approach:
- On the first build:
- All packages necessary for bootstrapping and building are identified and downloaded upfront.
- `base-apt` is used for bootstrapping, building and image creation.
- On the next build:
- `base-apt` is used for bootstrapping, building and image creation.
This series addresses issues 1-5 and provides the architecture for implementing further use cases.
The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach.
The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode:
```
# Create local `/build/ba` repository sufficient to debootstrap Debian system with `armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg
# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```
Changes since v6:
- Rebased on latest next.
- Fixed possible stuck when `cache-deb-src` feature was enabled.
- Fixed support of debian build profiles.
- Moved locking of `debrepo` context from the bbclass to the script itself. This makes usage from Isar simpler.
Changes since v5:
- Rebased on latest next.
- Changed order of the patches.
- Fixes in `debrepo` script that allow to use it outside Isar in standalone mode.
Changes since v4:
- Rebased on latest next.
- Rearranged patches since some of them are already in next.
- Added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes.
- Code passes both full and fast CI in both "old" and "new" modes.
Changes since v3:
- Rebased on latest next.
- Cross-building for raspberry supported.
- Code passes both full and fast CI.
Changes since v2:
- Populate base-apt before using at all steps of native build.
Changes since v1:
- Rebased on latest next.
- Updated patchset description.
Uladzimir Bely (10):
scripts: Add debrepo python script handling base-apt
meta: Add debrepo bbclass handling base-apt prefetching
meta: Always use base-apt repo in local mode
meta: Use cached base-apt repo to debootstrap
base-apt: Predownload packages to base-apt before install
meta: Add cache-deb-src functionality in base-apt mode
testsuite: Set ISAR_PREFETCH_BASE_APT by default
Disable deb-dl-dir in base-apt prefetch mode
kas: Add PREFETCH_BASE_APT config entry
ci_build.sh: Install python3-apt if not installed
RECIPE-API-CHANGELOG.md | 10 +
kas/opt/Kconfig | 13 +
kas/opt/prefetch-base-apt.yaml | 9 +
meta-test/conf/local.conf.sample | 3 +
meta/classes/crossvars.bbclass | 1 +
meta/classes/deb-dl-dir.bbclass | 21 +
meta/classes/debrepo.bbclass | 90 +++
meta/classes/dpkg-base.bbclass | 27 +-
meta/classes/dpkg.bbclass | 8 +
meta/classes/image-locales-extension.bbclass | 5 +
meta/classes/image-tools-extension.bbclass | 13 +
meta/classes/rootfs.bbclass | 12 +-
meta/conf/bitbake.conf | 5 +
.../isar-bootstrap/isar-bootstrap-host.bb | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 99 ++-
meta/recipes-devtools/base-apt/base-apt.bb | 21 +-
.../sbuild-chroot/sbuild-chroot-host.bb | 2 +
scripts/ci_build.sh | 8 +-
scripts/debrepo | 590 ++++++++++++++++++
testsuite/cibase.py | 4 +
testsuite/cibuilder.py | 8 +-
21 files changed, 927 insertions(+), 24 deletions(-)
create mode 100644 kas/opt/prefetch-base-apt.yaml
create mode 100644 meta/classes/debrepo.bbclass
create mode 100755 scripts/debrepo
--
2.44.2
The current approach in `next`:
- On the first build, debootstrap and sbuild are used for building Isar artifacts. The packages downloaded from the Internet are cached in local directories.
- On the next build:
- Analyze the logs from the previous build, save packages downloaded by the bootstraps, sbuilds and imagers into `base-apt`.
- Use `base-apt` for bootstrapping, building and image creation.
Some issues with the current approach:
1. Different policies must be followed for the first and the subsequent builds.
2. As we have multiple versions of the same package from the main and security repositories and rely on build logs and `find` for populating `base-apt`, extra care must be taken to ensure that the right package version lands in `base-apt`.
3. We rely on internal implementation of `debootstrap` and `sbuild` for saving and reusing the packages. Changing to e.g. `mmdebstrap` breaks the unrelated `base-apt` functionality.
4. Source packages are stored in a different flat directory, `apt-get source` for upstream packages is not possible.
5. At the moment of `base-apt` creation all we have is the package name. The knowledge about the upstream repositories is lost and no local repository policy implementation is possible (e.g., for the "multiple products, multiple distros" use case).
6. For implementing further use cases like "fetch all sources necessary for bootstrapping the base system itself", additional logic is necessary.
The new approach:
- On the first build:
- All packages necessary for bootstrapping and building are identified and downloaded upfront.
- `base-apt` is used for bootstrapping, building and image creation.
- On the next build:
- `base-apt` is used for bootstrapping, building and image creation.
This series addresses issues 1-5 and provides the architecture for implementing further use cases.
The new approach is enabled by default. Setting `ISAR_PREFETCH_BASE_APT` to zero falls back to the old approach.
The implementation uses `debrepo` script which can also be called manually for pre-fetching packages to the local 'base-apt' repository. It requires `python3-apt` to be installed on the build host. Some examples of its usage in standalone mode:
```
# Create local `/build/ba` repository sufficient to debootstrap Debian system with `armhf` architecture:
debrepo --init --workdir=/build/dr --repodir=/build/ba --arch=armhf
# Add some packages to this repo (e.g., build deps for some recipe):
debrepo --workdir=/build/dr locales gnupg
# Add srcpackages for some package to the repo:
debrepo --workdir=/build/dr --srcmode tzdata
```
Changes since v6:
- Rebased on latest next.
- Fixed possible stuck when `cache-deb-src` feature was enabled.
- Fixed support of debian build profiles.
- Moved locking of `debrepo` context from the bbclass to the script itself. This makes usage from Isar simpler.
Changes since v5:
- Rebased on latest next.
- Changed order of the patches.
- Fixes in `debrepo` script that allow to use it outside Isar in standalone mode.
Changes since v4:
- Rebased on latest next.
- Rearranged patches since some of them are already in next.
- Added possibility to select between new "prefetch" base-apt mode and old behaviour when it's populated on 2nd build with packages downloaded during 1st build. New behaviour is disabled by default, but enabled in local.conf.example for testing purposes.
- Code passes both full and fast CI in both "old" and "new" modes.
Changes since v3:
- Rebased on latest next.
- Cross-building for raspberry supported.
- Code passes both full and fast CI.
Changes since v2:
- Populate base-apt before using at all steps of native build.
Changes since v1:
- Rebased on latest next.
- Updated patchset description.
Uladzimir Bely (10):
scripts: Add debrepo python script handling base-apt
meta: Add debrepo bbclass handling base-apt prefetching
meta: Always use base-apt repo in local mode
meta: Use cached base-apt repo to debootstrap
base-apt: Predownload packages to base-apt before install
meta: Add cache-deb-src functionality in base-apt mode
testsuite: Set ISAR_PREFETCH_BASE_APT by default
Disable deb-dl-dir in base-apt prefetch mode
kas: Add PREFETCH_BASE_APT config entry
ci_build.sh: Install python3-apt if not installed
RECIPE-API-CHANGELOG.md | 10 +
kas/opt/Kconfig | 13 +
kas/opt/prefetch-base-apt.yaml | 9 +
meta-test/conf/local.conf.sample | 3 +
meta/classes/crossvars.bbclass | 1 +
meta/classes/deb-dl-dir.bbclass | 21 +
meta/classes/debrepo.bbclass | 90 +++
meta/classes/dpkg-base.bbclass | 27 +-
meta/classes/dpkg.bbclass | 8 +
meta/classes/image-locales-extension.bbclass | 5 +
meta/classes/image-tools-extension.bbclass | 13 +
meta/classes/rootfs.bbclass | 12 +-
meta/conf/bitbake.conf | 5 +
.../isar-bootstrap/isar-bootstrap-host.bb | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 99 ++-
meta/recipes-devtools/base-apt/base-apt.bb | 21 +-
.../sbuild-chroot/sbuild-chroot-host.bb | 2 +
scripts/ci_build.sh | 8 +-
scripts/debrepo | 590 ++++++++++++++++++
testsuite/cibase.py | 4 +
testsuite/cibuilder.py | 8 +-
21 files changed, 927 insertions(+), 24 deletions(-)
create mode 100644 kas/opt/prefetch-base-apt.yaml
create mode 100644 meta/classes/debrepo.bbclass
create mode 100755 scripts/debrepo
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:17 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This class uses 'scripts/debrepo' python script to prefetch given
packages or sources to local base-apt repository.
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 10 ++++
meta/classes/debrepo.bbclass | 90 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 5 ++
3 files changed, 105 insertions(+)
create mode 100644 meta/classes/debrepo.bbclass
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 12ea93ec..68204a96 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -629,3 +629,13 @@ into kernel kbuild package.
Only the "host" specific package is built automatically at cross builds.
* Support emulated module build with cross-compiled kernel for linux-module
+
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before running any task that need them. Separate
+`debrepo` script is used for populating base-apt repo.
diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..003c4e03
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,90 @@
+# This software is a part of Isar.
+# Copyright (C) 2024 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+# Prefetch to base-apt repo by default
+ISAR_PREFETCH_BASE_APT ??= "1"
+
+DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+
+ chroot_dir=${1}
+ apt_list=${2}
+
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+ -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+ -o Dir::Etc::SourceParts=\"-\" \
+ -o APT::Get::List-Cleanup=\"0\"
+ "
+}
+
+debrepo_add_packages() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ args=""
+ if [ "${1}" = "--srcmode" ]; then
+ args="${args} --srcmode"
+ shift
+ fi
+
+ if [ "${1}" = "--isarapt" ]; then
+ args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}"
+ shift
+ fi
+
+ workdir="${1}"
+ args="${args} ${2}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ ${SCRIPTSDIR}/debrepo \
+ --workdir="${workdir}" \
+ ${args}
+}
+
+debrepo_parse_dscfile() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ dscfile="${1}"
+ args=""
+
+ build_arch=${DISTRO_ARCH}
+ if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+ build_arch=${HOST_ARCH}
+ fi
+ if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+ args="--crossbuild \
+ crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \
+ dose-distcheck:${build_arch} \
+ libc-dev:${PACKAGE_ARCH} \
+ libstdc++-dev:${PACKAGE_ARCH} \
+ "
+ fi
+
+ args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ if [ -n "${DEB_BUILD_PROFILES}" ]; then
+ export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
+ fi
+
+ ${SCRIPTSDIR}/debrepo \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --dscfile="${dscfile}" \
+ ${args}
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 4cfa8b10..b0e33477 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux"
KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"
+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
MACHINEOVERRIDES ?= "${MACHINE}"
DISTROOVERRIDES ?= "${DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
--
2.44.2
packages or sources to local base-apt repository.
Signed-off-by: Uladzimir Bely <ub...@ilbers.de>
---
RECIPE-API-CHANGELOG.md | 10 ++++
meta/classes/debrepo.bbclass | 90 ++++++++++++++++++++++++++++++++++++
meta/conf/bitbake.conf | 5 ++
3 files changed, 105 insertions(+)
create mode 100644 meta/classes/debrepo.bbclass
diff --git a/RECIPE-API-CHANGELOG.md b/RECIPE-API-CHANGELOG.md
index 12ea93ec..68204a96 100644
--- a/RECIPE-API-CHANGELOG.md
+++ b/RECIPE-API-CHANGELOG.md
@@ -629,3 +629,13 @@ into kernel kbuild package.
Only the "host" specific package is built automatically at cross builds.
* Support emulated module build with cross-compiled kernel for linux-module
+
+### "Prefetch" mode for base-apt
+
+Originally, `base-apt` repo is created only during second build when variable
+ISAR_USE_CACHED_BASE_REPO is set. The repo is populated with every package that
+took part in the first build and was cached in DL_DIR.
+
+New ISAR_PREFETCH_BASE_APT variable changes the way `base-apt` is populated.
+Packages added to the repo before running any task that need them. Separate
+`debrepo` script is used for populating base-apt repo.
diff --git a/meta/classes/debrepo.bbclass b/meta/classes/debrepo.bbclass
new file mode 100644
index 00000000..003c4e03
--- /dev/null
+++ b/meta/classes/debrepo.bbclass
@@ -0,0 +1,90 @@
+# This software is a part of Isar.
+# Copyright (C) 2024 ilbers GmbH
+#
+# SPDX-License-Identifier: MIT
+
+# Prefetch to base-apt repo by default
+ISAR_PREFETCH_BASE_APT ??= "1"
+
+DEBREPO_WORKDIR ??= "${DEBREPO_TARGET_DIR}"
+
+debrepo_update_apt_source_list() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+
+ chroot_dir=${1}
+ apt_list=${2}
+
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sudo -E chroot ${chroot_dir} /usr/bin/apt-get update \
+ -o Dir::Etc::SourceList=\"sources.list.d/${apt_list}.list\" \
+ -o Dir::Etc::SourceParts=\"-\" \
+ -o APT::Get::List-Cleanup=\"0\"
+ "
+}
+
+debrepo_add_packages() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ args=""
+ if [ "${1}" = "--srcmode" ]; then
+ args="${args} --srcmode"
+ shift
+ fi
+
+ if [ "${1}" = "--isarapt" ]; then
+ args="${args} --extrarepo=${REPO_ISAR_DIR}/${DISTRO}"
+ shift
+ fi
+
+ workdir="${1}"
+ args="${args} ${2}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ ${SCRIPTSDIR}/debrepo \
+ --workdir="${workdir}" \
+ ${args}
+}
+
+debrepo_parse_dscfile() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ dscfile="${1}"
+ args=""
+
+ build_arch=${DISTRO_ARCH}
+ if [ "${ISAR_CROSS_COMPILE}" = "1" ]; then
+ build_arch=${HOST_ARCH}
+ fi
+ if [ "${PACKAGE_ARCH}" != "${build_arch}" ]; then
+ args="--crossbuild \
+ crossbuild-essential-${PACKAGE_ARCH}:${build_arch} \
+ dose-distcheck:${build_arch} \
+ libc-dev:${PACKAGE_ARCH} \
+ libstdc++-dev:${PACKAGE_ARCH} \
+ "
+ fi
+
+ args="${args} --extrarepo=${WORKDIR}/isar-apt/${DISTRO}-${DISTRO_ARCH}/apt/${DISTRO}"
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ else
+ export GNUPGHOME="${WORKDIR}/gpghome"
+ fi
+
+ if [ -n "${DEB_BUILD_PROFILES}" ]; then
+ export DEB_BUILD_PROFILES="${DEB_BUILD_PROFILES}"
+ fi
+
+ ${SCRIPTSDIR}/debrepo \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --dscfile="${dscfile}" \
+ ${args}
+}
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 4cfa8b10..b0e33477 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -69,6 +69,11 @@ KERNEL_FILE:mipsel ?= "vmlinux"
KERNEL_FILE:riscv64 ?= "vmlinux"
KERNEL_FILE:arm64 ?= "vmlinux"
+# debrepo config
+DEBREPO_DIR = "${TOPDIR}/debrepo"
+DEBREPO_HOST_DIR = "${DEBREPO_DIR}/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+DEBREPO_TARGET_DIR = "${DEBREPO_DIR}/${DISTRO}-${DISTRO_ARCH}"
+
MACHINEOVERRIDES ?= "${MACHINE}"
DISTROOVERRIDES ?= "${DISTRO}"
OVERRIDES = "${PACKAGE_ARCH}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${BASE_DISTRO_CODENAME}:forcevariable"
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:17 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This means only local URLs in apt sources.list* are present during
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.
meta/classes/rootfs.bbclass | 5 +++--
.../isar-bootstrap/isar-bootstrap.inc | 12 +++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++-------
3 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 2e091e0c..7d0bc0c8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -54,8 +54,9 @@ rootfs_do_mounts() {
mount -o bind,private '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
fi
- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${ROOTFSDIR}/base-apt'
mountpoint -q '${ROOTFSDIR}/base-apt' || \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 12f32ff0..9fd928ad 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -47,7 +47,8 @@ python () {
# installation afterwards. However, debootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
own_pub_key = d.getVar("BASE_REPO_KEY")
if own_pub_key:
distro_bootstrap_keys += own_pub_key.split()
@@ -121,7 +122,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
# this is executed during parsing. No error checking possible
use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
elif use_snapshot and snapshot_mirror:
premirrors = snapshot_mirror
@@ -299,7 +301,8 @@ do_bootstrap() {
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
debootstrap_args="$debootstrap_args --no-check-gpg"
fi
E="${@ isar_export_proxies(d)}"
@@ -326,7 +329,8 @@ do_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
if [ -z "${BASE_REPO_KEY}" ]; then
line="[trusted=yes] ${line}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 2766bc71..09a4509c 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -56,9 +56,12 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ fi
if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -66,14 +69,18 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${HOST_BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${HOST_BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ fi
fi
}
python do_cache() {
- if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+ not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
return 0
for key in d.getVar('BASE_REPO_KEY').split():
--
2.44.2
the build. Any installation of packages is done from local base-apt.
So, base-apt should be always mounted in *_do_mounts since now.
meta/classes/rootfs.bbclass | 5 +++--
.../isar-bootstrap/isar-bootstrap.inc | 12 +++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 21 ++++++++++++-------
3 files changed, 25 insertions(+), 13 deletions(-)
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 2e091e0c..7d0bc0c8 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -54,8 +54,9 @@ rootfs_do_mounts() {
mount -o bind,private '${REPO_ISAR_DIR}/${DISTRO}' '${ROOTFSDIR}/isar-apt'
fi
- # Mount base-apt if 'ISAR_USE_CACHED_BASE_REPO' is set
- if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
+ # Mount base-apt if 'ISAR_PREFETCH_BASE_APT' or 'ISAR_USE_CACHED_BASE_REPO' is set
+ if [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')))}" = 'True' ] || \
+ [ "${@repr(bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')))}" = 'True' ]
then
mkdir -p '${ROOTFSDIR}/base-apt'
mountpoint -q '${ROOTFSDIR}/base-apt' || \
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 12f32ff0..9fd928ad 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -47,7 +47,8 @@ python () {
# installation afterwards. However, debootstrap will include the key into
# the rootfs automatically thus the right place is distro_bootstrap_keys.
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
own_pub_key = d.getVar("BASE_REPO_KEY")
if own_pub_key:
distro_bootstrap_keys += own_pub_key.split()
@@ -121,7 +122,8 @@ def get_apt_source_mirror(d, aptsources_entry_list):
# this is executed during parsing. No error checking possible
use_snapshot = bb.utils.to_boolean(d.getVar('ISAR_USE_APT_SNAPSHOT'))
snapshot_mirror = d.getVar('DISTRO_APT_SNAPSHOT_PREMIRROR')
- if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) or \
+ bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) :
premirrors = "\S* file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO}\n"
elif use_snapshot and snapshot_mirror:
premirrors = snapshot_mirror
@@ -299,7 +301,8 @@ do_bootstrap() {
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
fi
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" -a -z "${BASE_REPO_KEY}" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" -a -z "${BASE_REPO_KEY}" ]; then
debootstrap_args="$debootstrap_args --no-check-gpg"
fi
E="${@ isar_export_proxies(d)}"
@@ -326,7 +329,8 @@ do_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] || \
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
line="file:///base-apt/${BOOTSTRAP_BASE_DISTRO} ${BASE_DISTRO_CODENAME} main"
if [ -z "${BASE_REPO_KEY}" ]; then
line="[trusted=yes] ${line}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 2766bc71..09a4509c 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -56,9 +56,12 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
+ fi
if [ '${BASE_DISTRO}' != '${HOST_BASE_DISTRO}' ]; then
repo_create "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
@@ -66,14 +69,18 @@ repo() {
"${BASE_DISTRO_CODENAME}" \
"${WORKDIR}/distributions.in" \
"${KEYFILES}"
- populate_base_apt "${HOST_BASE_DISTRO}"
- repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
- "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && \
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ]; then
+ populate_base_apt "${HOST_BASE_DISTRO}"
+ repo_sanity_test "${REPO_BASE_DIR}"/"${HOST_BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${HOST_BASE_DISTRO}"
+ fi
fi
}
python do_cache() {
- if not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
+ if not bb.utils.to_boolean(d.getVar('ISAR_PREFETCH_BASE_APT')) and \
+ not bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
return 0
for key in d.getVar('BASE_REPO_KEY').split():
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:17 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This patch makes local base-apt repo to be created before
debootstrap task. So, debootstrap is then done from it.
The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.
For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.
The goal is have workable base-apt repo before first build completed.
.../isar-bootstrap/isar-bootstrap-host.bb | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 87 +++++++++++++++++--
2 files changed, 81 insertions(+), 8 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 4f90fd01..1ace818f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -15,3 +15,5 @@ require isar-bootstrap.inc
HOST_DISTRO_BOOTSTRAP_KEYS ?= ""
DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}"
+
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 9fd928ad..589fa2c1 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -33,6 +33,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+
+# For newer distros "usr-is-merged" indirectly required by debootstrap
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged"
+
FILESEXTRAPATHS:append = ":${BBPATH}"
APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
@@ -291,12 +298,56 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
inherit compat
+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ debrepo_args=""
+ if [ "${BASE_DISTRO}" != "debian" ]; then
+ if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ else
+ if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+ fi
+
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+ debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+ fi
+
+
+ ${SCRIPTSDIR}/debrepo --init \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --aptsrcsfile="${APTSRCS_INIT}" \
+ --repodir="${REPO_BASE_DIR}" \
+ --repodbdir="${REPO_BASE_DB_DIR}" \
+ --mirror="${@get_distro_source(d)}" \
+ --arch="${BOOTSTRAP_DISTRO_ARCH}" \
+ --distro="${BOOTSTRAP_BASE_DISTRO}" \
+ --codename="${BASE_DISTRO_CODENAME}" \
+ ${debrepo_args} \
+ ${DISTRO_BOOTSTRAP_BASE_PACKAGES}
+}
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
+
+ debrepo_bootstrap_prepare
+
debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
sudo -E -s <<'EOSUDO'
set -e
if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}"
fi
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ $arch_param \
+ ${@get_distro_components_argument(d)} \
+ ${@get_distro_suite(d)} \
+ ${ROOTFSDIR} \
+ file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ "
+ else
${DEBOOTSTRAP} $debootstrap_args \
$arch_param \
${@get_distro_components_argument(d)} \
@@ -323,7 +385,7 @@ do_bootstrap() {
"${ROOTFSDIR}" \
"${@get_distro_source(d)}" \
${DISTRO_DEBOOTSTRAP_SCRIPT}
-
+ fi
# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
fi
-
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
- -o APT::Update::Error-Mode=any
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ "
+ else
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ fi
umount "${ROOTFSDIR}/dev/shm"
umount "${ROOTFSDIR}/dev/pts"
--
2.44.2
debootstrap task. So, debootstrap is then done from it.
The required packages are downloaded via python-apt and
reprepro creates debian-like repository from .deb files.
For debian targets host keyring is used while ubuntu/raspbian
targets use keys specified by DISTRO_BOOTSTRAP_KEYS variable.
The goal is have workable base-apt repo before first build completed.
.../isar-bootstrap/isar-bootstrap-host.bb | 2 +
.../isar-bootstrap/isar-bootstrap.inc | 87 +++++++++++++++++--
2 files changed, 81 insertions(+), 8 deletions(-)
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 4f90fd01..1ace818f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -15,3 +15,5 @@ require isar-bootstrap.inc
HOST_DISTRO_BOOTSTRAP_KEYS ?= ""
DISTRO_BOOTSTRAP_KEYS = "${HOST_DISTRO_BOOTSTRAP_KEYS}"
+
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 9fd928ad..589fa2c1 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -33,6 +33,13 @@ DISTRO_BOOTSTRAP_BASE_PACKAGES:append:https-support = ",ca-certificates"
DISTRO_VARS_PREFIX ?= "${@'HOST_' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else ''}"
BOOTSTRAP_DISTRO = "${@d.getVar('HOST_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'DISTRO')}"
BOOTSTRAP_BASE_DISTRO = "${@d.getVar('HOST_BASE_DISTRO' if bb.utils.to_boolean(d.getVar('BOOTSTRAP_FOR_HOST')) else 'BASE_DISTRO')}"
+BOOTSTRAP_DISTRO_ARCH = "${@d.getVar('HOST_ARCH' if d.getVar('BOOTSTRAP_FOR_HOST') == '1' else 'DISTRO_ARCH')}"
+
+# For newer distros "usr-is-merged" indirectly required by debootstrap
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:bookworm = ",usr-is-merged"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid = ",usr-is-merged"
+DISTRO_BOOTSTRAP_BASE_PACKAGES:append:sid-ports = ",usr-is-merged"
+
FILESEXTRAPATHS:append = ":${BBPATH}"
APT_SNAPSHOT_DATE = "${@ get_apt_snapshot_date(d)}"
@@ -291,12 +298,56 @@ do_bootstrap[network] = "${TASK_USE_NETWORK_AND_SUDO}"
inherit compat
+inherit debrepo
+
+debrepo_bootstrap_prepare() {
+ [ "${ISAR_PREFETCH_BASE_APT}" != "1" ] && return
+ [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ] && return
+
+ debrepo_args=""
+ if [ "${BASE_DISTRO}" != "debian" ]; then
+ if [ "${BASE_DISTRO}" != "raspbian" ] && [ "${BASE_DISTRO}" != "raspios" ] || [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ else
+ if [ "${BASE_DISTRO_CODENAME}" = "sid" ]; then
+ debrepo_args="$debrepo_args --keydir=${WORKDIR}"
+ fi
+ fi
+ if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
+ debrepo_args="$debrepo_args --compatarch=${COMPAT_DISTRO_ARCH}"
+ fi
+
+ if [ "${BOOTSTRAP_FOR_HOST}" = "1" ]; then
+ debrepo_args="$debrepo_args --crossarch=${DISTRO_ARCH}"
+ fi
+
+ if [ -n "${GNUPGHOME}" ]; then
+ export GNUPGHOME="${GNUPGHOME}"
+ fi
+ export GNUPGHOME="${GNUPGHOME}"
+
+ ${SCRIPTSDIR}/debrepo --init \
+ --workdir="${DEBREPO_WORKDIR}" \
+ --aptsrcsfile="${APTSRCS_INIT}" \
+ --repodir="${REPO_BASE_DIR}" \
+ --repodbdir="${REPO_BASE_DB_DIR}" \
+ --mirror="${@get_distro_source(d)}" \
+ --arch="${BOOTSTRAP_DISTRO_ARCH}" \
+ --distro="${BOOTSTRAP_BASE_DISTRO}" \
+ --codename="${BASE_DISTRO_CODENAME}" \
+ ${debrepo_args} \
+ ${DISTRO_BOOTSTRAP_BASE_PACKAGES}
+}
+
do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
if [ -z "${COMPAT_DISTRO_ARCH}" ]; then
bbfatal "${DISTRO_ARCH} does not have a compat arch"
fi
fi
+
+ debrepo_bootstrap_prepare
+
debootstrap_args="--verbose --variant=minbase --include=${DISTRO_BOOTSTRAP_BASE_PACKAGES}"
if [ -f "${DISTRO_BOOTSTRAP_KEYRING}" ]; then
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
@@ -314,8 +365,19 @@ do_bootstrap() {
debootstrap_args="$debootstrap_args --keyring=${DISTRO_BOOTSTRAP_KEYRING}"
sudo -E -s <<'EOSUDO'
set -e
if [ "${BOOTSTRAP_FOR_HOST}" = "0" ]; then
- arch_param="--arch=${DISTRO_ARCH}"
+ arch_param="--arch=${BOOTSTRAP_DISTRO_ARCH}"
fi
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ ${DEBOOTSTRAP} $debootstrap_args \
+ $arch_param \
+ ${@get_distro_components_argument(d)} \
+ ${@get_distro_suite(d)} \
+ ${ROOTFSDIR} \
+ file://${REPO_BASE_DIR}/${BOOTSTRAP_BASE_DISTRO} \
+ ${DISTRO_DEBOOTSTRAP_SCRIPT}
+ "
+ else
${DEBOOTSTRAP} $debootstrap_args \
$arch_param \
${@get_distro_components_argument(d)} \
@@ -323,7 +385,7 @@ do_bootstrap() {
"${ROOTFSDIR}" \
"${@get_distro_source(d)}" \
${DISTRO_DEBOOTSTRAP_SCRIPT}
-
+ fi
# Install apt config
mkdir -p "${ROOTFSDIR}/etc/apt/preferences.d"
install -v -m644 "${APTPREFS}" \
@@ -395,12 +457,21 @@ do_bootstrap() {
if [ "${ISAR_ENABLE_COMPAT_ARCH}" = "1" ]; then
chroot "${ROOTFSDIR}" /usr/bin/dpkg --add-architecture ${COMPAT_DISTRO_ARCH}
fi
-
- chroot "${ROOTFSDIR}" /usr/bin/apt-get update -y \
- -o APT::Update::Error-Mode=any
- chroot "${ROOTFSDIR}" /usr/bin/apt-get install -y -f
- chroot "${ROOTFSDIR}" /usr/bin/apt-get dist-upgrade -y \
- -o Debug::pkgProblemResolver=yes
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ "
+ else
+ chroot ${ROOTFSDIR} /usr/bin/apt-get update -y \
+ -o APT::Update::Error-Mode=any
+ chroot ${ROOTFSDIR} /usr/bin/apt-get install -y -f
+ chroot ${ROOTFSDIR} /usr/bin/apt-get dist-upgrade -y \
+ -o Debug::pkgProblemResolver=yes
+ fi
umount "${ROOTFSDIR}/dev/shm"
umount "${ROOTFSDIR}/dev/pts"
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:18 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
Fill base-apt repo with source packages.
meta/classes/deb-dl-dir.bbclass | 19 +++++++++++++++++++
meta/classes/rootfs.bbclass | 3 +++
2 files changed, 22 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 55e56c50..48233a62 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -41,6 +41,25 @@ debsrc_undo_mounts() {
EOSUDO
}
+debsrc_fill_base_apt() {
+ export rootfs="$1"
+
+ local srcpkgs=""
+ for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do
+ is_not_part_of_current_build "${package}" && continue
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+ local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
+ [ -n "$dscfile" ] && continue
+
+ if [ `echo ${srcpkgs} | grep -c "${src}=${version}"` -le 0 ]; then
+ srcpkgs="${srcpkgs} ${src}=${version}"
+ fi
+ done
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}"
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index f41fb498..f9c37576 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -235,6 +235,9 @@ cache_deb_src() {
sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ debsrc_fill_base_apt ${ROOTFSDIR}
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
--
2.44.2
meta/classes/deb-dl-dir.bbclass | 19 +++++++++++++++++++
meta/classes/rootfs.bbclass | 3 +++
2 files changed, 22 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 55e56c50..48233a62 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -41,6 +41,25 @@ debsrc_undo_mounts() {
EOSUDO
}
+debsrc_fill_base_apt() {
+ export rootfs="$1"
+
+ local srcpkgs=""
+ for package in $(find "${REPO_BASE_DIR}" -maxdepth 6 -type f -iname '*\.deb'); do
+ is_not_part_of_current_build "${package}" && continue
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+ local dscname="$(echo ${src}_${version} | sed -e 's/_[0-9]\+:/_/')"
+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${dscname}.dsc")
+ [ -n "$dscfile" ] && continue
+
+ if [ `echo ${srcpkgs} | grep -c "${src}=${version}"` -le 0 ]; then
+ srcpkgs="${srcpkgs} ${src}=${version}"
+ fi
+ done
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${srcpkgs}"
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index f41fb498..f9c37576 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -235,6 +235,9 @@ cache_deb_src() {
sudo cp -Trpn --reflink=auto "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"
deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
+
+ debsrc_fill_base_apt ${ROOTFSDIR}
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
debsrc_download ${ROOTFSDIR} ${ROOTFS_BASE_DISTRO}-${BASE_DISTRO_CODENAME}
sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:18 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This makes Isar use `base-apt` repo in different way. Any package
installation is done from `base-apt` repo which is prepopulated
from external mirrors.
This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.
In order to be able to run CI in old mode, allow CI read the option
from the environment. Also, adjust some tests (like repro one) to
make them work with ISAR_PREFETCH_BASE_APT set.
meta-test/conf/local.conf.sample | 3 +++
testsuite/cibase.py | 4 ++++
testsuite/cibuilder.py | 8 +++++++-
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index f692f533..941d7cce 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -34,6 +34,9 @@ IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsc
IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
+# Use new base-apt behaviour
+ISAR_PREFETCH_BASE_APT ?= "1"
+
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index b2a804b7..f5eec864 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -45,9 +45,13 @@ class CIBaseTest(CIBuilder):
self.fail('GPG import failed')
try:
+ self.move_in_build_dir('tmp', 'tmp_before_repro')
self.bitbake(targets, **kwargs)
self.move_in_build_dir('tmp', 'tmp_middle_repro_%s' % ('signed' if signed else 'unsigned'))
+
+ os.makedirs(f"{self.build_dir}/tmp/deploy/")
+ self.move_in_build_dir('tmp_middle_repro_%s/deploy/base-apt' % ('signed' if signed else 'unsigned'), 'tmp/deploy/base-apt')
self.configure(gpg_pub_key=gpg_pub_key if signed else None, offline=True, sstate_dir="", **kwargs)
self.bitbake(targets, **kwargs)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index a20e88f9..3b3d8708 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -104,6 +104,9 @@ class CIBuilder(Test):
# get parameters from environment
distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
+ # get prefetch base apt mode from environment
+ prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT')
+
self.log.info(f'===================================================\n'
f'Configuring build_dir {self.build_dir}\n'
f' compat_arch = {compat_arch}\n'
@@ -121,6 +124,7 @@ class CIBuilder(Test):
f' sstate_dir = {sstate_dir}\n'
f' ccache_dir = {ccache_dir}\n'
f' image_install = {image_install}\n'
+ f' prefetch_base_apt = {prefetch_base_apt}\n'
f'===================================================')
# determine bitbake_args
@@ -169,7 +173,9 @@ class CIBuilder(Test):
if sstate_dir:
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
if image_install is not None:
- f.write('IMAGE_INSTALL = "%s"' % image_install)
+ f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+ if prefetch_base_apt == "0":
+ f.write('ISAR_PREFETCH_BASE_APT = "0"\n')
# include ci_build.conf in local.conf
with open(self.build_dir + '/conf/local.conf', 'r+') as f:
--
2.44.2
installation is done from `base-apt` repo which is prepopulated
from external mirrors.
This behaviour is disabled by default for downstreams. To enable it,
set the variable to "1", like isar does in local.conf.sample.
In order to be able to run CI in old mode, allow CI read the option
from the environment. Also, adjust some tests (like repro one) to
make them work with ISAR_PREFETCH_BASE_APT set.
meta-test/conf/local.conf.sample | 3 +++
testsuite/cibase.py | 4 ++++
testsuite/cibuilder.py | 8 +++++++-
3 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index f692f533..941d7cce 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -34,6 +34,9 @@ IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsc
IMAGE_INSTALL:remove:qemuamd64-sb = "example-module-${KERNEL_NAME}"
IMAGE_INSTALL:append:qemuamd64-sb = " example-module-signed-${KERNEL_NAME}"
+# Use new base-apt behaviour
+ISAR_PREFETCH_BASE_APT ?= "1"
+
# Users and groups
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"
diff --git a/testsuite/cibase.py b/testsuite/cibase.py
index b2a804b7..f5eec864 100755
--- a/testsuite/cibase.py
+++ b/testsuite/cibase.py
@@ -45,9 +45,13 @@ class CIBaseTest(CIBuilder):
self.fail('GPG import failed')
try:
+ self.move_in_build_dir('tmp', 'tmp_before_repro')
self.bitbake(targets, **kwargs)
self.move_in_build_dir('tmp', 'tmp_middle_repro_%s' % ('signed' if signed else 'unsigned'))
+
+ os.makedirs(f"{self.build_dir}/tmp/deploy/")
+ self.move_in_build_dir('tmp_middle_repro_%s/deploy/base-apt' % ('signed' if signed else 'unsigned'), 'tmp/deploy/base-apt')
self.configure(gpg_pub_key=gpg_pub_key if signed else None, offline=True, sstate_dir="", **kwargs)
self.bitbake(targets, **kwargs)
diff --git a/testsuite/cibuilder.py b/testsuite/cibuilder.py
index a20e88f9..3b3d8708 100755
--- a/testsuite/cibuilder.py
+++ b/testsuite/cibuilder.py
@@ -104,6 +104,9 @@ class CIBuilder(Test):
# get parameters from environment
distro_apt_premir = os.getenv('DISTRO_APT_PREMIRRORS')
+ # get prefetch base apt mode from environment
+ prefetch_base_apt = os.getenv('ISAR_PREFETCH_BASE_APT')
+
self.log.info(f'===================================================\n'
f'Configuring build_dir {self.build_dir}\n'
f' compat_arch = {compat_arch}\n'
@@ -121,6 +124,7 @@ class CIBuilder(Test):
f' sstate_dir = {sstate_dir}\n'
f' ccache_dir = {ccache_dir}\n'
f' image_install = {image_install}\n'
+ f' prefetch_base_apt = {prefetch_base_apt}\n'
f'===================================================')
# determine bitbake_args
@@ -169,7 +173,9 @@ class CIBuilder(Test):
if sstate_dir:
f.write('SSTATE_DIR = "%s"\n' % sstate_dir)
if image_install is not None:
- f.write('IMAGE_INSTALL = "%s"' % image_install)
+ f.write('IMAGE_INSTALL = "%s"\n' % image_install)
+ if prefetch_base_apt == "0":
+ f.write('ISAR_PREFETCH_BASE_APT = "0"\n')
# include ci_build.conf in local.conf
with open(self.build_dir + '/conf/local.conf', 'r+') as f:
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:18 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This is the main utility responsible for prefetching packages
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").
Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.
In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.
scripts/debrepo | 590 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 590 insertions(+)
create mode 100755 scripts/debrepo
diff --git a/scripts/debrepo b/scripts/debrepo
new file mode 100755
index 00000000..b96fa58b
--- /dev/null
+++ b/scripts/debrepo
@@ -0,0 +1,590 @@
+#!/usr/bin/env python3
+
+"""
+
+When building the image, Isar downloads required Debian packages from external
+mirrors. After build completed, it can pick all downloaded packages from DL_DIR
+and build local 'base-apt' Debian-like repo from them.
+
+This tool allows to download packages and create local repo in advance. So,
+Isar just uses this local repository and does not interact with external
+mirrors. Such approach makes deb-dl import/export functionality redundant.
+
+Script `debrepo` works in so-called "context" directory. It means some
+rootfs-like directory with bare minimum of directories/files required for
+python3-apt to work.
+
+Context directory path is passed with "--workdir <dir>" command-line option.
+On context creating, all passed parameters are stored in the context directory
+and picked every time the context is used again.
+
+1. Repo for building Debian system
+```
+debrepo --workdir=d12 --init locales gnupg
+```
+Initialize the context in "d12" directory and create a repository in
+"d12/repo/apt" directory sufficient to debootstrap default system
+(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg"
+with all their dependencies will be available in this repo.
+
+```
+debrepo --workdir=d12 docbook-to-man
+```
+Adds "docbook-to-man" packages with its dependencies to earlier created repo.
+
+```
+debrepo --workdir=d12 --srcmode docbook-to-man
+```
+Downloads source package for "docbook-to-man" and adds it to the repo
+
+2. Repo for building Ubuntu system
+```
+debrepo --init --workdir=uf \
+--distro=ubuntu --codename=focal --arch=arm64 \
+--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \
+--repodir=repo/apt --repodbdir=repo/db \
+--mirror=http://ports.ubuntu.com/ubuntu-ports \
+locales gnupg
+```
+Initialize the context in "uf" directory and create a repository in "repo/apt"
+directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and
+source list are specified by corresponding arguments. Packages "locales" and
+" gnupg" with the dependencies will be also placed to the repo.
+
+```
+debrepo --workdir=uf gnupg,locales
+```
+Add "gnupg" and "locales" packages with their dependencies to earlier created
+ubuntu repo. Other parameters (distro, codename, arch) are ommited since they
+are picked from the context.
+
+3. Repo for cross-building Debian system
+```
+debrepo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf
+```
+Initialize the context in "d11" directory sufficient to deboostrap Debian
+Bullseye (amd64) with foreign "armhf" architecture support
+
+```
+debrepo --workdir=d11 gcc
+```
+Add "gcc" package (amd64 version) to earlier created repo.
+
+```
+debrepo --workdir=d11 --crossbuild gcc
+```
+Add "gcc" package (armhf version) to earlier created repo.
+"""
+
+import os
+import sys
+import fcntl
+
+import argparse
+import shutil
+import subprocess
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+REPREPRO_TIMEOUT = 1200
+
+
+class DebRepo(object):
+ class DebRepoCtx(object):
+ def __init__(self, workdir):
+ self.distro = "debian"
+ self.codename = "bullseye"
+ self.arch = "amd64"
+ self.mirror = "http://deb.debian.org/debian"
+
+ self.repodir = f"{workdir}/repo/apt"
+ self.repodbdir = f"{workdir}/repo/db"
+
+ self.crossarch = self.arch
+ self.compatarch = None
+ self.keydir = "/etc/apt/trusted.gpg.d"
+
+ def __init__(self, args):
+ self.workdir = os.path.abspath(args.workdir)
+ self.ctx = self.DebRepoCtx(self.workdir)
+
+ self.cache = None
+ self.depcache = None
+ self.sr = None
+ self.extrarepo = None
+
+ self.ctx_load()
+ self.ctx_update(args)
+ self.ctx_save()
+
+ print(
+ f"ctx workdir: {self.workdir}\n"
+ f" distro: {self.ctx.distro}\n"
+ f" codename: {self.ctx.codename}\n"
+ f" arch: {self.ctx.arch}\n"
+ f" mirror: {self.ctx.mirror}\n"
+ f" repodir: {self.ctx.repodir}\n"
+ f" repodbdir: {self.ctx.repodbdir}\n"
+ f" crossarch: {self.ctx.crossarch}\n"
+ f" compatarch: {self.ctx.compatarch}\n"
+ f" keydir: {self.ctx.keydir}"
+ )
+
+ if args.extrarepo:
+ self.extrarepo = os.path.abspath(args.extrarepo)
+
+ def ctx_load(self):
+ ctxfile = f"{self.workdir}/debrepo.ctx"
+
+ if os.path.isfile(ctxfile):
+ with open(ctxfile, 'rb') as f:
+ self.ctx = pickle.load(f)
+
+ def ctx_save(self):
+ ctxfile = f"{self.workdir}/debrepo.ctx"
+
+ with open(ctxfile, 'wb') as f:
+ pickle.dump(self.ctx, f)
+
+ def ctx_update(self, args):
+ if args.distro:
+ self.ctx.distro = args.distro
+ if args.codename:
+ self.ctx.codename = args.codename
+ if args.arch:
+ self.ctx.arch = args.arch
+ if args.mirror:
+ self.ctx.mirror = args.mirror
+
+ if args.repodir:
+ self.ctx.repodir = os.path.abspath(args.repodir)
+ if args.repodbdir:
+ self.ctx.repodbdir = os.path.abspath(args.repodbdir)
+
+ if args.crossarch:
+ self.ctx.crossarch = args.crossarch
+ if args.compatarch:
+ self.ctx.compatarch = args.compatarch
+ if args.keydir:
+ self.ctx.keydir = args.keydir
+
+ def create_rootfs(self, aptsrcsfile):
+ os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True)
+ with open(f"{self.workdir}/var/lib/dpkg/status", "w"):
+ pass
+
+ os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True)
+
+ srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list"
+ if aptsrcsfile and os.path.exists(aptsrcsfile):
+ shutil.copy(aptsrcsfile, srcfile)
+ else:
+ with open(srcfile, "w") as f:
+ repo = f"{self.ctx.mirror} {self.ctx.codename} main"
+ f.write(f"deb {repo}\n")
+ f.write(f"deb-src {repo}\n")
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial",
+ exist_ok=True)
+
+ os.makedirs(f"{self.workdir}/tmp", exist_ok=True)
+
+ def create_repo_dist(self):
+ conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf"
+ os.makedirs(conf_dir, exist_ok=True)
+ if not os.path.exists(f"{conf_dir}/distributions"):
+ with open(f"{conf_dir}/distributions", "w") as f:
+ f.write(f"Codename: {self.ctx.codename}\n")
+ f.write(
+ "Architectures: "
+ "i386 armhf arm64 amd64 mipsel riscv64 source\n")
+ f.write("Components: main\n")
+
+ def apt_config(self, init, crossbuild):
+ # Configure apt to work with empty directory
+ if not init and self.ctx.arch != self.ctx.crossarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch
+ apt_pkg.config["APT::Architectures::"] = self.ctx.arch
+
+ if not init and self.ctx.compatarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch
+
+ apt_pkg.config.set("APT::Architecture", self.ctx.arch)
+
+ apt_pkg.config.set("Dir", self.workdir)
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}")
+ apt_pkg.config.set("Dir::State::status",
+ f"{self.workdir}/var/lib/dpkg/status")
+
+ apt_pkg.config.set("APT::Install-Recommends", "0")
+ apt_pkg.config.set("APT::Install-Suggests", "0")
+
+ # Use host keys for authentification
+ apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir)
+
+ # Allow using repositories without keys
+ apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+ def mark_essential(self):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ if pkg.essential:
+ self.depcache.mark_install(pkg)
+
+ def mark_by_prio(self, priority):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ ver = self.depcache.get_candidate_ver(pkg)
+ if ver and ver.priority <= priority:
+ self.depcache.mark_install(pkg)
+
+ def mark_pkg(self, name, crossbuild):
+ pkgname = name
+
+ if pkgname and (pkgname not in self.cache):
+ # Try for cross arch
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ pkgname += f":{self.ctx.crossarch}"
+
+ if pkgname not in self.cache:
+ print(f"Error: package '{name}' not found")
+ return False
+
+ pkg = self.cache[pkgname]
+
+ if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions):
+ if (pkg.has_provides) and (not pkg.has_versions):
+ print("pkgname is virtual package, selecting best provide")
+ # Select first provide
+ pkg_provide = pkg.provides_list[0][2]
+ # Find better provide with higher version
+ for provide in pkg.provides_list:
+ if apt_pkg.version_compare(provide[2].ver_str,
+ pkg_provide.ver_str) > 0:
+ pkg_provide = provide[2]
+ self.depcache.mark_install(pkg_provide.parent_pkg)
+ else:
+ self.depcache.mark_install(pkg)
+ else:
+ version = pkg.version_list[0]
+ if version.arch == "all":
+ self.depcache.mark_install(pkg)
+ else:
+ if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+ if (pkgname, self.ctx.arch) in self.cache:
+ nativepkg = self.cache[pkgname, self.ctx.arch]
+ self.depcache.mark_install(nativepkg)
+ else:
+ return False
+ else:
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ crosspkg = self.cache[pkgname, self.ctx.crossarch]
+ self.depcache.mark_install(crosspkg)
+ else:
+ return False
+
+ return True
+
+ def mark_list(self, pkglist, crossbuild):
+ ret = True
+ if pkglist:
+ for pkgname in pkglist:
+ ret = ret and self.mark_pkg(pkgname, crossbuild)
+
+ return ret
+
+ def handle_deb(self, item):
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "includedeb",
+ self.ctx.codename,
+ item.destfile
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+
+ def handle_repo(self, fetcher):
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ fd = open(f"{self.workdir}/{dir_cache}.lock", "w")
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ fetcher.run()
+ fd.close()
+ for item in fetcher.items:
+ if item.status == item.STAT_ERROR:
+ print("Some error ocured: '%s'" % item.error_text)
+ pass
+ else:
+ self.handle_deb(item)
+
+ def get_filename(self, uri):
+ path = urllib.parse.urlparse(uri).path
+ unquoted_path = urllib.parse.unquote(path)
+ basename = os.path.basename(unquoted_path)
+ return basename
+
+ def fetch_file(self, uri):
+ filename = self.get_filename(uri)
+ subprocess.run([
+ "wget",
+ "-H",
+ "--timeout=30",
+ "--tries=3",
+ "-nv",
+ uri,
+ "-O",
+ f"{self.workdir}/tmp/{filename}"
+ ],
+ stdout=subprocess.PIPE)
+
+ def handle_dsc(self, uri):
+ filename = self.get_filename(uri)
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "-S", "-", "-P" "source",
+ "--delete",
+ "includedsc",
+ self.ctx.codename,
+ os.path.realpath(f"{self.workdir}/tmp/{filename}")
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+
+ def handle_src_list(self, pkgs):
+ if pkgs:
+ fetched_files = []
+ for pkg in pkgs:
+ pkgname = pkg
+ pkgver = ""
+ if '=' in pkg:
+ pkgname = pkg.split("=")[0]
+ pkgver = pkg.split("=")[1]
+
+ self.sr.restart()
+ while self.sr.lookup(pkgname):
+ if pkgver and pkgver != self.sr.version:
+ continue
+
+ for sr_file in self.sr.files:
+ print(self.sr.index.archive_uri(sr_file[2]))
+ filename = os.path.basename(sr_file.path)
+ if filename not in fetched_files:
+ self.fetch_file(self.sr.index.archive_uri(sr_file[2]))
+ fetched_files.append(filename)
+
+ dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2])
+ self.handle_dsc(dsc_uri)
+ break
+
+ def apt_run(self, init, srcmode, pkgs, dscfile, crossbuild):
+ apt_pkg.init()
+
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ if self.extrarepo:
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ with open(extrarepo_list, "w") as f:
+ distdir=os.path.join(self.extrarepo, "dists")
+ if os.path.isdir(distdir):
+ for dist in os.listdir(distdir):
+ repodir = os.path.join(distdir,dist)
+ if os.path.isdir(repodir):
+ for repo in os.listdir(repodir):
+ if os.path.isdir(os.path.join(repodir, repo)):
+ f.write(f"deb file://{self.extrarepo} "
+ f"{dist} {repo}\n")
+
+ sources = apt_pkg.SourceList()
+ sources.read_main_list()
+
+ progress = apt.progress.text.AcquireProgress()
+
+ self.cache = apt_pkg.Cache()
+ if init:
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+
+ if self.extrarepo:
+ apt_pkg.config.set("Dir::Etc::SourceList", extrarepo_list)
+ apt_pkg.config.set("APT::Get::List-Cleanup", "0")
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+ os.remove(extrarepo_list)
+
+ self.depcache = apt_pkg.DepCache(self.cache)
+ self.sr = apt_pkg.SourceRecords()
+
+ ret = True
+
+ if init:
+ self.mark_essential()
+ # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+ self.mark_by_prio(1)
+
+ pkgs = list(filter(None, ','.join(pkgs).split(',')))
+ if srcmode:
+ self.handle_src_list(set(pkgs))
+ else:
+ ret = self.mark_list(pkgs, crossbuild)
+
+ if dscfile:
+ fobj = open(dscfile, "r")
+
+ try:
+ tagfile = apt_pkg.TagFile(fobj)
+ while tagfile.step() == 1:
+ deps = tagfile.section.get("Build-Depends", "")
+ # Remove extra commas and spaces - apt_pkg.parse_src_depends
+ # doesnt like lines like ", device-tree-compiler"
+ deps = ', '.join(
+ [s.strip() for s in deps.split(',') if s.strip()]
+ )
+ print(f"parsed deps: {deps}")
+ for item in apt_pkg.parse_src_depends(deps, False):
+ pkgname = item[0][0]
+ self.mark_pkg(pkgname, crossbuild)
+
+ finally:
+ fobj.close()
+
+ if not ret:
+ sys.exit("Some of requested packages not found")
+
+ if init or not srcmode:
+ fetcher = apt_pkg.Acquire(progress)
+ pm = apt_pkg.PackageManager(self.depcache)
+
+ recs = apt_pkg.PackageRecords(self.cache)
+ pm.get_archives(fetcher, sources, recs)
+
+ self.handle_repo(fetcher)
+
+
+def parse_arguments():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--init",
+ default=False, action="store_true",
+ help="initialize context in WORKDIR")
+ parser.add_argument(
+ "--workdir",
+ type=str, required=True,
+ help="work directory storing debrepo context")
+ parser.add_argument(
+ "--aptsrcsfile",
+ type=str, metavar="PATH",
+ help="sources.list file to use when init")
+ parser.add_argument(
+ "--srcmode",
+ default=False, action="store_true",
+ help="add source packages instead of debs")
+ parser.add_argument(
+ "--repodir",
+ type=str, metavar="REPO",
+ help="repository directory")
+ parser.add_argument(
+ "--repodbdir",
+ type=str, metavar="REPODB",
+ help="repository database directory")
+ parser.add_argument(
+ "--extrarepo",
+ type=str, metavar="REPO",
+ help="extra repository to consider")
+ parser.add_argument(
+ "--mirror",
+ type=str,
+ help="use custom distro mirror")
+ parser.add_argument(
+ "--distro",
+ type=str,
+ help="select distro to use")
+ parser.add_argument(
+ "--codename",
+ type=str,
+ help="distro codename")
+ parser.add_argument(
+ "--arch",
+ type=str,
+ help="distro arch")
+ parser.add_argument(
+ "--compatarch",
+ type=str, metavar="ARCH",
+ help="compat arch to use")
+ parser.add_argument(
+ "--crossarch",
+ type=str, metavar="ARCH",
+ help="cross-build arch")
+ parser.add_argument(
+ "--keydir",
+ type=str,
+ help="directory with distro keys")
+ parser.add_argument(
+ "--no-check-gpg",
+ default=False, action="store_true",
+ help="allow insecure repositories")
+ parser.add_argument(
+ "--dscfile",
+ type=str, metavar="PATH",
+ help="Debian source file to parse")
+ parser.add_argument(
+ "--crossbuild",
+ default=False, action="store_true",
+ help="add packages with cross arch")
+
+ parser.add_argument(
+ "packages",
+ nargs='*', type=str,
+ help="space- or comma-separated list of packages to add")
+
+ args = parser.parse_args()
+
+ return args
+
+
+def main():
+ args = parse_arguments()
+
+ if not (args.init or args.packages or args.dscfile):
+ sys.exit("Nothing to do")
+
+ workdir = os.path.abspath(args.workdir)
+ os.makedirs(workdir, exist_ok=True)
+
+ with open(f"{workdir}/debrepo.lock", "a") as file:
+ fcntl.flock(file.fileno(), fcntl.LOCK_EX)
+
+ debrepo = DebRepo(args)
+
+ if args.init:
+ debrepo.create_rootfs(args.aptsrcsfile)
+ debrepo.create_repo_dist()
+
+ debrepo.apt_config(args.init, args.crossbuild)
+ debrepo.apt_run(args.init, args.srcmode, args.packages,
+ args.dscfile, args.crossbuild)
+
+ #Unlock debrepo context
+ fcntl.flock(file.fileno(), fcntl.LOCK_UN)
+
+
+if __name__ == "__main__":
+ main()
--
2.44.2
into local `base-apt` repo from external Debian mirrors. It uses
python-apt module and requires some kind of minimal `rootfs` to work
(let's call it "debrepo context").
Once initialized with `--init --workdir=<path>`, it stores the initial
configuration in `repo.opts` file inside the context and uses it at
futher calls.
In future, the logic `debrepo` script implements could be directly
implemented inside bitbake classes.
scripts/debrepo | 590 ++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 590 insertions(+)
create mode 100755 scripts/debrepo
diff --git a/scripts/debrepo b/scripts/debrepo
new file mode 100755
index 00000000..b96fa58b
--- /dev/null
+++ b/scripts/debrepo
@@ -0,0 +1,590 @@
+#!/usr/bin/env python3
+
+"""
+# This software is a part of Isar.
+# Copyright (C) 2024 ilbers GmbH
+
+# debrepo: build Debian-like repo using "python3-apt" library.
+# Copyright (C) 2024 ilbers GmbH
+
+
+When building the image, Isar downloads required Debian packages from external
+mirrors. After build completed, it can pick all downloaded packages from DL_DIR
+and build local 'base-apt' Debian-like repo from them.
+
+This tool allows to download packages and create local repo in advance. So,
+Isar just uses this local repository and does not interact with external
+mirrors. Such approach makes deb-dl import/export functionality redundant.
+
+Script `debrepo` works in so-called "context" directory. It means some
+rootfs-like directory with bare minimum of directories/files required for
+python3-apt to work.
+
+Context directory path is passed with "--workdir <dir>" command-line option.
+On context creating, all passed parameters are stored in the context directory
+and picked every time the context is used again.
+
+1. Repo for building Debian system
+```
+debrepo --workdir=d12 --init locales gnupg
+```
+Initialize the context in "d12" directory and create a repository in
+"d12/repo/apt" directory sufficient to debootstrap default system
+(e.g., debian-bookworm-amd64). Additionally, packages "locales" and "gnupg"
+with all their dependencies will be available in this repo.
+
+```
+debrepo --workdir=d12 docbook-to-man
+```
+Adds "docbook-to-man" packages with its dependencies to earlier created repo.
+
+```
+debrepo --workdir=d12 --srcmode docbook-to-man
+```
+Downloads source package for "docbook-to-man" and adds it to the repo
+
+2. Repo for building Ubuntu system
+```
+debrepo --init --workdir=uf \
+--distro=ubuntu --codename=focal --arch=arm64 \
+--aptsrcsfile=/work/isar/meta-isar/conf/distro/ubuntu-focal-ports.list \
+--repodir=repo/apt --repodbdir=repo/db \
+--mirror=http://ports.ubuntu.com/ubuntu-ports \
+locales gnupg
+```
+Initialize the context in "uf" directory and create a repository in "repo/apt"
+directory sufficient deboostraup ubuntu-focal arm64 system. Mirror to use and
+source list are specified by corresponding arguments. Packages "locales" and
+" gnupg" with the dependencies will be also placed to the repo.
+
+```
+debrepo --workdir=uf gnupg,locales
+```
+Add "gnupg" and "locales" packages with their dependencies to earlier created
+ubuntu repo. Other parameters (distro, codename, arch) are ommited since they
+are picked from the context.
+
+3. Repo for cross-building Debian system
+```
+debrepo --init --workdir=d11 --codename=bullseye--arch=amd64 --crossarch=armhf
+```
+Initialize the context in "d11" directory sufficient to deboostrap Debian
+Bullseye (amd64) with foreign "armhf" architecture support
+
+```
+debrepo --workdir=d11 gcc
+```
+Add "gcc" package (amd64 version) to earlier created repo.
+
+```
+debrepo --workdir=d11 --crossbuild gcc
+```
+Add "gcc" package (armhf version) to earlier created repo.
+"""
+
+import os
+import sys
+import fcntl
+
+import argparse
+import shutil
+import subprocess
+import pickle
+import urllib.parse
+
+import apt_pkg
+import apt.progress.base
+
+
+REPREPRO_TIMEOUT = 1200
+
+
+class DebRepo(object):
+ class DebRepoCtx(object):
+ def __init__(self, workdir):
+ self.distro = "debian"
+ self.codename = "bullseye"
+ self.arch = "amd64"
+ self.mirror = "http://deb.debian.org/debian"
+
+ self.repodir = f"{workdir}/repo/apt"
+ self.repodbdir = f"{workdir}/repo/db"
+
+ self.crossarch = self.arch
+ self.compatarch = None
+ self.keydir = "/etc/apt/trusted.gpg.d"
+
+ def __init__(self, args):
+ self.workdir = os.path.abspath(args.workdir)
+ self.ctx = self.DebRepoCtx(self.workdir)
+
+ self.cache = None
+ self.depcache = None
+ self.sr = None
+ self.extrarepo = None
+
+ self.ctx_load()
+ self.ctx_update(args)
+ self.ctx_save()
+
+ print(
+ f"ctx workdir: {self.workdir}\n"
+ f" distro: {self.ctx.distro}\n"
+ f" codename: {self.ctx.codename}\n"
+ f" arch: {self.ctx.arch}\n"
+ f" mirror: {self.ctx.mirror}\n"
+ f" repodir: {self.ctx.repodir}\n"
+ f" repodbdir: {self.ctx.repodbdir}\n"
+ f" crossarch: {self.ctx.crossarch}\n"
+ f" compatarch: {self.ctx.compatarch}\n"
+ f" keydir: {self.ctx.keydir}"
+ )
+
+ if args.extrarepo:
+ self.extrarepo = os.path.abspath(args.extrarepo)
+
+ def ctx_load(self):
+ ctxfile = f"{self.workdir}/debrepo.ctx"
+
+ if os.path.isfile(ctxfile):
+ with open(ctxfile, 'rb') as f:
+ self.ctx = pickle.load(f)
+
+ def ctx_save(self):
+ ctxfile = f"{self.workdir}/debrepo.ctx"
+
+ with open(ctxfile, 'wb') as f:
+ pickle.dump(self.ctx, f)
+
+ def ctx_update(self, args):
+ if args.distro:
+ self.ctx.distro = args.distro
+ if args.codename:
+ self.ctx.codename = args.codename
+ if args.arch:
+ self.ctx.arch = args.arch
+ if args.mirror:
+ self.ctx.mirror = args.mirror
+
+ if args.repodir:
+ self.ctx.repodir = os.path.abspath(args.repodir)
+ if args.repodbdir:
+ self.ctx.repodbdir = os.path.abspath(args.repodbdir)
+
+ if args.crossarch:
+ self.ctx.crossarch = args.crossarch
+ if args.compatarch:
+ self.ctx.compatarch = args.compatarch
+ if args.keydir:
+ self.ctx.keydir = args.keydir
+
+ def create_rootfs(self, aptsrcsfile):
+ os.makedirs(f"{self.workdir}/var/lib/dpkg", exist_ok=True)
+ with open(f"{self.workdir}/var/lib/dpkg/status", "w"):
+ pass
+
+ os.makedirs(f"{self.workdir}/etc/apt/sources.list.d", exist_ok=True)
+
+ srcfile = f"{self.workdir}/etc/apt/sources.list.d/bootstrap.list"
+ if aptsrcsfile and os.path.exists(aptsrcsfile):
+ shutil.copy(aptsrcsfile, srcfile)
+ else:
+ with open(srcfile, "w") as f:
+ repo = f"{self.ctx.mirror} {self.ctx.codename} main"
+ f.write(f"deb {repo}\n")
+ f.write(f"deb-src {repo}\n")
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ os.makedirs(f"{self.workdir}/{dir_cache}/archives/partial",
+ exist_ok=True)
+
+ os.makedirs(f"{self.workdir}/tmp", exist_ok=True)
+
+ def create_repo_dist(self):
+ conf_dir = f"{self.ctx.repodir}/{self.ctx.distro}/conf"
+ os.makedirs(conf_dir, exist_ok=True)
+ if not os.path.exists(f"{conf_dir}/distributions"):
+ with open(f"{conf_dir}/distributions", "w") as f:
+ f.write(f"Codename: {self.ctx.codename}\n")
+ f.write(
+ "Architectures: "
+ "i386 armhf arm64 amd64 mipsel riscv64 source\n")
+ f.write("Components: main\n")
+
+ def apt_config(self, init, crossbuild):
+ # Configure apt to work with empty directory
+ if not init and self.ctx.arch != self.ctx.crossarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.crossarch
+ apt_pkg.config["APT::Architectures::"] = self.ctx.arch
+
+ if not init and self.ctx.compatarch:
+ apt_pkg.config["APT::Architectures::"] = self.ctx.compatarch
+
+ apt_pkg.config.set("APT::Architecture", self.ctx.arch)
+
+ apt_pkg.config.set("Dir", self.workdir)
+
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ apt_pkg.config.set("Dir::Cache", f"{self.workdir}/{dir_cache}")
+ apt_pkg.config.set("Dir::State::status",
+ f"{self.workdir}/var/lib/dpkg/status")
+
+ apt_pkg.config.set("APT::Install-Recommends", "0")
+ apt_pkg.config.set("APT::Install-Suggests", "0")
+
+ # Use host keys for authentification
+ apt_pkg.config.set("Dir::Etc::TrustedParts", self.ctx.keydir)
+
+ # Allow using repositories without keys
+ apt_pkg.config.set("Acquire::AllowInsecureRepositories", "1")
+
+ def mark_essential(self):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ if pkg.essential:
+ self.depcache.mark_install(pkg)
+
+ def mark_by_prio(self, priority):
+ for pkg in self.cache.packages:
+ if pkg.architecture == self.ctx.arch:
+ ver = self.depcache.get_candidate_ver(pkg)
+ if ver and ver.priority <= priority:
+ self.depcache.mark_install(pkg)
+
+ def mark_pkg(self, name, crossbuild):
+ pkgname = name
+
+ if pkgname and (pkgname not in self.cache):
+ # Try for cross arch
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ pkgname += f":{self.ctx.crossarch}"
+
+ if pkgname not in self.cache:
+ print(f"Error: package '{name}' not found")
+ return False
+
+ pkg = self.cache[pkgname]
+
+ if (not crossbuild) or (':' in pkgname) or (not pkg.has_versions):
+ if (pkg.has_provides) and (not pkg.has_versions):
+ print("pkgname is virtual package, selecting best provide")
+ # Select first provide
+ pkg_provide = pkg.provides_list[0][2]
+ # Find better provide with higher version
+ for provide in pkg.provides_list:
+ if apt_pkg.version_compare(provide[2].ver_str,
+ pkg_provide.ver_str) > 0:
+ pkg_provide = provide[2]
+ self.depcache.mark_install(pkg_provide.parent_pkg)
+ else:
+ self.depcache.mark_install(pkg)
+ else:
+ version = pkg.version_list[0]
+ if version.arch == "all":
+ self.depcache.mark_install(pkg)
+ else:
+ if version.multi_arch == version.MULTI_ARCH_FOREIGN:
+ if (pkgname, self.ctx.arch) in self.cache:
+ nativepkg = self.cache[pkgname, self.ctx.arch]
+ self.depcache.mark_install(nativepkg)
+ else:
+ return False
+ else:
+ if (pkgname, self.ctx.crossarch) in self.cache:
+ crosspkg = self.cache[pkgname, self.ctx.crossarch]
+ self.depcache.mark_install(crosspkg)
+ else:
+ return False
+
+ return True
+
+ def mark_list(self, pkglist, crossbuild):
+ ret = True
+ if pkglist:
+ for pkgname in pkglist:
+ ret = ret and self.mark_pkg(pkgname, crossbuild)
+
+ return ret
+
+ def handle_deb(self, item):
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "includedeb",
+ self.ctx.codename,
+ item.destfile
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+
+ def handle_repo(self, fetcher):
+ dir_cache = f"../apt_cache/{self.ctx.distro}-{self.ctx.codename}"
+ fd = open(f"{self.workdir}/{dir_cache}.lock", "w")
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ fetcher.run()
+ fd.close()
+ for item in fetcher.items:
+ if item.status == item.STAT_ERROR:
+ print("Some error ocured: '%s'" % item.error_text)
+ pass
+ else:
+ self.handle_deb(item)
+
+ def get_filename(self, uri):
+ path = urllib.parse.urlparse(uri).path
+ unquoted_path = urllib.parse.unquote(path)
+ basename = os.path.basename(unquoted_path)
+ return basename
+
+ def fetch_file(self, uri):
+ filename = self.get_filename(uri)
+ subprocess.run([
+ "wget",
+ "-H",
+ "--timeout=30",
+ "--tries=3",
+ "-nv",
+ uri,
+ "-O",
+ f"{self.workdir}/tmp/{filename}"
+ ],
+ stdout=subprocess.PIPE)
+
+ def handle_dsc(self, uri):
+ filename = self.get_filename(uri)
+ fd = open(f"{self.ctx.repodir}/repo.lock", 'w')
+ fcntl.flock(fd, fcntl.LOCK_EX)
+ subprocess.run([
+ "reprepro",
+ "--dbdir", f"{self.ctx.repodbdir}/{self.ctx.distro}",
+ "--outdir", f"{self.ctx.repodir}/{self.ctx.distro}",
+ "--confdir", f"{self.ctx.repodir}/{self.ctx.distro}/conf",
+ "-C", "main",
+ "-S", "-", "-P" "source",
+ "--delete",
+ "includedsc",
+ self.ctx.codename,
+ os.path.realpath(f"{self.workdir}/tmp/{filename}")
+ ], timeout=REPREPRO_TIMEOUT)
+ fd.close()
+
+ def handle_src_list(self, pkgs):
+ if pkgs:
+ fetched_files = []
+ for pkg in pkgs:
+ pkgname = pkg
+ pkgver = ""
+ if '=' in pkg:
+ pkgname = pkg.split("=")[0]
+ pkgver = pkg.split("=")[1]
+
+ self.sr.restart()
+ while self.sr.lookup(pkgname):
+ if pkgver and pkgver != self.sr.version:
+ continue
+
+ for sr_file in self.sr.files:
+ print(self.sr.index.archive_uri(sr_file[2]))
+ filename = os.path.basename(sr_file.path)
+ if filename not in fetched_files:
+ self.fetch_file(self.sr.index.archive_uri(sr_file[2]))
+ fetched_files.append(filename)
+
+ dsc_uri = self.sr.index.archive_uri(self.sr.files[0][2])
+ self.handle_dsc(dsc_uri)
+ break
+
+ def apt_run(self, init, srcmode, pkgs, dscfile, crossbuild):
+ apt_pkg.init()
+
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ if self.extrarepo:
+ extrarepo_list = f"{self.workdir}/etc/apt/sources.list.d/extrarepo.list"
+ with open(extrarepo_list, "w") as f:
+ distdir=os.path.join(self.extrarepo, "dists")
+ if os.path.isdir(distdir):
+ for dist in os.listdir(distdir):
+ repodir = os.path.join(distdir,dist)
+ if os.path.isdir(repodir):
+ for repo in os.listdir(repodir):
+ if os.path.isdir(os.path.join(repodir, repo)):
+ f.write(f"deb file://{self.extrarepo} "
+ f"{dist} {repo}\n")
+
+ sources = apt_pkg.SourceList()
+ sources.read_main_list()
+
+ progress = apt.progress.text.AcquireProgress()
+
+ self.cache = apt_pkg.Cache()
+ if init:
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+
+ if self.extrarepo:
+ apt_pkg.config.set("Dir::Etc::SourceList", extrarepo_list)
+ apt_pkg.config.set("APT::Get::List-Cleanup", "0")
+ self.cache.update(progress, sources)
+ self.cache = apt_pkg.Cache()
+ os.remove(extrarepo_list)
+
+ self.depcache = apt_pkg.DepCache(self.cache)
+ self.sr = apt_pkg.SourceRecords()
+
+ ret = True
+
+ if init:
+ self.mark_essential()
+ # 1(required), 2(important), 3(standard), 4(optional), 5(extra)
+ self.mark_by_prio(1)
+
+ pkgs = list(filter(None, ','.join(pkgs).split(',')))
+ if srcmode:
+ self.handle_src_list(set(pkgs))
+ else:
+ ret = self.mark_list(pkgs, crossbuild)
+
+ if dscfile:
+ fobj = open(dscfile, "r")
+
+ try:
+ tagfile = apt_pkg.TagFile(fobj)
+ while tagfile.step() == 1:
+ deps = tagfile.section.get("Build-Depends", "")
+ # Remove extra commas and spaces - apt_pkg.parse_src_depends
+ # doesnt like lines like ", device-tree-compiler"
+ deps = ', '.join(
+ [s.strip() for s in deps.split(',') if s.strip()]
+ )
+ print(f"parsed deps: {deps}")
+ for item in apt_pkg.parse_src_depends(deps, False):
+ pkgname = item[0][0]
+ self.mark_pkg(pkgname, crossbuild)
+
+ finally:
+ fobj.close()
+
+ if not ret:
+ sys.exit("Some of requested packages not found")
+
+ if init or not srcmode:
+ fetcher = apt_pkg.Acquire(progress)
+ pm = apt_pkg.PackageManager(self.depcache)
+
+ recs = apt_pkg.PackageRecords(self.cache)
+ pm.get_archives(fetcher, sources, recs)
+
+ self.handle_repo(fetcher)
+
+
+def parse_arguments():
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--init",
+ default=False, action="store_true",
+ help="initialize context in WORKDIR")
+ parser.add_argument(
+ "--workdir",
+ type=str, required=True,
+ help="work directory storing debrepo context")
+ parser.add_argument(
+ "--aptsrcsfile",
+ type=str, metavar="PATH",
+ help="sources.list file to use when init")
+ parser.add_argument(
+ "--srcmode",
+ default=False, action="store_true",
+ help="add source packages instead of debs")
+ parser.add_argument(
+ "--repodir",
+ type=str, metavar="REPO",
+ help="repository directory")
+ parser.add_argument(
+ "--repodbdir",
+ type=str, metavar="REPODB",
+ help="repository database directory")
+ parser.add_argument(
+ "--extrarepo",
+ type=str, metavar="REPO",
+ help="extra repository to consider")
+ parser.add_argument(
+ "--mirror",
+ type=str,
+ help="use custom distro mirror")
+ parser.add_argument(
+ "--distro",
+ type=str,
+ help="select distro to use")
+ parser.add_argument(
+ "--codename",
+ type=str,
+ help="distro codename")
+ parser.add_argument(
+ "--arch",
+ type=str,
+ help="distro arch")
+ parser.add_argument(
+ "--compatarch",
+ type=str, metavar="ARCH",
+ help="compat arch to use")
+ parser.add_argument(
+ "--crossarch",
+ type=str, metavar="ARCH",
+ help="cross-build arch")
+ parser.add_argument(
+ "--keydir",
+ type=str,
+ help="directory with distro keys")
+ parser.add_argument(
+ "--no-check-gpg",
+ default=False, action="store_true",
+ help="allow insecure repositories")
+ parser.add_argument(
+ "--dscfile",
+ type=str, metavar="PATH",
+ help="Debian source file to parse")
+ parser.add_argument(
+ "--crossbuild",
+ default=False, action="store_true",
+ help="add packages with cross arch")
+
+ parser.add_argument(
+ "packages",
+ nargs='*', type=str,
+ help="space- or comma-separated list of packages to add")
+
+ args = parser.parse_args()
+
+ return args
+
+
+def main():
+ args = parse_arguments()
+
+ if not (args.init or args.packages or args.dscfile):
+ sys.exit("Nothing to do")
+
+ workdir = os.path.abspath(args.workdir)
+ os.makedirs(workdir, exist_ok=True)
+
+ with open(f"{workdir}/debrepo.lock", "a") as file:
+ fcntl.flock(file.fileno(), fcntl.LOCK_EX)
+
+ debrepo = DebRepo(args)
+
+ if args.init:
+ debrepo.create_rootfs(args.aptsrcsfile)
+ debrepo.create_repo_dist()
+
+ debrepo.apt_config(args.init, args.crossbuild)
+ debrepo.apt_run(args.init, args.srcmode, args.packages,
+ args.dscfile, args.crossbuild)
+
+ #Unlock debrepo context
+ fcntl.flock(file.fileno(), fcntl.LOCK_UN)
+
+
+if __name__ == "__main__":
+ main()
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:18 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This patch uses debrepo script to predownload packages to base-apt
repository before they are installed in rootfs.
meta/classes/crossvars.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 27 ++++++++++++++++++-
meta/classes/dpkg.bbclass | 8 ++++++
meta/classes/image-locales-extension.bbclass | 5 ++++
meta/classes/image-tools-extension.bbclass | 13 +++++++++
meta/classes/rootfs.bbclass | 4 +++
.../sbuild-chroot/sbuild-chroot-host.bb | 2 ++
7 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass
index 00326c9a..03a74ebe 100644
--- a/meta/classes/crossvars.bbclass
+++ b/meta/classes/crossvars.bbclass
@@ -27,6 +27,7 @@ python __anonymous() {
schroot_dir = d.getVar('SCHROOT_HOST_DIR', False)
sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build"
sdk_toolchain = "crossbuild-essential-" + distro_arch
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR'))
else:
d.setVar('BUILD_ARCH', distro_arch)
schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..367ea52d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -11,6 +11,7 @@ inherit terminal
inherit repository
inherit deb-dl-dir
inherit essential
+inherit debrepo
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
@@ -114,6 +115,14 @@ do_apt_fetch() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}"
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
schroot -r -c ${session_id} -d / -u root -- \
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
schroot -r -c ${session_id} -d / -- \
@@ -136,18 +145,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"
# Add dependency from the correct schroot: host or target
do_apt_fetch[depends] += "${SCHROOT_DEP}"
+# Debrepo context is created by target bootstrap, need this dependency too
+do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap"
do_apt_unpack() {
rm -rf ${S}
schroot_create_configs
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+ echo "Started session: ${session_id}"
+
schroot_cleanup() {
+ schroot -q -f -e -c ${session_id} > /dev/null 2>&1
schroot_delete_configs
}
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
+ schroot -r -c ${session_id} -d / -- \
sh -c '
set -e
for uri in $2; do
@@ -157,6 +179,9 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"
done' \
my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}"
+
+ # End chroot session
+ schroot -e -c ${session_id}
schroot_delete_configs
}
do_apt_unpack[network] = "${TASK_USE_SUDO}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index bcc3f828..bf3994a6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -111,6 +111,12 @@ dpkg_runbuild() {
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ debrepo_parse_dscfile "${DSC_FILE}"
+
+ locked_update_cmd=":"
+ fi
sbuild -A -n -c ${SBUILD_CHROOT} \
--host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
@@ -122,9 +128,11 @@ dpkg_runbuild() {
--chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="mkdir -p ${deb_dir}" \
--chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \
+ --chroot-setup-commands="${locked_update_cmd}" \
--chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \
+ --finished-build-commands="mkdir -p ${ext_root}" \
--finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index 9149d643..4dd93f1f 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
# This class extends the image.bbclass for setting locales and purging unneeded
# ones.
+inherit debrepo
+
LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
en_US ISO-8859-1\n"
LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
image_install_localepurge_download[weight] = "40"
image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
image_install_localepurge_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
}
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index e8ace8f5..ddb046a8 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -6,6 +6,11 @@
# This file extends the image.bbclass to supply tools for futher imager functions
inherit sbuild
+inherit debrepo
+
+python __anonymous() {
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR'))
+}
IMAGER_INSTALL ??= ""
IMAGER_BUILD_DEPS ??= ""
@@ -40,12 +45,20 @@ imager_run() {
echo "Installing imager deps: ${local_install}"
distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
+
+ debrepo_add_packages --isarapt "${debrepo_workdir}" "${local_install}"
+ debrepo_update_apt_source_list "${schroot_dir}" "base-apt"
+
${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOAPT
schroot -r -c ${session_id} -d / -u root -- sh -c " \
apt-get update \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 7d0bc0c8..f41fb498 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -2,6 +2,7 @@
# Copyright (c) Siemens AG, 2020
inherit deb-dl-dir
+inherit debrepo
ROOTFS_ARCH ?= "${DISTRO_ARCH}"
ROOTFS_DISTRO ?= "${DISTRO}"
@@ -153,6 +154,9 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
rootfs_install_pkgs_download() {
+ debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
index 255d6937..6734eb18 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -24,3 +24,5 @@ SBUILD_CHROOT_PREINSTALL:riscv64 ?= " \
gcc-riscv64-linux-gnu \
g++-riscv64-linux-gnu \
dpkg-cross"
2.44.2
repository before they are installed in rootfs.
meta/classes/crossvars.bbclass | 1 +
meta/classes/dpkg-base.bbclass | 27 ++++++++++++++++++-
meta/classes/dpkg.bbclass | 8 ++++++
meta/classes/image-locales-extension.bbclass | 5 ++++
meta/classes/image-tools-extension.bbclass | 13 +++++++++
meta/classes/rootfs.bbclass | 4 +++
.../sbuild-chroot/sbuild-chroot-host.bb | 2 ++
7 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/meta/classes/crossvars.bbclass b/meta/classes/crossvars.bbclass
index 00326c9a..03a74ebe 100644
--- a/meta/classes/crossvars.bbclass
+++ b/meta/classes/crossvars.bbclass
@@ -27,6 +27,7 @@ python __anonymous() {
schroot_dir = d.getVar('SCHROOT_HOST_DIR', False)
sbuild_dep = "sbuild-chroot-host" + flavor_suffix + ":do_build"
sdk_toolchain = "crossbuild-essential-" + distro_arch
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_HOST_DIR'))
else:
d.setVar('BUILD_ARCH', distro_arch)
schroot_dir = d.getVar('SCHROOT_TARGET_DIR', False)
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 789d6c74..367ea52d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -11,6 +11,7 @@ inherit terminal
inherit repository
inherit deb-dl-dir
inherit essential
+inherit debrepo
DEPENDS ?= ""
RPROVIDES ?= "${PROVIDES}"
@@ -114,6 +115,14 @@ do_apt_fetch() {
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
+ debrepo_add_packages --srcmode "${DEBREPO_TARGET_DIR}" "${SRC_APT}"
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ schroot -r -c ${session_id} -d / -u root -- \
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
schroot -r -c ${session_id} -d / -u root -- \
rm /etc/apt/sources.list.d/isar-apt.list /etc/apt/preferences.d/isar-apt
schroot -r -c ${session_id} -d / -- \
@@ -136,18 +145,31 @@ do_apt_fetch[network] = "${TASK_USE_NETWORK_AND_SUDO}"
# Add dependency from the correct schroot: host or target
do_apt_fetch[depends] += "${SCHROOT_DEP}"
+# Debrepo context is created by target bootstrap, need this dependency too
+do_apt_fetch[depends] += "isar-bootstrap-target:do_bootstrap"
do_apt_unpack() {
rm -rf ${S}
schroot_create_configs
+ session_id=$(schroot -b -c ${SBUILD_CHROOT})
+ echo "Started session: ${session_id}"
+
schroot_cleanup() {
+ schroot -q -f -e -c ${session_id} > /dev/null 2>&1
schroot_delete_configs
}
trap 'exit 1' INT HUP QUIT TERM ALRM USR1
trap 'schroot_cleanup' EXIT
- schroot -d / -c ${SBUILD_CHROOT} -- \
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ schroot -r -c ${session_id} -d / -u root -- \
+ flock -x "${REPO_BASE_DIR}/repo.lock" -c "
+ sh -c 'apt-get -y update -o Dir::Etc::SourceList=\"sources.list.d/base-apt.list\" -o Dir::Etc::SourceParts=\"-\" '
+ "
+ fi
+
+ schroot -r -c ${session_id} -d / -- \
sh -c '
set -e
for uri in $2; do
@@ -157,6 +179,9 @@ do_apt_unpack() {
dpkg-source -x "${dscfile}" "${PPS}"
done' \
my_script "${BASE_DISTRO}-${BASE_DISTRO_CODENAME}" "${SRC_APT}"
+
+ # End chroot session
+ schroot -e -c ${session_id}
schroot_delete_configs
}
do_apt_unpack[network] = "${TASK_USE_SUDO}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index bcc3f828..bf3994a6 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -111,6 +111,12 @@ dpkg_runbuild() {
echo '$stalled_pkg_timeout = ${DPKG_BUILD_TIMEOUT};' >> ${SBUILD_CONFIG}
DSC_FILE=$(find ${WORKDIR} -maxdepth 1 -name "${DEBIAN_SOURCE}_*.dsc" -print)
+ debrepo_parse_dscfile "${DSC_FILE}"
+
+ locked_update_cmd=":"
+ if [ "${ISAR_PREFETCH_BASE_APT}" = "1" ]; then
+ locked_update_cmd="flock -x /base-apt/repo.lock -c 'apt-get -y update'"
+ fi
sbuild -A -n -c ${SBUILD_CHROOT} \
--host=${PACKAGE_ARCH} --build=${BUILD_ARCH} ${profiles} \
@@ -122,9 +128,11 @@ dpkg_runbuild() {
--chroot-setup-commands="rm -f /var/log/dpkg.log" \
--chroot-setup-commands="mkdir -p ${deb_dir}" \
--chroot-setup-commands="find ${ext_deb_dir} -maxdepth 1 -name '*.deb' -exec ln -t ${deb_dir}/ -sf {} +" \
+ --chroot-setup-commands="${locked_update_cmd}" \
--chroot-setup-commands="apt-get update -o Dir::Etc::SourceList=\"sources.list.d/isar-apt.list\" -o Dir::Etc::SourceParts=\"-\" -o APT::Get::List-Cleanup=\"0\"" \
--finished-build-commands="rm -f ${deb_dir}/sbuild-build-depends-main-dummy_*.deb" \
--finished-build-commands="find ${deb_dir} -maxdepth 1 -type f -name '*.deb' -print -exec cp ${CP_FLAGS} -t ${ext_deb_dir}/ {} +" \
+ --finished-build-commands="mkdir -p ${ext_root}" \
--finished-build-commands="cp /var/log/dpkg.log ${ext_root}/dpkg_partial.log" \
--build-dir=${WORKDIR} --dist="isar" ${DSC_FILE}
diff --git a/meta/classes/image-locales-extension.bbclass b/meta/classes/image-locales-extension.bbclass
index 9149d643..4dd93f1f 100644
--- a/meta/classes/image-locales-extension.bbclass
+++ b/meta/classes/image-locales-extension.bbclass
@@ -6,6 +6,8 @@
# This class extends the image.bbclass for setting locales and purging unneeded
# ones.
+inherit debrepo
+
LOCALE_GEN ?= "en_US.UTF-8 UTF-8\n\
en_US ISO-8859-1\n"
LOCALE_DEFAULT ?= "en_US.UTF-8"
@@ -29,6 +31,9 @@ ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT += "image_install_localepurge_download"
image_install_localepurge_download[weight] = "40"
image_install_localepurge_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
image_install_localepurge_download() {
+ debrepo_add_packages "${DEBREPO_WORKDIR}" "localepurge"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only localepurge
}
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index e8ace8f5..ddb046a8 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -6,6 +6,11 @@
# This file extends the image.bbclass to supply tools for futher imager functions
inherit sbuild
+inherit debrepo
+
+python __anonymous() {
+ d.setVar('DEBREPO_WORKDIR', d.getVar('DEBREPO_TARGET_DIR'))
+}
IMAGER_INSTALL ??= ""
IMAGER_BUILD_DEPS ??= ""
@@ -40,12 +45,20 @@ imager_run() {
echo "Installing imager deps: ${local_install}"
distro="${BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ debrepo_workdir=${DEBREPO_TARGET_DIR}
if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
distro="${HOST_BASE_DISTRO}-${BASE_DISTRO_CODENAME}"
+ if [ ${HOST_ARCH} != ${DISTRO_ARCH} ]; then
+ debrepo_workdir=${DEBREPO_HOST_DIR}
+ fi
fi
E="${@ isar_export_proxies(d)}"
deb_dl_dir_import ${schroot_dir} ${distro}
E="${@ isar_export_proxies(d)}"
+
+ debrepo_add_packages --isarapt "${debrepo_workdir}" "${local_install}"
+ debrepo_update_apt_source_list "${schroot_dir}" "base-apt"
+
${SCRIPTSDIR}/lockrun.py -r -f "${REPO_ISAR_DIR}/isar.lock" -s <<EOAPT
schroot -r -c ${session_id} -d / -u root -- sh -c " \
apt-get update \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 7d0bc0c8..f41fb498 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -2,6 +2,7 @@
# Copyright (c) Siemens AG, 2020
inherit deb-dl-dir
+inherit debrepo
ROOTFS_ARCH ?= "${DISTRO_ARCH}"
ROOTFS_DISTRO ?= "${DISTRO}"
@@ -153,6 +154,9 @@ rootfs_install_pkgs_download[weight] = "600"
rootfs_install_pkgs_download[isar-apt-lock] = "release-after"
rootfs_install_pkgs_download[network] = "${TASK_USE_NETWORK_AND_SUDO}"
rootfs_install_pkgs_download() {
+ debrepo_add_packages --isarapt "${DEBREPO_WORKDIR}" "${ROOTFS_PACKAGES}"
+ debrepo_update_apt_source_list "${ROOTFSDIR}" "base-apt"
+
sudo -E chroot '${ROOTFSDIR}' \
/usr/bin/apt-get ${ROOTFS_APT_ARGS} --download-only ${ROOTFS_PACKAGES}
}
diff --git a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
index 255d6937..6734eb18 100644
--- a/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
+++ b/meta/recipes-devtools/sbuild-chroot/sbuild-chroot-host.bb
@@ -24,3 +24,5 @@ SBUILD_CHROOT_PREINSTALL:riscv64 ?= " \
gcc-riscv64-linux-gnu \
g++-riscv64-linux-gnu \
dpkg-cross"
+
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
--
+DEBREPO_WORKDIR = "${DEBREPO_HOST_DIR}"
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:19 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This is mostly related to gitlab CI that migth use an image without
preinstalled python3-apt.
Also, make system python packages available in virtualenv.
scripts/ci_build.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 0aa2403d..b17d4622 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
sudo apt-get update -qq
sudo apt-get install -y virtualenv
rm -rf /tmp/avocado_venv
- virtualenv --python python3 /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv --system-site-packages
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install avocado-framework==103.0
@@ -131,6 +131,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
fi
fi
+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+ sudo apt-get update -qq
+ sudo apt-get install -y python3-apt
+fi
+
# Provide working path
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
--
2.44.2
preinstalled python3-apt.
Also, make system python packages available in virtualenv.
scripts/ci_build.sh | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index 0aa2403d..b17d4622 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -19,7 +19,7 @@ if ! command -v avocado > /dev/null; then
sudo apt-get update -qq
sudo apt-get install -y virtualenv
rm -rf /tmp/avocado_venv
- virtualenv --python python3 /tmp/avocado_venv
+ virtualenv --python python3 /tmp/avocado_venv --system-site-packages
# shellcheck source=/dev/null
source /tmp/avocado_venv/bin/activate
pip install avocado-framework==103.0
@@ -131,6 +131,12 @@ if echo "$TAGS" | grep -Fqive "-startvm"; then
fi
fi
+# install python3-apt
+if [ ! -f /usr/share/doc/python3-apt/copyright ]; then
+ sudo apt-get update -qq
+ sudo apt-get install -y python3-apt
+fi
+
# Provide working path
mkdir -p .config/avocado
cat <<EOF > .config/avocado/avocado.conf
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:19 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
Since all packages and source packages are placed to base-apt repo
during build, there is no need to have them in one more place.
meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 48233a62..327e8be5 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -91,6 +91,7 @@ deb_dl_dir_import() {
export rootfs="${1}"
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
[ ! -d "${pc}" ] && return 0
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
flock -s "${pc}".lock sudo -Es << 'EOSUDO'
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -108,6 +109,7 @@ deb_dl_dir_export() {
export rootfs="${1}"
export owner=$(id -u):$(id -g)
mkdir -p "${pc}"
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
isar_debs="$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \
"find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")"
--
2.44.2
during build, there is no need to have them in one more place.
meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 48233a62..327e8be5 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -91,6 +91,7 @@ deb_dl_dir_import() {
export rootfs="${1}"
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
[ ! -d "${pc}" ] && return 0
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
flock -s "${pc}".lock sudo -Es << 'EOSUDO'
set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x
@@ -108,6 +109,7 @@ deb_dl_dir_export() {
export rootfs="${1}"
export owner=$(id -u):$(id -g)
mkdir -p "${pc}"
+ [ "${ISAR_PREFETCH_BASE_APT}" = "1" ] && return 0
isar_debs="$(${SCRIPTSDIR}/lockrun.py -r -f '${REPO_ISAR_DIR}/isar.lock' -c \
"find '${REPO_ISAR_DIR}/${DISTRO}' -name '*.deb' -print")"
--
2.44.2
Uladzimir Bely
Jul 25, 2024, 11:10:19 AM (3 days ago) Jul 25
to isar-...@googlegroups.com
This option allows to set ISAR_PREFETCH_BASE_APT to "0" or "1" and
choose between old and new base-apt behaviour.
Docker image kas uses should have "python3-apt" preinstalled
in order to have new functionality working.
kas/opt/Kconfig | 13 +++++++++++++
kas/opt/prefetch-base-apt.yaml | 9 +++++++++
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index 701bd263..e1476b8a 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -157,3 +157,16 @@ config KAS_INCLUDE_CACHE_DEB_SRC
string
default "kas/opt/cache-deb-src.yaml"
depends on CACHE_DEB_SRC
+
+
+config PREFETCH_BASE_APT
+ bool "Prefetch base-apt repo"
+ default y
+ help
+ This makse Isar always take packages from base-apt repository where they
+ are prefetched by debrepo script before requested.
+
+config KAS_INCLUDE_PREFETCH_BASE_APT
+ string
+ default "kas/opt/prefetch-base-apt.yaml"
+ depends on PREFETCH_BASE_APT
diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml
new file mode 100644
index 00000000..0cbdb93f
--- /dev/null
+++ b/kas/opt/prefetch-base-apt.yaml
@@ -0,0 +1,9 @@
+ version: 14
+
+local_conf_header:
+ prefetch-base-apt: |
+ ISAR_PREFETCH_BASE_APT = "1"
--
2.44.2
choose between old and new base-apt behaviour.
Docker image kas uses should have "python3-apt" preinstalled
in order to have new functionality working.
kas/opt/Kconfig | 13 +++++++++++++
kas/opt/prefetch-base-apt.yaml | 9 +++++++++
2 files changed, 22 insertions(+)
create mode 100644 kas/opt/prefetch-base-apt.yaml
diff --git a/kas/opt/Kconfig b/kas/opt/Kconfig
index 701bd263..e1476b8a 100644
--- a/kas/opt/Kconfig
+++ b/kas/opt/Kconfig
@@ -157,3 +157,16 @@ config KAS_INCLUDE_CACHE_DEB_SRC
string
default "kas/opt/cache-deb-src.yaml"
depends on CACHE_DEB_SRC
+
+
+config PREFETCH_BASE_APT
+ bool "Prefetch base-apt repo"
+ default y
+ help
+ This makse Isar always take packages from base-apt repository where they
+ are prefetched by debrepo script before requested.
+
+config KAS_INCLUDE_PREFETCH_BASE_APT
+ string
+ default "kas/opt/prefetch-base-apt.yaml"
+ depends on PREFETCH_BASE_APT
diff --git a/kas/opt/prefetch-base-apt.yaml b/kas/opt/prefetch-base-apt.yaml
new file mode 100644
index 00000000..0cbdb93f
--- /dev/null
+++ b/kas/opt/prefetch-base-apt.yaml
@@ -0,0 +1,9 @@
+# This software is a part of Isar.
+# Copyright (C) 2024 ilbers GmbH
+
+header:
+# Copyright (C) 2024 ilbers GmbH
+
+ version: 14
+
+local_conf_header:
+ prefetch-base-apt: |
+ ISAR_PREFETCH_BASE_APT = "1"
--
2.44.2
Reply all
Reply to author
Forward
0 new messages