[PATCH v6 00/13] Deb-src caching

Vijai Kumar K

unread,

Sep 30, 2020, 2:03:54 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Changes in v6:
- Rebase against latest next
- Address review comments from Henning
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3

Changes in v5:
The major changes in this series are as below,
- (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
of debs from different distro which inturn helps the src caching logic.
- (P5) Avoid downloading package from other builds.
- (P8) Add a reprepro based sanity test to check if the repo contains
the sources for all the debs.
- (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
to enable or disable various base-apt features. (cache-deb-src for now)
- Some fixes in (P12 & P13)
Also, addressed some review comments from Henning.
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5

Changes in v4:
- Use <source package>=<version> format instead of just using <packagename>
to download the right version of source package.

Changes in v3:
- Take care of non-existent downloads/deb-src directory.

Changes in v2:
- Introduced additional patch to cache deb src
- Rebased on top of henning/staging4 tree

Vijai Kumar K (13):
rootfs: Make rootfs finalize a separate task
deb-dl-dir: Cache host distro debs separately
meta: cache deb srcs as part of postprocessing
deb-dl-dir: Make debsrc_download faster
deb-dl-dir: Download files only belonging to the current image
deb-dl-dir: Factor out the mounting part
deb-dl-dir: Fix skipping of removed files
repository: Add a sanity test to check missing sources
base-apt: Introduce BASE_REPO_FEATURES
repository: Fix failures due to missing section
scripts/ci_build.sh: Enable deb-src caching
rootfs: Fix possible overwrite of existing resolv.conf
rootfs: Handle failures when postprocess is rerun

meta-isar/conf/local.conf.sample | 4 ++
meta/classes/deb-dl-dir.bbclass | 61 +++++++++++++++++-
meta/classes/dpkg-gbp.bbclass | 8 ++-
meta/classes/dpkg.bbclass | 8 ++-
meta/classes/image-tools-extension.bbclass | 4 +-
meta/classes/image.bbclass | 43 ++++++++++++-
meta/classes/repository.bbclass | 13 +++-
meta/classes/rootfs.bbclass | 62 +++++++------------
.../isar-bootstrap/isar-bootstrap.inc | 12 +++-
meta/recipes-devtools/base-apt/base-apt.bb | 3 +
scripts/ci_build.sh | 1 +
11 files changed, 166 insertions(+), 53 deletions(-)

--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:03:56 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

With the current implementation it is difficult to append a
postprocess function which requires a chroot environment.
For example, to add a postprocess function which runs apt-get to
download all source of packages installed in the target.

rootfs_postprocess_finalize is not actually an optional feature
but instead a necessary cleanup function for image class.
So, move the implementation to image class and make it as a task.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---
meta/classes/image.bbclass | 41 ++++++++++++++++++++++++++++++++++++-
meta/classes/rootfs.bbclass | 39 -----------------------------------
2 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a296cc0..2391529 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

@@ -173,5 +173,44 @@ python do_deploy() {
}
addtask deploy before do_build after do_image

+do_rootfs_finalize() {
+ sudo -s <<'EOSUDO'
+ test -e "${ROOTFSDIR}/chroot-setup.sh" && \
+ "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
+ rm -f "${ROOTFSDIR}/chroot-setup.sh"
+
+ test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
+ find "${ROOTFSDIR}/usr/bin" \
+ -maxdepth 1 -name 'qemu-*-static' -type f -delete
+
+ mountpoint -q '${ROOTFSDIR}/isar-apt' && \
+ umount -l ${ROOTFSDIR}/isar-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
+
+ mountpoint -q '${ROOTFSDIR}/base-apt' && \
+ umount -l ${ROOTFSDIR}/base-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
+
+ mountpoint -q '${ROOTFSDIR}/dev' && \
+ umount -l ${ROOTFSDIR}/dev
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/proc
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/sys
+
+ rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
+ rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+
+ mv "${ROOTFSDIR}/etc/apt/sources-list" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources-list"
+EOSUDO
+}
+addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
+
# Last so that the image type can overwrite tasks if needed
inherit ${IMAGE_TYPE}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cb..eae14d5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -201,45 +201,6 @@ rootfs_generate_manifest () {
${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
}

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
-rootfs_postprocess_finalize() {
- sudo -s <<'EOSUDO'
- test -e "${ROOTFSDIR}/chroot-setup.sh" && \
- "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
- rm -f "${ROOTFSDIR}/chroot-setup.sh"
-
- test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
- find "${ROOTFSDIR}/usr/bin" \
- -maxdepth 1 -name 'qemu-*-static' -type f -delete
-
- mountpoint -q '${ROOTFSDIR}/isar-apt' && \
- umount -l ${ROOTFSDIR}/isar-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
-
- mountpoint -q '${ROOTFSDIR}/base-apt' && \
- umount -l ${ROOTFSDIR}/base-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
-
- mountpoint -q '${ROOTFSDIR}/dev' && \
- umount -l ${ROOTFSDIR}/dev
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/proc
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/sys
-
- rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
- rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
- mv "${ROOTFSDIR}/etc/apt/sources-list" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources-list"
-EOSUDO
-}
-
do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
python do_rootfs_postprocess() {
# Take care that its correctly mounted:
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:03:59 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
we were still caching the debs from both the distros into the same
DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be
cached in the relevant subdirectory and avoids mixing of debs from
two separate distros.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 4 ++--
meta/classes/dpkg-gbp.bbclass | 8 ++++++--
meta/classes/dpkg.bbclass | 8 ++++++--
meta/classes/image-tools-extension.bbclass | 4 ++--
meta/classes/rootfs.bbclass | 4 ++--
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++--
6 files changed, 28 insertions(+), 12 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e996703..29a3d67 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -6,7 +6,7 @@
inherit repository

deb_dl_dir_import() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
[ ! -d "${pc}" ] && return 0
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -20,7 +20,7 @@ deb_dl_dir_import() {
}

deb_dl_dir_export() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
mkdir -p "${pc}"
flock "${pc}".lock -c '
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index afa1e19..ba5c3eb 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

do_install_builddeps_append() {
dpkg_do_mounts
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a24386d..4e7c2f7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,10 +9,14 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
do_install_builddeps() {
dpkg_do_mounts
E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH} --download-only
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH}
dpkg_undo_mounts
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 6590ee7..0b067ff 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,7 +25,7 @@ do_install_imager_deps() {
buildchroot_do_mounts

E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import ${BUILDCHROOT_DIR}
+ deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get update \
-o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
@@ -35,7 +35,7 @@ do_install_imager_deps() {
--allow-unauthenticated --allow-downgrades --download-only install \
${IMAGER_INSTALL}'

- deb_dl_dir_export ${BUILDCHROOT_DIR}
+ deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
--allow-unauthenticated --allow-downgrades install \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index eae14d5..6316321 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -115,7 +115,7 @@ rootfs_install_resolvconf() {
ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
rootfs_import_package_cache[weight] = "5"
rootfs_import_package_cache() {
- deb_dl_dir_import ${ROOTFSDIR}
+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
}

ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
@@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
rootfs_export_package_cache[weight] = "5"
rootfs_export_package_cache() {
- deb_dl_dir_export ${ROOTFSDIR}
+ deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO}
}

ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index fbfe669..3b19914 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -255,7 +255,11 @@ isar_bootstrap() {
export IS_HOST debootstrap_args E
if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
sudo rm -rf --one-file-system "${ROOTFSDIR}"
- deb_dl_dir_import "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+ fi

sudo -E -s <<'EOSUDO'
set -e
@@ -356,7 +360,11 @@ isar_bootstrap() {
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
fi
- deb_dl_dir_export "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
+ fi
}

CLEANFUNCS = "clean_deploy"
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:04:02 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Collect the deb sources of the corresponding deb binaries cached
in DEBDIR as part of image postprocess.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 27 +++++++++++++++++++++++++++
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 8 ++++++++
3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 29a3d67..659fe4b 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,33 @@

inherit repository

+debsrc_download() {
+ export rootfs="$1"
+ export rootfs_distro="$2"
+ mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+ sudo -E -s <<'EOSUDO'
+ mkdir -p "${rootfs}/deb-src"
+ mountpoint -q "${rootfs}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
+EOSUDO
+ ( flock 9
+ set -e
+ printenv | grep -q BB_VERBOSE_LOGS && set -x
+ find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ done
+ ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
+ sudo -E -s <<'EOSUDO'
+ mountpoint -q "${rootfs}/deb-src" && \
+ umount -l "${rootfs}/deb-src"
+ rm -rf "${rootfs}/deb-src"
+EOSUDO
+}
+
deb_dl_dir_import() {

export pc="${DEBDIR}/${2}"
export rootfs="${1}"

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2391529..6ca5759 100644

--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"

-ROOTFS_FEATURES += "clean-package-cache generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 6316321..cfabeae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,6 +185,14 @@ python do_rootfs_install() {
}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
+cache_deb_src() {
+ rootfs_install_resolvconf

+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

+ debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
+ rootfs_install_clean_files
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"
rootfs_postprocess_clean_package_cache() {
sudo -E chroot '${ROOTFSDIR}' \
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:07:27 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough apt-get source skips redownloading of files, it is still
slow and takes a lot of time. Instead, lookup if the dsc file is already
present in the cache and skip based on it.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 659fe4b..79a9a88 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -20,6 +20,8 @@ EOSUDO

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"

local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
+ [ -n "$dscfile" ] && continue

sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:07:34 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Avoid downloading deb-srcs for debs cached from other image builds.
One way to ensure that is to see if the package is present in the dpkg
status file.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 79a9a88..ce691cf 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,15 @@

inherit repository

+is_not_part_of_current_build() {
+ local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
+ local output="$( grep -hs "^Package: ${package}" \
+ "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ [ -z "${output}" ]
+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"
@@ -18,6 +27,7 @@ EOSUDO
set -e

printenv | grep -q BB_VERBOSE_LOGS && set -x

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

+ is_not_part_of_current_build "${package}" && continue

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")

--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:07:41 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Factor out the mount and unmount section to separate function for
easy readability. No functional change intended.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 32 ++++++++++++++++++++++----------
1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ce691cf..b2d94e7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,15 +14,30 @@ is_not_part_of_current_build() {
[ -z "${output}" ]
}

+debsrc_do_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+EOSUDO
+}
+
+debsrc_undo_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" && \
+ umount -l "${1}/deb-src"
+ rm -rf "${1}/deb-src"
+EOSUDO

+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"

mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
- sudo -E -s <<'EOSUDO'
- mkdir -p "${rootfs}/deb-src"
- mountpoint -q "${rootfs}/deb-src" || \
- mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_do_mounts "${rootfs}"
+
( flock 9
set -e

printenv | grep -q BB_VERBOSE_LOGS && set -x

@@ -37,11 +52,8 @@ EOSUDO

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

done

) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

- sudo -E -s <<'EOSUDO'
- mountpoint -q "${rootfs}/deb-src" && \
- umount -l "${rootfs}/deb-src"
- rm -rf "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_undo_mounts "${rootfs}"
}

deb_dl_dir_import() {
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:07:49 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Some packages are installed and then removed, like localepurge from
image-locales-extension.bbclass. Those information would not be
available in dpkg status file. Use dpkg log instead to see if the
package has been used on the target.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b2d94e7..a94fb10 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -7,10 +7,16 @@ inherit repository

is_not_part_of_current_build() {
local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
- local output="$( grep -hs "^Package: ${package}" \
- "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
- "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
- "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
+ local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
+ # Since we are parsing all the debs in DEBDIR, we can to some extend
+ # try to eliminate some debs that are not part of the current multiconfig
+ # build using the below method.
+ local output="$( grep -hs "status installed ${package}:${arch} ${version}" \
+ "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+ "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
+ "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
+
[ -z "${output}" ]
}

--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:07:52 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Add a sanity test routine to test if sources for all packages
are present in the repo.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 10 ++++++++++
meta/recipes-devtools/base-apt/base-apt.bb | 2 ++
2 files changed, 12 insertions(+)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index c70138f..7b6e47b 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -96,3 +96,13 @@ repo_contains_package() {
# no
return 2
}
+
+repo_sanity_test() {
+ local dir="$1"
+ local dbdir="$2"
+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then
+ bbwarn "One or more sources are missing in base-apt"
+ bbwarn "${output}"
+ fi
+}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index da3e32e..8940ae8 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -66,6 +66,8 @@ repo() {
fi

populate_base_apt
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
}

python do_cache() {
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:19:01 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough we are collecting the debsrcs as part of postprocess,
it could not be considered a ROOTFS_FEATURE, instead a base-apt one.
Introduce BASE_REPO_FEATURES, to provide user with control to enable
or disable cache-deb-src. Disabled by default, since it is not required
for normal offline build to work.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 4 +++-
meta/classes/repository.bbclass | 9 +++++----
meta/classes/rootfs.bbclass | 1 -
meta/recipes-devtools/base-apt/base-apt.bb | 1 +
5 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 07a7781..ec0a384 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -181,6 +181,10 @@ ISAR_CROSS_COMPILE ?= "0"
# NOTE: this works for amd64 and arm64 targets so far
#ISAR_ENABLE_COMPAT_ARCH ?= "1"

+# Uncomment this to enable caching of all source packages.
+# Without this feature, only sources of packages downloaded with apt:// are downloaded.
+#BASE_REPO_FEATURES ?= "cache-deb-src"
+
#
# Uncomment this to enable use of cached base repository
#ISAR_USE_CACHED_BASE_REPO ?= "1"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6ca5759..5c2c88c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,10 +63,12 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)} "
+
inherit rootfs
inherit image-sdk-extension
inherit image-tools-extension
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 7b6e47b..1f475dc 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -100,9 +100,10 @@ repo_contains_package() {
repo_sanity_test() {
local dir="$1"
local dbdir="$2"
- local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
- if [ -n "${output}" ]; then
- bbwarn "One or more sources are missing in base-apt"
- bbwarn "${output}"
+ if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ];then

+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then

+ bbfatal "One or more sources are missing in repo. ${output}"
+ fi
fi
}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cfabeae..0aa5502 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,7 +185,6 @@ python do_rootfs_install() {

}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
cache_deb_src() {
rootfs_install_resolvconf

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 8940ae8..506a28f 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -10,6 +10,7 @@ SRC_URI = "file://distributions.in"

BASE_REPO_KEY ?= ""
KEYFILES ?= ""
+BASE_REPO_FEATURES ?= ""

populate_base_apt() {
find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:19:07 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

reprepro's includedsc failed for certain packages like makedev while
complaining about missing Section information. This information is
optional according to Debian[1]. Use a dummy value to avoid such
failures.

[1] https://wiki.debian.org/DebianRepository/Format

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 1f475dc..ac395db 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -42,7 +42,7 @@ repo_add_srcpackage() {
if [ -n "${GNUPGHOME}" ]; then
export GNUPGHOME="${GNUPGHOME}"
fi
- reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
+ reprepro -b "${dir}" --dbdir "${dbdir}" -C main -S - -P source \
includedsc "${codename}" \
"$@"
}
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:19:17 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

Enable debsrc caching for CI builds.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

scripts/ci_build.sh | 1 +
1 file changed, 1 insertion(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a5eee82..09a0b4b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -184,6 +184,7 @@ if [ -n "$REPRO_BUILD" ]; then
sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
fi

+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
# Start cross build for the defined set of configurations
sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
bitbake $BB_ARGS $CROSS_TARGETS_SET
--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:19:17 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

There is a possiblilty that one of the packages installed in the
rootfs provides /etc/resolv.conf and we might accidentally remove
it. Fix it by taking a backup of any existing resolv conf files
and restoring it later. This is needed since we could not effectively
move caching before rootfs_install_clean_files since we need the
latest dpkg log.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0aa5502..ed43fb9 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -186,10 +186,18 @@ python do_rootfs_install() {

addtask rootfs_install before do_rootfs_postprocess after do_unpack

cache_deb_src() {
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
+ fi
rootfs_install_resolvconf
+

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

- rootfs_install_clean_files
+
+ sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf.isar "${ROOTFSDIR}"/etc/resolv.conf
+ fi

}

ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"

--
2.17.1

Vijai Kumar K

unread,

Sep 30, 2020, 2:21:46 AM9/30/20

to isar-...@googlegroups.com, Vijai Kumar K

The apt state information in var/lib/apt/lists is cleared as
part of postprocessing. This makes apt-get calls in cache_deb_src
fail when rerunning the postprocess task.

Since we cannot run apt-get update again to refresh the state
information, copy the apt state information from the initial
bootstrapped image.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ed43fb9..bebc0c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -190,6 +190,10 @@ cache_deb_src() {

sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar

fi
rootfs_install_resolvconf
+ # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
+ # relies on that through out the build. Copy that state information instead of apt-get update
+ # which generates a new state from upstream.
+ sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

--
2.17.1

vijaikumar....@gmail.com

unread,

Nov 23, 2020, 10:08:18 AM11/23/20

to isar-users

Any review comment on this series?

Thanks,

Vijai Kumar K

Jan Kiszka

unread,

Nov 23, 2020, 10:38:37 AM11/23/20

to vijaikumar....@gmail.com, isar-users

On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
>
> Any review comment on this series?
>

Do we have this anchored in CI via some tests already?

Jan

> meta/recipes-devtools/base-apt/base-apt.bb <http://base-apt.bb> | 3 +

> scripts/ci_build.sh | 1 +
> 11 files changed, 166 insertions(+), 53 deletions(-)
>
> --
> 2.17.1
>

> --
> You received this message because you are subscribed to the Google
> Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to isar-users+...@googlegroups.com
> <mailto:isar-users+...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com
> <https://groups.google.com/d/msgid/isar-users/bf7f91cf-250b-4807-9ad4-110704f89cd3n%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

vijaikumar....@gmail.com

unread,

Nov 23, 2020, 10:47:39 AM11/23/20

to isar-users

On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:

On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
>
> Any review comment on this series?
>

Do we have this anchored in CI via some tests already?

Hi Jan,

Yes. We have it enabled in CI[1].

[1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ

Thanks,

Vijai Kumar K

Jan Kiszka

unread,

Nov 24, 2020, 12:41:52 AM11/24/20

to vijaikumar....@gmail.com, isar-users

On 23.11.20 16:47, vijaikumar....@gmail.com wrote:
>
>
> On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:
>
> On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
> >
> > Any review comment on this series?
> >
>
> Do we have this anchored in CI via some tests already?
>
>
> Hi Jan,
>
> Yes. We have it enabled in CI[1].
>
> [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ
>

OK, it's executed, and my one runs in our nightly builds didn't fail.
But is caching effectively checked this way?

Jan

vijaikumar....@gmail.com

unread,

Nov 24, 2020, 1:30:36 AM11/24/20

to isar-users

On Tuesday, November 24, 2020 at 11:11:52 AM UTC+5:30 Jan Kiszka wrote:

On 23.11.20 16:47, vijaikumar....@gmail.com wrote:
>
>
> On Monday, November 23, 2020 at 9:08:37 PM UTC+5:30 Jan Kiszka wrote:
>
> On 23.11.20 16:08, vijaikumar....@gmail.com wrote:
> >
> > Any review comment on this series?
> >
>
> Do we have this anchored in CI via some tests already?
>
>
> Hi Jan,
>
> Yes. We have it enabled in CI[1].
>
> [1]https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/k0zKlMVhBAAJ
>

OK, it's executed, and my one runs in our nightly builds didn't fail.
But is caching effectively checked this way?

Yes. There is a sanity test[1] after base-apt creation which doesnot throw a warning. It should when some sources

are missing.

[1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/LeacMMNhBAAJ

Thanks,

Vijai Kumar K

Baurzhan Ismagulov

unread,

Nov 24, 2020, 2:45:30 PM11/24/20

to isar-...@googlegroups.com

Hello Vijai Kumar,

On Wed, Sep 30, 2020 at 11:32:46AM +0530, Vijai Kumar K wrote:
> Changes in v6:
> - Rebase against latest next
> - Address review comments from Henning
> Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3

The code looks good to me. There is a couple of small issues, I'll reply to the
respective patches.

How do I use this? I've bitbake mc:qemuamd64-buster:isar-image-base, see
downloads/deb-src/debian-buster, which contains only hello. Is this intended?

With kind regards,
Baurzhan.

Baurzhan Ismagulov

unread,

Nov 24, 2020, 2:47:20 PM11/24/20

to isar-...@googlegroups.com

On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote:
> diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
> index afec1cb..eae14d5 100644
> --- a/meta/classes/rootfs.bbclass
> +++ b/meta/classes/rootfs.bbclass
> @@ -201,45 +201,6 @@ rootfs_generate_manifest () {
> ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> }
>
> -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"

Suggest also removing the 'finalize-rootfs' comment before ROOTFS_FEATURES ?=
"".

With kind regards,
Baurzhan.

Baurzhan Ismagulov

unread,

Nov 24, 2020, 2:49:36 PM11/24/20

to isar-...@googlegroups.com

On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 2391529..6ca5759 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -63,7 +63,7 @@ image_do_mounts() {
> }
>
> ROOTFSDIR = "${IMAGE_ROOTFS}"
> -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"

Should it be "cache-deb-src"? The same for p9.

With kind regards,
Baurzhan.

Jan Kiszka

unread,

Nov 25, 2020, 12:51:20 AM11/25/20

to isar-...@googlegroups.com

I would expect that everything a build pulled from deb-src goes into the
cache. IIRC, only hello is rebuilt from sources in these runs.

I wonder, though, and that might be a topic for some user manual
extension, how this series can also be used to fetch all sources
corresponding to all deployed binaries (compliance use case). Or does
this require extra logic?

vijaikumar....@gmail.com

unread,

Nov 25, 2020, 1:04:24 AM11/25/20

to isar-users

Hi Baurzhan,

You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf. By default its disabled. Not everyone

want to cache the deb sources. It takes a lot of time.

[1] https://groups.google.com/g/isar-users/c/on_PWru5Os8/m/_ofw8MFhBAAJ

Thanks,

Vijai Kumar K

With kind regards,
Baurzhan.

Jan Kiszka

unread,

Nov 25, 2020, 1:48:40 AM11/25/20

to vijaikumar....@gmail.com, isar-users

Please add a documentation patch to your series to explain all use cases
and knobs of it. See also my other reply.

vijaikumar....@gmail.com

unread,

Nov 25, 2020, 2:16:48 AM11/25/20

to isar-users

Yes Jan. I am currently adding that in usermanual. Will send the patch soon.

Thanks,

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:27 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Changes in v7:
- Document details in user manual.

Changes in v6:
- Rebase against latest next
- Address review comments from Henning
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc6.3

Changes in v5:
The major changes in this series are as below,
- (P2) Handle cases where HOST_DISTRO!=DISTRO. This avoids mixing
of debs from different distro which inturn helps the src caching logic.
- (P5) Avoid downloading package from other builds.
- (P8) Add a reprepro based sanity test to check if the repo contains
the sources for all the debs.
- (P9) Introduce a new variable BASE_REPO_FEATURES which provides means
to enable or disable various base-apt features. (cache-deb-src for now)
- Some fixes in (P12 & P13)
Also, addressed some review comments from Henning.
Git Tree: https://github.com/vj-kumar/isar/tree/vijai/debsrc5

Changes in v4:
- Use <source package>=<version> format instead of just using <packagename>
to download the right version of source package.

Changes in v3:
- Take care of non-existent downloads/deb-src directory.

Changes in v2:
- Introduced additional patch to cache deb src
- Rebased on top of henning/staging4 tree

Vijai Kumar K (14):

rootfs: Make rootfs finalize a separate task
deb-dl-dir: Cache host distro debs separately
meta: cache deb srcs as part of postprocessing
deb-dl-dir: Make debsrc_download faster
deb-dl-dir: Download files only belonging to the current image
deb-dl-dir: Factor out the mounting part
deb-dl-dir: Fix skipping of removed files
repository: Add a sanity test to check missing sources
base-apt: Introduce BASE_REPO_FEATURES
repository: Fix failures due to missing section
scripts/ci_build.sh: Enable deb-src caching
rootfs: Fix possible overwrite of existing resolv.conf
rootfs: Handle failures when postprocess is rerun

doc/user_manual.md: Document details about deb-src caching

doc/user_manual.md | 25 ++++++++

meta/recipes-devtools/base-apt/base-apt.bb | 3 +
scripts/ci_build.sh | 1 +
12 files changed, 191 insertions(+), 53 deletions(-)

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:29 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

With the current implementation it is difficult to append a
postprocess function which requires a chroot environment.
For example, to add a postprocess function which runs apt-get to
download all source of packages installed in the target.

rootfs_postprocess_finalize is not actually an optional feature
but instead a necessary cleanup function for image class.
So, move the implementation to image class and make it as a task.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/image.bbclass | 41 ++++++++++++++++++++++++++++++++++++-
meta/classes/rootfs.bbclass | 39 -----------------------------------
2 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a296cc0..2391529 100644

--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"

-ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

@@ -173,5 +173,44 @@ python do_deploy() {
}
addtask deploy before do_build after do_image

+do_rootfs_finalize() {
+ sudo -s <<'EOSUDO'
+ test -e "${ROOTFSDIR}/chroot-setup.sh" && \
+ "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
+ rm -f "${ROOTFSDIR}/chroot-setup.sh"
+
+ test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
+ find "${ROOTFSDIR}/usr/bin" \
+ -maxdepth 1 -name 'qemu-*-static' -type f -delete
+
+ mountpoint -q '${ROOTFSDIR}/isar-apt' && \
+ umount -l ${ROOTFSDIR}/isar-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
+
+ mountpoint -q '${ROOTFSDIR}/base-apt' && \
+ umount -l ${ROOTFSDIR}/base-apt
+ rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
+
+ mountpoint -q '${ROOTFSDIR}/dev' && \
+ umount -l ${ROOTFSDIR}/dev
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/proc
+ mountpoint -q '${ROOTFSDIR}/sys' && \
+ umount -l ${ROOTFSDIR}/sys
+
+ rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
+ rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
+ rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+
+ mv "${ROOTFSDIR}/etc/apt/sources-list" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+
+ rm -f "${ROOTFSDIR}/etc/apt/sources-list"
+EOSUDO
+}
+addtask rootfs_finalize before do_rootfs after do_rootfs_postprocess
+
# Last so that the image type can overwrite tasks if needed
inherit ${IMAGE_TYPE}

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cb..eae14d5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -201,45 +201,6 @@ rootfs_generate_manifest () {
${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
}

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"

-rootfs_postprocess_finalize() {
- sudo -s <<'EOSUDO'
- test -e "${ROOTFSDIR}/chroot-setup.sh" && \
- "${ROOTFSDIR}/chroot-setup.sh" "cleanup" "${ROOTFSDIR}"
- rm -f "${ROOTFSDIR}/chroot-setup.sh"
-
- test ! -e "${ROOTFSDIR}/usr/share/doc/qemu-user-static" && \
- find "${ROOTFSDIR}/usr/bin" \
- -maxdepth 1 -name 'qemu-*-static' -type f -delete
-
- mountpoint -q '${ROOTFSDIR}/isar-apt' && \
- umount -l ${ROOTFSDIR}/isar-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/isar-apt
-
- mountpoint -q '${ROOTFSDIR}/base-apt' && \
- umount -l ${ROOTFSDIR}/base-apt
- rmdir --ignore-fail-on-non-empty ${ROOTFSDIR}/base-apt
-
- mountpoint -q '${ROOTFSDIR}/dev' && \
- umount -l ${ROOTFSDIR}/dev
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/proc
- mountpoint -q '${ROOTFSDIR}/sys' && \
- umount -l ${ROOTFSDIR}/sys
-
- rm -f "${ROOTFSDIR}/etc/apt/apt.conf.d/55isar-fallback.conf"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/isar-apt.list"
- rm -f "${ROOTFSDIR}/etc/apt/preferences.d/isar-apt"
- rm -f "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
-
- mv "${ROOTFSDIR}/etc/apt/sources-list" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
-
- rm -f "${ROOTFSDIR}/etc/apt/sources-list"
-EOSUDO
-}
-
do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
python do_rootfs_postprocess() {
# Take care that its correctly mounted:
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:33 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
we were still caching the debs from both the distros into the same
DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be
cached in the relevant subdirectory and avoids mixing of debs from
two separate distros.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 4 ++--
meta/classes/dpkg-gbp.bbclass | 8 ++++++--
meta/classes/dpkg.bbclass | 8 ++++++--
meta/classes/image-tools-extension.bbclass | 4 ++--
meta/classes/rootfs.bbclass | 4 ++--
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 12 ++++++++++--
6 files changed, 28 insertions(+), 12 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e996703..29a3d67 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -6,7 +6,7 @@
inherit repository

deb_dl_dir_import() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
[ ! -d "${pc}" ] && return 0
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -20,7 +20,7 @@ deb_dl_dir_import() {
}

deb_dl_dir_export() {
- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"
mkdir -p "${pc}"
flock "${pc}".lock -c '
diff --git a/meta/classes/dpkg-gbp.bbclass b/meta/classes/dpkg-gbp.bbclass
index afa1e19..ba5c3eb 100644
--- a/meta/classes/dpkg-gbp.bbclass
+++ b/meta/classes/dpkg-gbp.bbclass
@@ -12,11 +12,15 @@ GBP_EXTRA_OPTIONS ?= "--git-pristine-tar"

do_install_builddeps_append() {
dpkg_do_mounts
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends --download-only ${GBP_DEPENDS}
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} \
apt-get install -y -o Debug::pkgProblemResolver=yes \
--no-install-recommends ${GBP_DEPENDS}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index a24386d..4e7c2f7 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -9,10 +9,14 @@ PACKAGE_ARCH ?= "${DISTRO_ARCH}"
do_install_builddeps() {
dpkg_do_mounts
E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import "${BUILDCHROOT_DIR}"
+ distro="${DISTRO}"
+ if [ ${ISAR_CROSS_COMPILE} -eq 1 ]; then
+ distro="${HOST_DISTRO}"
+ fi
+ deb_dl_dir_import "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH} --download-only
- deb_dl_dir_export "${BUILDCHROOT_DIR}"
+ deb_dl_dir_export "${BUILDCHROOT_DIR}" "${distro}"
sudo -E chroot ${BUILDCHROOT_DIR} /isar/deps.sh \
${PP}/${PPS} ${PACKAGE_ARCH}
dpkg_undo_mounts
diff --git a/meta/classes/image-tools-extension.bbclass b/meta/classes/image-tools-extension.bbclass
index 6590ee7..0b067ff 100644
--- a/meta/classes/image-tools-extension.bbclass
+++ b/meta/classes/image-tools-extension.bbclass
@@ -25,7 +25,7 @@ do_install_imager_deps() {
buildchroot_do_mounts

E="${@ isar_export_proxies(d)}"
- deb_dl_dir_import ${BUILDCHROOT_DIR}
+ deb_dl_dir_import ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get update \
-o Dir::Etc::SourceList="sources.list.d/isar-apt.list" \
@@ -35,7 +35,7 @@ do_install_imager_deps() {
--allow-unauthenticated --allow-downgrades --download-only install \
${IMAGER_INSTALL}'

- deb_dl_dir_export ${BUILDCHROOT_DIR}
+ deb_dl_dir_export ${BUILDCHROOT_DIR} ${DISTRO}
sudo -E chroot ${BUILDCHROOT_DIR} sh -c ' \
apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y \
--allow-unauthenticated --allow-downgrades install \
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index eae14d5..6316321 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -115,7 +115,7 @@ rootfs_install_resolvconf() {
ROOTFS_INSTALL_COMMAND += "rootfs_import_package_cache"
rootfs_import_package_cache[weight] = "5"
rootfs_import_package_cache() {
- deb_dl_dir_import ${ROOTFSDIR}
+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}
}

ROOTFS_INSTALL_COMMAND += "rootfs_install_pkgs_download"
@@ -132,7 +132,7 @@ ROOTFS_INSTALL_COMMAND += "${ROOTFS_INSTALL_COMMAND_BEFORE_EXPORT}"
ROOTFS_INSTALL_COMMAND += "rootfs_export_package_cache"
rootfs_export_package_cache[weight] = "5"
rootfs_export_package_cache() {
- deb_dl_dir_export ${ROOTFSDIR}
+ deb_dl_dir_export ${ROOTFSDIR} ${ROOTFS_DISTRO}
}

ROOTFS_INSTALL_COMMAND += "${@ 'rootfs_install_clean_files' if (d.getVar('ROOTFS_CLEAN_FILES') or '').strip() else ''}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index fbfe669..3b19914 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -255,7 +255,11 @@ isar_bootstrap() {
export IS_HOST debootstrap_args E
if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
sudo rm -rf --one-file-system "${ROOTFSDIR}"
- deb_dl_dir_import "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_import "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_import "${ROOTFSDIR}" "${DISTRO}"
+ fi

sudo -E -s <<'EOSUDO'
set -e
@@ -356,7 +360,11 @@ isar_bootstrap() {
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
EOSUDO
fi
- deb_dl_dir_export "${ROOTFSDIR}"
+ if [ "${IS_HOST}" ];then
+ deb_dl_dir_export "${ROOTFSDIR}" "${HOST_DISTRO}"
+ else
+ deb_dl_dir_export "${ROOTFSDIR}" "${DISTRO}"
+ fi
}

CLEANFUNCS = "clean_deploy"
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:36 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Collect the deb sources of the corresponding deb binaries cached
in DEBDIR as part of image postprocess.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 27 +++++++++++++++++++++++++++
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 8 ++++++++
3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 29a3d67..659fe4b 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,33 @@

inherit repository

+debsrc_download() {
+ export rootfs="$1"
+ export rootfs_distro="$2"
+ mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+ sudo -E -s <<'EOSUDO'
+ mkdir -p "${rootfs}/deb-src"
+ mountpoint -q "${rootfs}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
+EOSUDO
+ ( flock 9
+ set -e
+ printenv | grep -q BB_VERBOSE_LOGS && set -x
+ find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ done
+ ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"
+ sudo -E -s <<'EOSUDO'
+ mountpoint -q "${rootfs}/deb-src" && \
+ umount -l "${rootfs}/deb-src"
+ rm -rf "${rootfs}/deb-src"
+EOSUDO
+}
+
deb_dl_dir_import() {

export pc="${DEBDIR}/${2}"
export rootfs="${1}"

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2391529..6ca5759 100644

--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"

-ROOTFS_FEATURES += "clean-package-cache generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 6316321..cfabeae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,6 +185,14 @@ python do_rootfs_install() {

}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
+cache_deb_src() {
+ rootfs_install_resolvconf

+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

+ debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
+ rootfs_install_clean_files
+}
+

ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"

rootfs_postprocess_clean_package_cache() {
sudo -E chroot '${ROOTFSDIR}' \
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:39 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough apt-get source skips redownloading of files, it is still
slow and takes a lot of time. Instead, lookup if the dsc file is already
present in the cache and skip based on it.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 659fe4b..79a9a88 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -20,6 +20,8 @@ EOSUDO

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"

local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
+ [ -n "$dscfile" ] && continue

sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:42 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Avoid downloading deb-srcs for debs cached from other image builds.
One way to ensure that is to see if the package is present in the dpkg
status file.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 79a9a88..ce691cf 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -5,6 +5,15 @@

inherit repository

+is_not_part_of_current_build() {
+ local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
+ local output="$( grep -hs "^Package: ${package}" \
+ "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ [ -z "${output}" ]

+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"

@@ -18,6 +27,7 @@ EOSUDO
set -e

printenv | grep -q BB_VERBOSE_LOGS && set -x

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

+ is_not_part_of_current_build "${package}" && continue

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:45 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Factor out the mount and unmount section to separate function for
easy readability. No functional change intended.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 32 ++++++++++++++++++++++----------
1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ce691cf..b2d94e7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -14,15 +14,30 @@ is_not_part_of_current_build() {
[ -z "${output}" ]
}

+debsrc_do_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+EOSUDO
+}
+
+debsrc_undo_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" && \
+ umount -l "${1}/deb-src"
+ rm -rf "${1}/deb-src"
+EOSUDO

+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"

mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
- sudo -E -s <<'EOSUDO'
- mkdir -p "${rootfs}/deb-src"
- mountpoint -q "${rootfs}/deb-src" || \
- mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_do_mounts "${rootfs}"
+
( flock 9

set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x

@@ -37,11 +52,8 @@ EOSUDO

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

done

) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

- sudo -E -s <<'EOSUDO'
- mountpoint -q "${rootfs}/deb-src" && \
- umount -l "${rootfs}/deb-src"
- rm -rf "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_undo_mounts "${rootfs}"
}

deb_dl_dir_import() {
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:48 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Some packages are installed and then removed, like localepurge from
image-locales-extension.bbclass. Those information would not be
available in dpkg status file. Use dpkg log instead to see if the
package has been used on the target.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b2d94e7..a94fb10 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -7,10 +7,16 @@ inherit repository

is_not_part_of_current_build() {

local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"

- local output="$( grep -hs "^Package: ${package}" \
- "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
- "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
- "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
+ local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
+ # Since we are parsing all the debs in DEBDIR, we can to some extend
+ # try to eliminate some debs that are not part of the current multiconfig
+ # build using the below method.
+ local output="$( grep -hs "status installed ${package}:${arch} ${version}" \
+ "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+ "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \
+ "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
+
[ -z "${output}" ]
}

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:03:51 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Add a sanity test routine to test if sources for all packages
are present in the repo.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 10 ++++++++++
meta/recipes-devtools/base-apt/base-apt.bb | 2 ++
2 files changed, 12 insertions(+)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index c70138f..7b6e47b 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -96,3 +96,13 @@ repo_contains_package() {
# no
return 2
}
+
+repo_sanity_test() {
+ local dir="$1"
+ local dbdir="$2"

+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then

+ bbwarn "One or more sources are missing in base-apt"
+ bbwarn "${output}"
+ fi
+}
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index da3e32e..8940ae8 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -66,6 +66,8 @@ repo() {
fi

populate_base_apt
+ repo_sanity_test "${REPO_BASE_DIR}"/"${BASE_DISTRO}" \
+ "${REPO_BASE_DB_DIR}"/"${BASE_DISTRO}"
}

python do_cache() {
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:06:00 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough we are collecting the debsrcs as part of postprocess,
it could not be considered a ROOTFS_FEATURE, instead a base-apt one.
Introduce BASE_REPO_FEATURES, to provide user with control to enable
or disable cache-deb-src. Disabled by default, since it is not required
for normal offline build to work.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 4 +++-
meta/classes/repository.bbclass | 9 +++++----
meta/classes/rootfs.bbclass | 1 -
meta/recipes-devtools/base-apt/base-apt.bb | 1 +
5 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 07a7781..ec0a384 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -181,6 +181,10 @@ ISAR_CROSS_COMPILE ?= "0"
# NOTE: this works for amd64 and arm64 targets so far
#ISAR_ENABLE_COMPAT_ARCH ?= "1"

+# Uncomment this to enable caching of all source packages.
+# Without this feature, only sources of packages downloaded with apt:// are downloaded.
+#BASE_REPO_FEATURES ?= "cache-deb-src"
+
#
# Uncomment this to enable use of cached base repository
#ISAR_USE_CACHED_BASE_REPO ?= "1"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6ca5759..5c2c88c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,10 +63,12 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)} "
+
inherit rootfs
inherit image-sdk-extension
inherit image-tools-extension
diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 7b6e47b..1f475dc 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -100,9 +100,10 @@ repo_contains_package() {
repo_sanity_test() {
local dir="$1"
local dbdir="$2"
- local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
- if [ -n "${output}" ]; then
- bbwarn "One or more sources are missing in base-apt"
- bbwarn "${output}"
+ if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ];then

+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then

+ bbfatal "One or more sources are missing in repo. ${output}"
+ fi
fi
}
diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cfabeae..0aa5502 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,7 +185,6 @@ python do_rootfs_install() {

}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
cache_deb_src() {
rootfs_install_resolvconf

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 8940ae8..506a28f 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -10,6 +10,7 @@ SRC_URI = "file://distributions.in"

BASE_REPO_KEY ?= ""
KEYFILES ?= ""
+BASE_REPO_FEATURES ?= ""

populate_base_apt() {
find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:06:01 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

reprepro's includedsc failed for certain packages like makedev while
complaining about missing Section information. This information is
optional according to Debian[1]. Use a dummy value to avoid such
failures.

[1] https://wiki.debian.org/DebianRepository/Format

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 1f475dc..ac395db 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass
@@ -42,7 +42,7 @@ repo_add_srcpackage() {
if [ -n "${GNUPGHOME}" ]; then
export GNUPGHOME="${GNUPGHOME}"
fi
- reprepro -b "${dir}" --dbdir "${dbdir}" -C main -P source \
+ reprepro -b "${dir}" --dbdir "${dbdir}" -C main -S - -P source \
includedsc "${codename}" \
"$@"
}
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:06:05 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Enable debsrc caching for CI builds.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

scripts/ci_build.sh | 1 +
1 file changed, 1 insertion(+)

diff --git a/scripts/ci_build.sh b/scripts/ci_build.sh
index a5eee82..09a0b4b 100755
--- a/scripts/ci_build.sh
+++ b/scripts/ci_build.sh
@@ -184,6 +184,7 @@ if [ -n "$REPRO_BUILD" ]; then
sed -i -e 's/^BB_NO_NETWORK/#BB_NO_NETWORK/g' conf/local.conf
fi

+sed -i -e 's/^#BASE_REPO_FEATURES ?= "cache-deb-src"/BASE_REPO_FEATURES ?= "cache-deb-src"/g' conf/local.conf
# Start cross build for the defined set of configurations
sed -i -e 's/ISAR_CROSS_COMPILE ?= "0"/ISAR_CROSS_COMPILE ?= "1"/g' conf/local.conf
bitbake $BB_ARGS $CROSS_TARGETS_SET
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:06:08 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

There is a possiblilty that one of the packages installed in the
rootfs provides /etc/resolv.conf and we might accidentally remove
it. Fix it by taking a backup of any existing resolv conf files
and restoring it later. This is needed since we could not effectively
move caching before rootfs_install_clean_files since we need the
latest dpkg log.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0aa5502..ed43fb9 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -186,10 +186,18 @@ python do_rootfs_install() {

addtask rootfs_install before do_rootfs_postprocess after do_unpack

cache_deb_src() {
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
+ fi
rootfs_install_resolvconf
+

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

- rootfs_install_clean_files
+
+ sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf.isar "${ROOTFSDIR}"/etc/resolv.conf
+ fi

}

ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:06:11 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

The apt state information in var/lib/apt/lists is cleared as
part of postprocessing. This makes apt-get calls in cache_deb_src
fail when rerunning the postprocess task.

Since we cannot run apt-get update again to refresh the state
information, copy the apt state information from the initial
bootstrapped image.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ed43fb9..bebc0c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -190,6 +190,10 @@ cache_deb_src() {

sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar

fi
rootfs_install_resolvconf
+ # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
+ # relies on that through out the build. Copy that state information instead of apt-get update
+ # which generates a new state from upstream.
+ sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 3:07:17 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Add details about deb-src caching.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

doc/user_manual.md | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 8d04cd2..c0f842a 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -918,3 +918,28 @@ And build the corresponding image target:
```
bitbake mc:qemuarm64-buster:isar-image-base
```
+## Cache all upstream Debian source packages in local apt
+
+### Motivation
+
+For OSS license clearance, we might need to have the sources of all Debian packages we
+ship with the particular image.
+
+Currently the local-apt generated has only Debian binary packages. Extend the local-apt
+to have Debian source packages as well, for OSS clearing purposes.
+
+### Solution
+
+ - Trigger download of Debian source packages as part of rootfs postprocess.
+
+With the current base-apt implementation, we already cache all the binary packages and
+generate a local-apt for offline build.
+
+Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
+download the corresponding Debian source file using "apt-get source" command.
+
+By default, the Debian source caching is not enabled. If you want to cache the Debian
+sources as well, as part of your image build, add the below line to your local.conf file.
+```
+BASE_REPO_FEATURES = "cache-deb-src"
+```
--
2.17.1

Jan Kiszka

unread,

Nov 25, 2020, 6:35:45 AM11/25/20

to Vijai Kumar K, isar-...@googlegroups.com

On 25.11.20 09:06, Vijai Kumar K wrote:
> Add details about deb-src caching.
>
> Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
> ---
> doc/user_manual.md | 25 +++++++++++++++++++++++++
> 1 file changed, 25 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 8d04cd2..c0f842a 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -918,3 +918,28 @@ And build the corresponding image target:
> ```
> bitbake mc:qemuarm64-buster:isar-image-base
> ```
> +## Cache all upstream Debian source packages in local apt
> +
> +### Motivation
> +
> +For OSS license clearance, we might need to have the sources of all Debian packages we
> +ship with the particular image.

OSS license *compliance*: Some licenses require to provide the
corresponding sources code, other require copyright attributions that
may be best provided via the source code. In addition, you may want to
archive the code locally in order to ensure reproducibility (and
modifiability) in the future.

> +
> +Currently the local-apt generated has only Debian binary packages. Extend the local-apt
> +to have Debian source packages as well, for OSS clearing purposes.
> +
> +### Solution
> +
> + - Trigger download of Debian source packages as part of rootfs postprocess.
> +
> +With the current base-apt implementation, we already cache all the binary packages and
> +generate a local-apt for offline build.
> +
> +Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
> +download the corresponding Debian source file using "apt-get source" command.
> +
> +By default, the Debian source caching is not enabled. If you want to cache the Debian
> +sources as well, as part of your image build, add the below line to your local.conf file.
> +```
> +BASE_REPO_FEATURES = "cache-deb-src"
> +```
>

So this will fetch and and store ALL sources corresponding to the binary
packages that the target image installs? Or rather that any rootfs
installs, thus also buildchroots and sdkrootfs? Please make this clearer.

Jan

vijaikumar....@gmail.com

unread,

Nov 25, 2020, 8:14:58 AM11/25/20

to isar-users

On Wednesday, November 25, 2020 at 5:05:45 PM UTC+5:30 Jan Kiszka wrote:

On 25.11.20 09:06, Vijai Kumar K wrote:
> Add details about deb-src caching.
>
> Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
> ---
> doc/user_manual.md | 25 +++++++++++++++++++++++++
> 1 file changed, 25 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index 8d04cd2..c0f842a 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -918,3 +918,28 @@ And build the corresponding image target:
> ```
> bitbake mc:qemuarm64-buster:isar-image-base
> ```
> +## Cache all upstream Debian source packages in local apt
> +
> +### Motivation
> +
> +For OSS license clearance, we might need to have the sources of all Debian packages we
> +ship with the particular image.

OSS license *compliance*: Some licenses require to provide the
corresponding sources code, other require copyright attributions that
may be best provided via the source code. In addition, you may want to
archive the code locally in order to ensure reproducibility (and
modifiability) in the future.

Will include this line.

> +
> +Currently the local-apt generated has only Debian binary packages. Extend the local-apt
> +to have Debian source packages as well, for OSS clearing purposes.
> +
> +### Solution
> +
> + - Trigger download of Debian source packages as part of rootfs postprocess.
> +
> +With the current base-apt implementation, we already cache all the binary packages and
> +generate a local-apt for offline build.
> +
> +Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
> +download the corresponding Debian source file using "apt-get source" command.
> +
> +By default, the Debian source caching is not enabled. If you want to cache the Debian
> +sources as well, as part of your image build, add the below line to your local.conf file.
> +```
> +BASE_REPO_FEATURES = "cache-deb-src"
> +```
>

So this will fetch and and store ALL sources corresponding to the binary
packages that the target image installs? Or rather that any rootfs
installs, thus also buildchroots and sdkrootfs? Please make this clearer.

Yes. Exactly. I will update the document.

Thanks,

Vijai Kumar K

unread,

Nov 25, 2020, 8:43:07 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Changes in v8:
- Fix documentation as per Jan comments.

doc/user_manual.md | 30 +++++++++

12 files changed, 196 insertions(+), 53 deletions(-)

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:43:09 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

With the current implementation it is difficult to append a
postprocess function which requires a chroot environment.
For example, to add a postprocess function which runs apt-get to
download all source of packages installed in the target.

rootfs_postprocess_finalize is not actually an optional feature
but instead a necessary cleanup function for image class.
So, move the implementation to image class and make it as a task.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/image.bbclass | 41 ++++++++++++++++++++++++++++++++++++-
meta/classes/rootfs.bbclass | 39 -----------------------------------
2 files changed, 40 insertions(+), 40 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a296cc0..2391529 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"
-ROOTFS_FEATURES += "clean-package-cache finalize-rootfs generate-manifest"

+ROOTFS_FEATURES += "clean-package-cache generate-manifest"
ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

@@ -173,5 +173,44 @@ python do_deploy() {
}
addtask deploy before do_build after do_image

+do_rootfs_finalize() {

+ sudo -s <<'EOSUDO'

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index afec1cb..eae14d5 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass

Vijai Kumar K

unread,

Nov 25, 2020, 8:43:12 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

In case of targets where HOST_DISTRO!=DISTRO, like rpi-stretch,
we were still caching the debs from both the distros into the same
DEBDIR/DISTRO directory. With this change, HOST_DISTRO would be

cached in the relevant subdirectory and avoids mixing of debs from
two separate distros.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

6 files changed, 28 insertions(+), 12 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index e996703..29a3d67 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass

@@ -6,7 +6,7 @@
inherit repository

deb_dl_dir_import() {

- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"

[ ! -d "${pc}" ] && return 0
sudo mkdir -p "${rootfs}"/var/cache/apt/archives/
@@ -20,7 +20,7 @@ deb_dl_dir_import() {
}

deb_dl_dir_export() {

- export pc="${DEBDIR}/${DISTRO}/"
+ export pc="${DEBDIR}/${2}"
export rootfs="${1}"

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index eae14d5..6316321 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass

Vijai Kumar K

unread,

Nov 25, 2020, 8:43:15 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Collect the deb sources of the corresponding deb binaries cached
in DEBDIR as part of image postprocess.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 27 +++++++++++++++++++++++++++
meta/classes/image.bbclass | 2 +-
meta/classes/rootfs.bbclass | 8 ++++++++
3 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 29a3d67..659fe4b 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass

@@ -5,6 +5,33 @@

inherit repository

+debsrc_download() {
+ export rootfs="$1"
+ export rootfs_distro="$2"
+ mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
+ sudo -E -s <<'EOSUDO'
+ mkdir -p "${rootfs}/deb-src"
+ mountpoint -q "${rootfs}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
+EOSUDO
+ ( flock 9
+ set -e

+ printenv | grep -q BB_VERBOSE_LOGS && set -x
+ find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do
+ local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
+ local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"
+
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \
+ sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"
+ done
+ ) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

+ sudo -E -s <<'EOSUDO'

+ mountpoint -q "${rootfs}/deb-src" && \
+ umount -l "${rootfs}/deb-src"
+ rm -rf "${rootfs}/deb-src"
+EOSUDO
+}
+
deb_dl_dir_import() {

export pc="${DEBDIR}/${2}"
export rootfs="${1}"

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2391529..6ca5759 100644

--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,7 +63,7 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"

-ROOTFS_FEATURES += "clean-package-cache generate-manifest"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 6316321..cfabeae 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,6 +185,14 @@ python do_rootfs_install() {

}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
+cache_deb_src() {
+ rootfs_install_resolvconf

+ deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

+ debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}
+ rootfs_install_clean_files
+}
+

ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"

Vijai Kumar K

unread,

Nov 25, 2020, 8:43:18 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough apt-get source skips redownloading of files, it is still
slow and takes a lot of time. Instead, lookup if the dsc file is already
present in the cache and skip based on it.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 659fe4b..79a9a88 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass
@@ -20,6 +20,8 @@ EOSUDO

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"

local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

+ local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")
+ [ -n "$dscfile" ] && continue

sudo -E chroot --userspec=$( id -u ):$( id -g ) ${rootfs} \

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:44:40 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Avoid downloading deb-srcs for debs cached from other image builds.
One way to ensure that is to see if the package is present in the dpkg
status file.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index 79a9a88..ce691cf 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass

@@ -5,6 +5,15 @@

inherit repository

+is_not_part_of_current_build() {
+ local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"
+ local output="$( grep -hs "^Package: ${package}" \
+ "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
+ "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \

+ "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ [ -z "${output}" ]

+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"

@@ -18,6 +27,7 @@ EOSUDO
set -e

printenv | grep -q BB_VERBOSE_LOGS && set -x

find "${rootfs}/var/cache/apt/archives/" -maxdepth 1 -type f -iname '*\.deb' | while read package; do

+ is_not_part_of_current_build "${package}" && continue

local src="$( dpkg-deb --show --showformat '${source:Package}' "${package}" )"
local version="$( dpkg-deb --show --showformat '${source:Version}' "${package}" )"

local dscfile=$(find "${DEBSRCDIR}"/"${rootfs_distro}" -name "${src}_${version}.dsc")

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:44:42 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Factor out the mount and unmount section to separate function for
easy readability. No functional change intended.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 32 ++++++++++++++++++++++----------
1 file changed, 22 insertions(+), 10 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index ce691cf..b2d94e7 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass

@@ -14,15 +14,30 @@ is_not_part_of_current_build() {
[ -z "${output}" ]
}

+debsrc_do_mounts() {
+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" || \
+ mount --bind "${DEBSRCDIR}" "${1}/deb-src"
+EOSUDO
+}
+
+debsrc_undo_mounts() {

+ sudo -s <<EOSUDO
+ mkdir -p "${1}/deb-src"
+ mountpoint -q "${1}/deb-src" && \
+ umount -l "${1}/deb-src"
+ rm -rf "${1}/deb-src"
+EOSUDO

+}
+
debsrc_download() {
export rootfs="$1"
export rootfs_distro="$2"

mkdir -p "${DEBSRCDIR}"/"${rootfs_distro}"
- sudo -E -s <<'EOSUDO'
- mkdir -p "${rootfs}/deb-src"
- mountpoint -q "${rootfs}/deb-src" || \
- mount --bind "${DEBSRCDIR}" "${rootfs}/deb-src"
-EOSUDO
+
+ debsrc_do_mounts "${rootfs}"
+
( flock 9

set -e
printenv | grep -q BB_VERBOSE_LOGS && set -x

@@ -37,11 +52,8 @@ EOSUDO

sh -c ' mkdir -p "/deb-src/${1}/${2}" && cd "/deb-src/${1}/${2}" && apt-get -y --download-only --only-source source "$2"="$3" ' download-src "${rootfs_distro}" "${src}" "${version}"

done

) 9>"${DEBSRCDIR}/${rootfs_distro}.lock"

- sudo -E -s <<'EOSUDO'

Vijai Kumar K

unread,

Nov 25, 2020, 8:44:45 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Some packages are installed and then removed, like localepurge from
image-locales-extension.bbclass. Those information would not be
available in dpkg status file. Use dpkg log instead to see if the
package has been used on the target.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/deb-dl-dir.bbclass | 14 ++++++++++----
1 file changed, 10 insertions(+), 4 deletions(-)

diff --git a/meta/classes/deb-dl-dir.bbclass b/meta/classes/deb-dl-dir.bbclass
index b2d94e7..a94fb10 100644
--- a/meta/classes/deb-dl-dir.bbclass
+++ b/meta/classes/deb-dl-dir.bbclass

@@ -7,10 +7,16 @@ inherit repository

is_not_part_of_current_build() {

local package="$( dpkg-deb --show --showformat '${Package}' "${1}" )"

- local output="$( grep -hs "^Package: ${package}" \
- "${IMAGE_ROOTFS}"/var/lib/dpkg/status \
- "${BUILDCHROOT_HOST_DIR}"/var/lib/dpkg/status \
- "${BUILDCHROOT_TARGET_DIR}"/var/lib/dpkg/status )"
+ local arch="$( dpkg-deb --show --showformat '${Architecture}' "${1}" )"
+ local version="$( dpkg-deb --show --showformat '${Version}' "${1}" )"
+ # Since we are parsing all the debs in DEBDIR, we can to some extend
+ # try to eliminate some debs that are not part of the current multiconfig
+ # build using the below method.
+ local output="$( grep -hs "status installed ${package}:${arch} ${version}" \
+ "${IMAGE_ROOTFS}"/var/log/dpkg.log \
+ "${BUILDCHROOT_HOST_DIR}"/var/log/dpkg.log \

+ "${BUILDCHROOT_TARGET_DIR}"/var/log/dpkg.log | head -1 )"
+
[ -z "${output}" ]
}

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:44:48 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Add a sanity test routine to test if sources for all packages
are present in the repo.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 10 ++++++++++
meta/recipes-devtools/base-apt/base-apt.bb | 2 ++
2 files changed, 12 insertions(+)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index c70138f..7b6e47b 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass

@@ -96,3 +96,13 @@ repo_contains_package() {
# no
return 2
}
+
+repo_sanity_test() {
+ local dir="$1"
+ local dbdir="$2"

+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then

+ bbwarn "One or more sources are missing in base-apt"
+ bbwarn "${output}"
+ fi
+}

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index da3e32e..8940ae8 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb

Vijai Kumar K

unread,

Nov 25, 2020, 8:44:51 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Eventhough we are collecting the debsrcs as part of postprocess,
it could not be considered a ROOTFS_FEATURE, instead a base-apt one.
Introduce BASE_REPO_FEATURES, to provide user with control to enable
or disable cache-deb-src. Disabled by default, since it is not required
for normal offline build to work.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 4 +++-
meta/classes/repository.bbclass | 9 +++++----
meta/classes/rootfs.bbclass | 1 -
meta/recipes-devtools/base-apt/base-apt.bb | 1 +
5 files changed, 13 insertions(+), 6 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 07a7781..ec0a384 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -181,6 +181,10 @@ ISAR_CROSS_COMPILE ?= "0"
# NOTE: this works for amd64 and arm64 targets so far
#ISAR_ENABLE_COMPAT_ARCH ?= "1"

+# Uncomment this to enable caching of all source packages.
+# Without this feature, only sources of packages downloaded with apt:// are downloaded.
+#BASE_REPO_FEATURES ?= "cache-deb-src"
+
#
# Uncomment this to enable use of cached base repository

#ISAR_USE_CACHED_BASE_REPO ?= "1"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6ca5759..5c2c88c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,10 +63,12 @@ image_do_mounts() {
}

ROOTFSDIR = "${IMAGE_ROOTFS}"

-ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
+ROOTFS_FEATURES += "clean-package-cache generate-manifest"

ROOTFS_PACKAGES += "${IMAGE_PREINSTALL} ${IMAGE_INSTALL}"
ROOTFS_MANIFEST_DEPLOY_DIR ?= "${DEPLOY_DIR_IMAGE}"

+ROOTFS_POSTPROCESS_COMMAND_prepend = "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)} "
+
inherit rootfs
inherit image-sdk-extension
inherit image-tools-extension

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 7b6e47b..1f475dc 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass

@@ -100,9 +100,10 @@ repo_contains_package() {
repo_sanity_test() {
local dir="$1"
local dbdir="$2"
- local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
- if [ -n "${output}" ]; then
- bbwarn "One or more sources are missing in base-apt"
- bbwarn "${output}"
+ if [ "${@bb.utils.contains('BASE_REPO_FEATURES', 'cache-deb-src', 'yes', 'no', d)}" = "yes" ];then

+ local output="$( reprepro -s -b "${dir}" --dbdir "${dbdir}" sourcemissing )"
+ if [ -n "${output}" ]; then

+ bbfatal "One or more sources are missing in repo. ${output}"
+ fi
fi
}

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index cfabeae..0aa5502 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -185,7 +185,6 @@ python do_rootfs_install() {

}
addtask rootfs_install before do_rootfs_postprocess after do_unpack

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'cache-deb-src', 'cache_deb_src', '', d)}"
cache_deb_src() {
rootfs_install_resolvconf

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 8940ae8..506a28f 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb

@@ -10,6 +10,7 @@ SRC_URI = "file://distributions.in"

BASE_REPO_KEY ?= ""
KEYFILES ?= ""
+BASE_REPO_FEATURES ?= ""

populate_base_apt() {

find "${DEBDIR}"/"${DISTRO}" -name '*\.deb' | while read package; do
--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:46:23 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

reprepro's includedsc failed for certain packages like makedev while
complaining about missing Section information. This information is
optional according to Debian[1]. Use a dummy value to avoid such
failures.

[1] https://wiki.debian.org/DebianRepository/Format

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/repository.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/repository.bbclass b/meta/classes/repository.bbclass
index 1f475dc..ac395db 100644
--- a/meta/classes/repository.bbclass
+++ b/meta/classes/repository.bbclass

Vijai Kumar K

unread,

Nov 25, 2020, 8:46:25 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Enable debsrc caching for CI builds.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

Vijai Kumar K

unread,

Nov 25, 2020, 8:46:28 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

There is a possiblilty that one of the packages installed in the
rootfs provides /etc/resolv.conf and we might accidentally remove
it. Fix it by taking a backup of any existing resolv conf files
and restoring it later. This is needed since we could not effectively
move caching before rootfs_install_clean_files since we need the
latest dpkg log.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index 0aa5502..ed43fb9 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -186,10 +186,18 @@ python do_rootfs_install() {

addtask rootfs_install before do_rootfs_postprocess after do_unpack

cache_deb_src() {
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf ]; then

+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar
+ fi
rootfs_install_resolvconf
+

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

- rootfs_install_clean_files
+
+ sudo rm -f "${ROOTFSDIR}"/etc/resolv.conf
+ if [ -e "${ROOTFSDIR}"/etc/resolv.conf.isar ]; then
+ sudo mv "${ROOTFSDIR}"/etc/resolv.conf.isar "${ROOTFSDIR}"/etc/resolv.conf
+ fi
}

ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'clean-package-cache', 'rootfs_postprocess_clean_package_cache', '', d)}"

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:46:31 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

The apt state information in var/lib/apt/lists is cleared as
part of postprocessing. This makes apt-get calls in cache_deb_src
fail when rerunning the postprocess task.

Since we cannot run apt-get update again to refresh the state
information, copy the apt state information from the initial
bootstrapped image.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

meta/classes/rootfs.bbclass | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index ed43fb9..bebc0c4 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -190,6 +190,10 @@ cache_deb_src() {

sudo mv "${ROOTFSDIR}"/etc/resolv.conf "${ROOTFSDIR}"/etc/resolv.conf.isar

fi
rootfs_install_resolvconf
+ # Note: ISAR updates the apt state information(apt-get update) only once during bootstrap and
+ # relies on that through out the build. Copy that state information instead of apt-get update
+ # which generates a new state from upstream.
+ sudo cp -Trpn "${BOOTSTRAP_SRC}/var/lib/apt/lists/" "${ROOTFSDIR}/var/lib/apt/lists/"

deb_dl_dir_import ${ROOTFSDIR} ${ROOTFS_DISTRO}

debsrc_download ${ROOTFSDIR} ${ROOTFS_DISTRO}

--
2.17.1

Vijai Kumar K

unread,

Nov 25, 2020, 8:46:34 AM11/25/20

to isar-...@googlegroups.com, Vijai Kumar K

Add details about deb-src caching.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

doc/user_manual.md | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 8d04cd2..cc7e47f 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -918,3 +918,33 @@ And build the corresponding image target:

```
bitbake mc:qemuarm64-buster:isar-image-base
```
+## Cache all upstream Debian source packages in local apt
+
+### Motivation
+

+OSS license compliance: Some licenses require to provide the corresponding sources code,
+other require copyright attributions that may be best provided via the source code. In
+addition, you may want to archive the code locally in order to ensure reproducibility (and
+modifiability) in the future.

+
+Currently the local-apt generated has only Debian binary packages. Extend the local-apt

+to have Debian source packages as well.

+
+### Solution
+
+ - Trigger download of Debian source packages as part of rootfs postprocess.
+

+With the current base-apt implementation, we already cache all the binary packages that
+we download and install onto the target rootfs and buildchroot. This is then used to

+generate a local-apt for offline build.
+
+Use rootfs postprocessing to parse through the the list fo deb files in ${DEDDIR} and
+download the corresponding Debian source file using "apt-get source" command.

+This caches the sources of all the Debian packages that are downloaded and installed onto
+the target rootfs and buildchroots.

+
+By default, the Debian source caching is not enabled.

+To enable it, add the below line to your local.conf file.

+```
+BASE_REPO_FEATURES = "cache-deb-src"
+```

--
2.17.1

Baurzhan Ismagulov

unread,

Nov 26, 2020, 5:21:34 AM11/26/20

to isar-users

On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote:
> You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf.
> By default its disabled. Not everyone
> want to cache the deb sources. It takes a lot of time.

Thanks Vijai Kumar, will play with v8. It's good to have this in the manual.

What about the issues below? Should I include them myself?

p1:

On Tue, Nov 24, 2020 at 08:47:11PM +0100, Baurzhan Ismagulov wrote:

> On Wed, Sep 30, 2020 at 11:32:47AM +0530, Vijai Kumar K wrote:
> > diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass

> > index afec1cb..eae14d5 100644
> > --- a/meta/classes/rootfs.bbclass
> > +++ b/meta/classes/rootfs.bbclass

> > @@ -201,45 +201,6 @@ rootfs_generate_manifest () {
> > ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> > }
> >
> > -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
>

> Suggest also removing the 'finalize-rootfs' comment before ROOTFS_FEATURES ?=
> "".

p3, p9:

On Tue, Nov 24, 2020 at 08:49:19PM +0100, Baurzhan Ismagulov wrote:

> On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass

> > index 2391529..6ca5759 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -63,7 +63,7 @@ image_do_mounts() {

> > }
> >
> > ROOTFSDIR = "${IMAGE_ROOTFS}"
> > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"

> > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
>

> Should it be "cache-deb-src"? The same for p9.

With kind regards,
Baurzhan.

vijaikumar....@gmail.com

unread,

Nov 26, 2020, 5:43:06 AM11/26/20

to isar-users

On Thursday, November 26, 2020 at 3:51:34 PM UTC+5:30 i...@radix50.net wrote:

On Tue, Nov 24, 2020 at 10:04:24PM -0800, vijaikumar....@gmail.com wrote:
> You might want to set BASE_REPO_FEATURES ?= "cache-deb-src" in local conf.
> By default its disabled. Not everyone
> want to cache the deb sources. It takes a lot of time.

Thanks Vijai Kumar, will play with v8. It's good to have this in the manual.

What about the issues below? Should I include them myself?

Oops. Sorry about that. I somehow missed to see these comments. I will address these in v9.

Thanks,

Vijai Kumar K

vijaikumar....@gmail.com

unread,

Nov 26, 2020, 5:52:22 AM11/26/20

to isar-users

On Wednesday, November 25, 2020 at 1:19:36 AM UTC+5:30 i...@radix50.net wrote:

On Wed, Sep 30, 2020 at 11:32:49AM +0530, Vijai Kumar K wrote:
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 2391529..6ca5759 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -63,7 +63,7 @@ image_do_mounts() {
> }
>
> ROOTFSDIR = "${IMAGE_ROOTFS}"
> -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"

Should it be "cache-deb-src"? The same for p9.

Yes. It is cache-deb-src with hypens(-). The original function is with underscore.

If you see a better name let me know. We can change it.

Thanks,

Vijai Kumar K

With kind regards,
Baurzhan.

Baurzhan Ismagulov

unread,

Nov 26, 2020, 6:03:13 AM11/26/20

to isar-users

On Thu, Nov 26, 2020 at 02:52:22AM -0800, vijaikumar....@gmail.com wrote:
> > > -ROOTFS_FEATURES += "clean-package-cache generate-manifest"
> > > +ROOTFS_FEATURES += "clean-package-cache generate-manifest cach-deb-src"
> >
> > Should it be "cache-deb-src"? The same for p9.
>
> Yes. It is cache-deb-src with hypens(-). The original function is with
> underscore.
>
> If you see a better name let me know. We can change it.

Sorry for the confusion, I meant that ROOTFS_FEATURES mentions cach-deb-src,
"cach" without "e".

With kind regards,
Baurzhan.

vijaikumar....@gmail.com

unread,

Nov 26, 2020, 6:07:05 AM11/26/20

to isar-users

Aha. Indeed. Good catch. will fix that in v9.

Thanks,

Vijai Kumar

With kind regards,
Baurzhan.

Reply all

Reply to author

Forward