[PATCH 0/9] Introduce local apt repo to cache upstream debian packages for offline usage

[Maxim Yu. Osipov]

Hello everybody,

This series by Alexander Smirnov,

1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages

2) Caches in base-apt repo upstream Debian packages during image generation.

3) After this step, image can be built offline using only base-apt repo.

Usage instructions can be found in the last patch.

TODO:
Sign the repo with gpg

Kind regards,
Maxim.

Alexander Smirnov (8):
base-apt: Add helper class
base-apt: Introduce base implementaiton
isar-boot-strap: Add option to keep cache
image: Add cache_base_repo task
isar-bootstrap: Make possible to reuse the cache
buildchroot: Make it buildable from base-apt
workaround: Use --allow-unauthenticated working with base-apt
local.conf: Add option to use cached base repository

Maxim Yu. Osipov (1):
doc: Creation of local apt repo caching upstream Debian packages

doc/user_manual.md | 39 ++++++++++++++++
meta-isar/conf/layer.conf | 10 ++--
meta-isar/conf/local.conf.sample | 4 ++
meta-isar/recipes-core/images/isar-image-base.bb | 3 +-
meta/classes/base-apt-helper.bbclass | 53 ++++++++++++++++++++++
meta/classes/image.bbclass | 21 +++++++++
meta/classes/isar-bootstrap-helper.bbclass | 13 +++++-
.../isar-bootstrap/files/base-apt-sources | 1 +
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
.../base-apt/files/distributions.in | 3 ++
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
12 files changed, 196 insertions(+), 16 deletions(-)
create mode 100644 meta/classes/base-apt-helper.bbclass
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources
create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in

--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

base-apt intended to store original upstream debs to re-use them
later offline. This class helps to populate base-apt with the packages
used during build.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta/classes/base-apt-helper.bbclass | 53 ++++++++++++++++++++++++++++++++++++
1 file changed, 53 insertions(+)
create mode 100644 meta/classes/base-apt-helper.bbclass

diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
new file mode 100644
index 0000000..5e3fe36
--- /dev/null
+++ b/meta/classes/base-apt-helper.bbclass
@@ -0,0 +1,53 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
+DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
+
+populate_base_apt() {
+ search_dir=$1
+
+ for package in $(find $search_dir -name '*.deb'); do
+ # NOTE: due to packages stored by reprepro are not modified, we can
+ # use search by filename to check if package is already in repo. In
+ # addition, m5sums could be compared to ensure, that package is the
+ # same and should not be overwritten. This method is easier and more
+ # robust than querying reprepro by name.
+
+ # Check if this package is taken from Isar-apt, if so - ingore it.
+ isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $package)
+ if [ -n "$isar_package" ]; then
+ # Check if MD5 sums are iendtical. This helps to avoid the case
+ # when packages is overriden from another repo.
+ md1=$(md5sum $package)
+ md2=$(md5sum $isar_package)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+ fi
+
+ # Check if this package is already in base-apt
+ isar_package=$(find ${BASE_APT_DIR}/${DISTRO_NAME} -name $package)
+ if [ -n "$isar_package" ]; then
+ md1=$(md5sum $package)
+ md2=$(md5sum $isar_package)
+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+
+ # md5sum differs, so remove the package from base-apt
+ name=$(basename $package | cut -d '_' -f 1)
+ reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
+ --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
+ -C main -A ${DISTRO_ARCH} \
+ remove ${DISTRO_SUITE} \
+ $name
+ fi
+
+ reprepro -b ${BASE_APT_DIR}/${DISTRO_NAME} \
+ --dbdir ${BASE_APT_DB}/${DISTRO_NAME} \
+ -C main \
+ includedeb ${DISTRO_SUITE} \
+ $package
+ done
+}
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

This task puts all the packages using in build to base-apt.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---
meta-isar/recipes-core/images/isar-image-base.bb | 3 ++-
meta/classes/image.bbclass | 21 +++++++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 4899593..bf606cc 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -41,7 +41,8 @@ devtmpfs /dev devtmpfs mode=0755,nosuid 0 0
# End /etc/fstab
EOF

- setup_root_file_system --clean --fstab "${WORKDIR}/fstab" \
+ setup_root_file_system --clean --keep-apt-cache \
+ --fstab "${WORKDIR}/fstab" \
"${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}

# Configure root filesystem
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ea8cbf5..0505a73 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -123,6 +123,27 @@ do_populate_sdk[depends] = "sdkchroot:do_build"

addtask populate_sdk after do_rootfs

+inherit base-apt-helper
+
+do_cache_base_repo[depends] = "base-apt:do_cache_config"
+do_cache_base_repo[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_cache_base_repo() {
+ if [ -d ${WORKDIR}/apt_cache ]; then
+ populate_base_apt ${WORKDIR}/apt_cache
+ fi
+
+ if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
+ fi
+
+ if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt
+ fi
+}
+
+addtask cache_base_repo after do_rootfs
+
# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"

--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

The default ISAR behavior assumes to remove all the packages
from apt cache, but we need them to put into base-apt.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 597f6b1..4a00cb0 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -70,6 +70,7 @@ setup_root_file_system() {
--fstab) FSTAB=$2; shift ;;
--host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
--host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
+ --keep-apt-cache) KEEP_APT_CACHE=1 ;;
-*) bbfatal "$0: invalid option specified: $1" ;;
*) break ;;
esac
@@ -131,6 +132,11 @@ setup_root_file_system() {
/usr/bin/apt-get purge --yes $pkg
done
if [ ${CLEAN} ]; then
+ if [ ${KEEP_APT_CACHE} ]; then
+ mkdir -p ${WORKDIR}/apt_cache
+ sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb') ${WORKDIR}/apt_cache
+ sudo chown $USER ${WORKDIR}/apt_cache/*
+ fi
sudo -E chroot "$ROOTFSDIR" \
/usr/bin/apt-get autoremove --purge --yes
sudo -E chroot "$ROOTFSDIR" \
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/conf/layer.conf | 10 ++++---
meta/recipes-devtools/base-apt/base-apt.bb | 31 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +++
3 files changed, 40 insertions(+), 4 deletions(-)

create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index cd42f06..ae1b3c5 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -20,8 +20,10 @@ LAYERDIR_isar = "${LAYERDIR}"
# Codename of the repository created by the caching class
DEBDISTRONAME = "isar"

-# Path to the Isar apt repository
-DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
+# Isar apt repository paths
+DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/isar-apt/apt"
+DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/isar-apt/db"

-# Path to the Isar databases used by `reprepro`
-DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
+# Base apt repository paths
+BASE_APT_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
+BASE_APT_DB ?= "${DEPLOY_DIR}/base-apt/db"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
new file mode 100644
index 0000000..6ff1164
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -0,0 +1,31 @@

+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+

+SRC_URI = "file://distributions.in"
+
+inherit base-apt-helper
+
+CACHE_CONF_DIR = "${BASE_APT_DIR}/${DISTRO_NAME}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+do_cache_config[lockfiles] = "${BASE_APT_DIR}/isar.lock"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \
+ ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+
+ path_cache="${BASE_APT_DIR}/${DISTRO_NAME}"
+ path_databases="${BASE_APT_DB}/${DISTRO_NAME}"
+
+ if [ ! -d "${path_databases}" ]; then
+ reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DISTRO_SUITE}
+ fi
+}
+
+addtask cache_config after do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
new file mode 100644
index 0000000..cc82c57
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf arm64 amd64 source
+Components: main
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

.../isar-bootstrap/files/base-apt-sources | 1 +
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 ++++++++++++++++------
2 files changed, 24 insertions(+), 9 deletions(-)
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources

diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
new file mode 100644
index 0000000..594db56
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
@@ -0,0 +1 @@
+deb file:///base-apt/debian {DISTRO} main
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index cfad136..9b5a894 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -12,18 +12,24 @@ SRC_URI = " \
file://isar-apt.conf \
file://isar-apt-fallback.conf \
file://locale \
- file://chroot-setup.sh"
+ file://chroot-setup.sh \
+ file://base-apt-sources"
PV = "1.0"

DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
+BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
APTKEYFILES = ""
APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${BASE_APT_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
+
+inherit base-apt-helper
+
python () {
from urllib.parse import urlparse
distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
@@ -171,6 +177,10 @@ isar_bootstrap() {
esac
shift
done
+ debootstrap_args="--verbose --variant=minbase --include=locales "
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ debootstrap_args="$debootstrap_args --no-check-gpg"
+ fi
E="${@bb.utils.export_proxies(d)}"
sudo -E flock "${ISAR_BOOTSTRAP_LOCK}" -c "\
set -e
@@ -181,9 +191,7 @@ isar_bootstrap() {
rm -rf "${ROOTFSDIR}"
fi
if [ ${IS_HOST} ]; then
- ${DEBOOTSTRAP} --verbose \
- --variant=minbase \
- --include=locales \
+ ${DEBOOTSTRAP} $debootstrap_args \
${@get_distro_components_argument(d, True)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, True)}" \
@@ -191,10 +199,8 @@ isar_bootstrap() {
"${@get_distro_source(d, True)}"

else
- "${DEBOOTSTRAP}" --verbose \
- --variant=minbase \
+ "${DEBOOTSTRAP}" $debootstrap_args \
--arch="${DISTRO_ARCH}" \
- --include=locales \
${@get_distro_components_argument(d, False)} \
${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, False)}" \
@@ -207,8 +213,16 @@ isar_bootstrap() {
install -v -m644 "${APTPREFS}" \
"${ROOTFSDIR}/etc/apt/preferences.d/bootstrap"
mkdir -p "${ROOTFSDIR}/etc/apt/sources.list.d"
- install -v -m644 "${APTSRCS}" \
- "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
+ sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g" ${BASEAPTSRCS}
+ mkdir -p ${ROOTFSDIR}/base-apt
+ sudo mount --bind ${BASE_APT_DIR} ${ROOTFSDIR}/base-apt
+ install -v -m644 "${BASEAPTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/base-apt.list"
+ else
+ install -v -m644 "${APTSRCS}" \
+ "${ROOTFSDIR}/etc/apt/sources.list.d/bootstrap.list"
+ fi
rm -f "${ROOTFSDIR}/etc/apt/sources.list"
mkdir -p "${ROOTFSDIR}/etc/apt/apt.conf.d"
install -v -m644 "${WORKDIR}/isar-apt.conf" \
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta/classes/isar-bootstrap-helper.bbclass | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 4a00cb0..9dabc25 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -98,6 +98,11 @@ setup_root_file_system() {
else
sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
fi
+

+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

+ sudo mount --bind ${BASE_APT_DIR} ${ROOTFSDIR}/base-apt

+ fi
+
sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc

--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

---
meta/classes/isar-bootstrap-helper.bbclass | 2 +-
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 9dabc25..b7ab552 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -79,7 +79,7 @@ setup_root_file_system() {
ROOTFSDIR="$1"
shift
PACKAGES="$@"
- APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
+ APT_ARGS="install --yes --allow-unauthenticated -o Debug::pkgProblemResolver=yes"
CLEAN_FILES="${ROOTFSDIR}/etc/hostname ${ROOTFSDIR}/etc/resolv.conf"

sudo cp -Trpfx \
diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 4bd604f..2fc2f66 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -10,7 +10,7 @@ source /isar/common.sh
# Notes:
# 1) everything before the -y switch is unchanged from the defaults
# 2) we add -y to go non-interactive
-install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y"
+install_cmd="apt-get -o Debug::pkgProblemResolver=yes --no-install-recommends -y --allow-unauthenticated"

# Make sure that we have latest isar-apt content.
# Options meaning:
--
2.11.0

[Maxim Yu. Osipov]

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index db58cf1..d69dd0d 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
- [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
- [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
- [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
+ - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)
+

## Introduction

@@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
/usr/share/doc/libhello-dev/copyright
~#
```
+
+## Creation of local apt repo caching upstream Debian packages
+
+### Motivation
+
+Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
+
+### Solution
+
+ - Trigger creation of local apt caching Debian packages during image generation.
+
+```
+bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
+```
+
+ - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
+
+```
+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"
+```
+ - Remove build artifacts to use only local base-apt:
+
+```
+sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
+
+```
+
+ - Trigger again generation of image (now using local caching repo):
+
+```
+bitbake multiconfig:qemuarm-stretch:isar-image-base
+```
+
+### Limitation
+
+So far the local base-apt repo is not gpg signed.
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index b8b8495..f3a960a 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -165,3 +165,7 @@ IMAGE_INSTALL = "example-hello example-raw example-module enable-fsck"
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "0"
+
+#

+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"

--
2.11.0

[Claudius Heine]

Hi Maxim,

find ... | while read package; do

or

while read package; do ...; done <<< $(find ...)

is better performance wise since the loop is iterated as results come in.

> + # NOTE: due to packages stored by reprepro are not modified, we can
> + # use search by filename to check if package is already in repo. In
> + # addition, m5sums could be compared to ensure, that package is the
> + # same and should not be overwritten. This method is easier and more
> + # robust than querying reprepro by name.
> +
> + # Check if this package is taken from Isar-apt, if so - ingore it.
> + isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $package)

Don't you need a `${package##*/}` here to remove the leading path?

> + if [ -n "$isar_package" ]; then
> + # Check if MD5 sums are iendtical. This helps to avoid the case

identical

> + # when packages is overriden from another repo.

overridden

> + md1=$(md5sum $package)
> + md2=$(md5sum $isar_package)

md5sum produces output with like this:

$ md5sum conf/local.conf
c543793c1df3b1f45a5555d92b6f3ee2 conf/local.conf

So you need to remove that before comparing. The same issues are in the
following code as well.

Cheers,
Claudius

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

[Claudius Heine]

Hi,

No that doesn't make sense to me.

In my opinion its very important to not have such flags or commands.
Caching and reproducible builds should be the default operation,
upgrading or removing the cache should be an extra step.

The cache should be created and used on every image build. On the first
build the cache is empty and on the next build its full, so it will be used.

Use apt preferences to git the cache a higher priority than the upstream
repos to archive that.

Also if the sources.list is different when setting
`ISAR_USE_CACHED_BASE_REPO` than the image is different from the first one.

Also please avoid those binary settings like
`ISAR_USE_CACHED_BASE_REPO`. That just bad design because it leads to
every tiny setting having an own variable. Use FEATURE variables with
entries like:

`ISAR_BUILD_FEATURE = "cross-build use-cache ..."`

Yes cross-builds should have been done like this as well IMO.

Cheers,
Claudius

> +```
> + - Remove build artifacts to use only local base-apt:
> +
> +```
> +sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
> +
> +```
> +
> + - Trigger again generation of image (now using local caching repo):
> +
> +```
> +bitbake multiconfig:qemuarm-stretch:isar-image-base
> +```
> +
> +### Limitation
> +
> +So far the local base-apt repo is not gpg signed.
>

--

[Claudius Heine]

On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:

> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
> ---
> doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/doc/user_manual.md b/doc/user_manual.md
> index db58cf1..d69dd0d 100644
> --- a/doc/user_manual.md
> +++ b/doc/user_manual.md
> @@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
> - [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
> - [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
> - [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)
> + - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)

On another point. Why not use links like?

[Creation of local apt repo caching upstream Debian

packages](#creation-repo-caching-upstream-debian)

[Jan Kiszka]

On 02.10.18 16:02, [ext] Claudius Heine wrote:
> Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`. That
> just bad design because it leads to every tiny setting having an own variable.
> Use FEATURE variables with entries like:
>
> `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
>
> Yes cross-builds should have been done like this as well IMO.

We do have a process for "properly" breaking the recipe API by now... ;)

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Claudius Heine]

Hi Maxim,

On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:

I don't like those variable names. IMO it would be ok to break the
variable backwards compatibility here and unify that.

How about:

REPO_ISAR_DIR
REPO_ISAR_DB_DIR
REPO_CACHE_DIR
REPO_CACHE_DB_DIR

The db directory is reprepro specific so the main variable should point
the the correct repository path. IMO

> diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
> new file mode 100644
> index 0000000..6ff1164
> --- /dev/null
> +++ b/meta/recipes-devtools/base-apt/base-apt.bb
> @@ -0,0 +1,31 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2018 ilbers GmbH
> +
> +SRC_URI = "file://distributions.in"
> +
> +inherit base-apt-helper
> +
> +CACHE_CONF_DIR = "${BASE_APT_DIR}/${DISTRO_NAME}/conf"
> +do_cache_config[dirs] = "${CACHE_CONF_DIR}"
> +do_cache_config[stamp-extra-info] = "${DISTRO}"
> +do_cache_config[lockfiles] = "${BASE_APT_DIR}/isar.lock"
> +
> +# Generate reprepro config for current distro if it doesn't exist. Once it's
> +# generated, this task should do nothing.
> +do_cache_config() {
> + if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> + sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \

Mixing DISTRO_NAME and DISTRO_SUITE. They are different, try to be
consistent.

Cheers,
Claudius

> + ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
> + fi
> +
> + path_cache="${BASE_APT_DIR}/${DISTRO_NAME}"
> + path_databases="${BASE_APT_DB}/${DISTRO_NAME}"
> +
> + if [ ! -d "${path_databases}" ]; then
> + reprepro -b ${path_cache} \
> + --dbdir ${path_databases} \
> + export ${DISTRO_SUITE}
> + fi
> +}
> +
> +addtask cache_config after do_build
> diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
> new file mode 100644
> index 0000000..cc82c57
> --- /dev/null
> +++ b/meta/recipes-devtools/base-apt/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf arm64 amd64 source
> +Components: main
>

--

[Claudius Heine]

Hi,

what also belongs to this patch are the options that force apt not to
remove anything from the cache like APT::Clean-Installed just to be save
that no automatic cleaning is done by apt.

Cheers,
Claudius

On 10/2/18 2:19 PM, Maxim Yu. Osipov wrote:

[Baurzhan Ismagulov]

On Tue, Oct 02, 2018 at 04:02:54PM +0200, Claudius Heine wrote:
> > +# Uncomment this to enable use of cached base repository
> > +#ISAR_USE_CACHED_BASE_REPO ?= "1"
>
> No that doesn't make sense to me.
>
> In my opinion its very important to not have such flags or commands. Caching
> and reproducible builds should be the default operation, upgrading or
> removing the cache should be an extra step.

The first step is to provide the mechanism which is usable manually or
automatically. Defining policy and setting it as default is best done when the
feature has been stabilized; I'd be reluctant to slap a new feature as default
as soon as it is introduced. I'd suggest to address that in later steps. That
said, the option doesn't take it away from you -- just set it to true in your
configuration.

> Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`.
> That just bad design because it leads to every tiny setting having an own
> variable. Use FEATURE variables with entries like:
>
> `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
>
> Yes cross-builds should have been done like this as well IMO.

Hmm, that's an interesting idea. Conceptually, I fail to see any difference
from boolean vars: One still has boolean feature names for every tiny setting.
I personally don't like features since they are difficult to remove from
default settings.


With kind regards,
Baurzhan.

[Claudius Heine]

Hi Baurzhan,

Quoting Baurzhan Ismagulov (2018-10-04 11:03:11)

> On Tue, Oct 02, 2018 at 04:02:54PM +0200, Claudius Heine wrote:
> > > +# Uncomment this to enable use of cached base repository
> > > +#ISAR_USE_CACHED_BASE_REPO ?= "1"
> >
> > No that doesn't make sense to me.
> >
> > In my opinion its very important to not have such flags or commands. Caching
> > and reproducible builds should be the default operation, upgrading or
> > removing the cache should be an extra step.
>
> The first step is to provide the mechanism which is usable manually or
> automatically. Defining policy and setting it as default is best done when the
> feature has been stabilized; I'd be reluctant to slap a new feature as default
> as soon as it is introduced. I'd suggest to address that in later steps. That
> said, the option doesn't take it away from you -- just set it to true in your
> configuration.

Ok, but then the machanism should be designed that way so that this
feature can be enabled all the time, and the build should work on the first run
with this setting enabled as well.

> > Also please avoid those binary settings like `ISAR_USE_CACHED_BASE_REPO`.
> > That just bad design because it leads to every tiny setting having an own
> > variable. Use FEATURE variables with entries like:
> >
> > `ISAR_BUILD_FEATURE = "cross-build use-cache ..."`
> >
> > Yes cross-builds should have been done like this as well IMO.
>
> Hmm, that's an interesting idea. Conceptually, I fail to see any difference
> from boolean vars: One still has boolean feature names for every tiny setting.

If you do 'bitbake -e ..' you see all used features in one variable
instead of searching for many different variables all over to find what
features are enabled.

> I personally don't like features since they are difficult to remove from
> default settings.

I don't know if I understand you correctly here. I think you meant that
removing an entry from a variable is difficult. If that is the case, then
you could just do:

ISAR_BUILD_FEATURE_remove = "use-cache"

As documented here: [1]

If you meant something different, please explain further.

Cheers,
Claudius

[1] https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#removing-override-style-syntax

>
>
> With kind regards,
> Baurzhan.
>
> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To post to this group, send email to isar-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20181004090311.GA12890%40yssyq.m.ilbers.de.
> For more options, visit https://groups.google.com/d/optout.

--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
Keyserver: hkp://pool.sks-keyservers.net

[Baurzhan Ismagulov]

On Fri, Oct 05, 2018 at 02:09:05PM +0200, Claudius Heine wrote:
> Ok, but then the machanism should be designed that way so that this
> feature can be enabled all the time, and the build should work on the first run
> with this setting enabled as well.

The mechanism doesn't preclude that. For this step, our goal is to have it
under user's control, explicitly leaving the policy you'd like to have to the
next steps. When we arrive there, we could decide whether
ISAR_USE_CACHED_BASE_REPO should become the default, and whether we need it to
be configurable. Till then, I'd really like to provide the choice for the user.

> If you do 'bitbake -e ..' you see all used features in one variable
> instead of searching for many different variables all over to find what
> features are enabled.

With a long enough list, it would end up being many different feature names all
over. I guess we'd have to play with that when we implement something and see
ourselves. For now, I don't see any decisive advantage.

> I don't know if I understand you correctly here. I think you meant that
> removing an entry from a variable is difficult. If that is the case, then
> you could just do:
>
> ISAR_BUILD_FEATURE_remove = "use-cache"

Yes, this is what I meant, thanks for the pointer. It does address the problem
of removing and, as I've read, produces also better results when appending.


With kind regards,
Baurzhan.

[Maxim Yu. Osipov]

Hello everybody,

This series contains fixes/improvements suggested by Claudius Heine
during v1 series review.

Changes to v2:
- Fixes/improvements found in base-apt-helper class
- Unifies path names to local repositories

1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages

2) Caches in base-apt repo upstream Debian packages during image generation.

3) After this step, image can be built offline using only base-apt repo.

Usage instructions can be found in the last patch.

TODO:
Sign the repo with gpg

Kind regards,
Maxim.

Alexander Smirnov (8):
base-apt: Add helper class

base-apt: Introduce base implementation

isar-boot-strap: Add option to keep cache
image: Add cache_base_repo task
isar-bootstrap: Make possible to reuse the cache
buildchroot: Make it buildable from base-apt
workaround: Use --allow-unauthenticated working with base-apt
local.conf: Add option to use cached base repository

Maxim Yu. Osipov (2):
meta: Unify path names to local repositories

doc: Creation of local apt repo caching upstream Debian packages

doc/user_manual.md | 39 ++++++++++++++++
meta-isar/conf/layer.conf | 10 ++--
meta-isar/conf/local.conf.sample | 4 ++
meta-isar/recipes-core/images/isar-image-base.bb | 3 +-

meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/dpkg-base.bbclass | 10 ++--
meta/classes/dpkg.bbclass | 2 +-
meta/classes/image.bbclass | 25 +++++++++-
meta/classes/isar-bootstrap-helper.bbclass | 17 +++++--
.../isar-bootstrap/files/base-apt-sources | 1 +

.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 +++++++++----
meta/recipes-devtools/base-apt/base-apt.bb | 31 +++++++++++++
.../base-apt/files/distributions.in | 3 ++
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-

meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++--
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
17 files changed, 213 insertions(+), 32 deletions(-)

create mode 100644 meta/classes/base-apt-helper.bbclass
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources

create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in

--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta/recipes-devtools/base-apt/base-apt.bb | 31 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +++

2 files changed, 34 insertions(+)

create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
new file mode 100644

index 0000000..2370905

--- /dev/null
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -0,0 +1,31 @@
+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+
+SRC_URI = "file://distributions.in"
+
+inherit base-apt-helper
+

+CACHE_CONF_DIR = "${REPO_BASE_DIR}/${DISTRO_NAME}/conf"

+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"

+do_cache_config[lockfiles] = "${REPO_BASE_DIR}/isar.lock"

+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \

+ ${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+

+ path_cache="${REPO_BASE_DIR}/${DISTRO_NAME}"
+ path_databases="${REPO_BASE_DB_DIR}/${DISTRO_NAME}"

+
+ if [ ! -d "${path_databases}" ]; then
+ reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DISTRO_SUITE}
+ fi
+}
+
+addtask cache_config after do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
new file mode 100644
index 0000000..cc82c57
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf arm64 amd64 source
+Components: main
--

2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

base-apt intended to store original upstream debs to re-use them
later offline. This class helps to populate base-apt with the packages
used during build.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Reviewed-by: Claudius Heine <c...@denx.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta-isar/conf/layer.conf | 4 +++
meta/classes/base-apt-helper.bbclass | 54 ++++++++++++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
create mode 100644 meta/classes/base-apt-helper.bbclass

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index cd42f06..22b2ff2 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -25,3 +25,7 @@ DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"

# Path to the Isar databases used by `reprepro`

DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
+

+# Base apt repository paths

+REPO_BASE_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
+REPO_BASE_DB_DIR ?= "${DEPLOY_DIR}/base-apt/db"
diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
new file mode 100644
index 0000000..9c03a7e
--- /dev/null
+++ b/meta/classes/base-apt-helper.bbclass
@@ -0,0 +1,54 @@

+# This software is a part of ISAR.
+# Copyright (C) 2018 ilbers GmbH
+

+DISTRO_NAME ?= "${@ d.getVar('DISTRO', True).split('-')[0]}"
+DISTRO_SUITE ?= "${@ d.getVar('DISTRO', True).split('-')[1]}"
+
+populate_base_apt() {
+ search_dir=$1
+

+ find $search_dir -name '*.deb' | while read package; do

+ # NOTE: due to packages stored by reprepro are not modified, we can
+ # use search by filename to check if package is already in repo. In
+ # addition, m5sums could be compared to ensure, that package is the
+ # same and should not be overwritten. This method is easier and more
+ # robust than querying reprepro by name.
+
+ # Check if this package is taken from Isar-apt, if so - ingore it.

+ base_name=${package##*/}
+ isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $base_name)

+ if [ -n "$isar_package" ]; then

+ # Check if MD5 sums are identical. This helps to avoid the case
+ # when packages is overridden from another repo.
+ md1=$(md5sum $package | cut -d ' ' -f 1)
+ md2=$(md5sum $isar_package | cut -d ' ' -f 1)

+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+ fi
+
+ # Check if this package is already in base-apt

+ isar_package=$(find ${REPO_BASE_DIR}/${DISTRO_NAME} -name $base_name)

+ if [ -n "$isar_package" ]; then

+ md1=$(md5sum $package | cut -d ' ' -f 1)
+ md2=$(md5sum $isar_package | cut -d ' ' -f 1)

+ if [ "$md1" = "$md2" ]; then
+ continue
+ fi
+
+ # md5sum differs, so remove the package from base-apt

+ name=$($base_name | cut -d '_' -f 1)
+ reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
+ --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \

+ -C main -A ${DISTRO_ARCH} \
+ remove ${DISTRO_SUITE} \
+ $name
+ fi
+

+ reprepro -b ${REPO_BASE_DIR}/${DISTRO_NAME} \
+ --dbdir ${REPO_BASE_DB_DIR}/${DISTRO_NAME} \

+ -C main \
+ includedeb ${DISTRO_SUITE} \
+ $package
+ done
+}
--

2.11.0

[Maxim Yu. Osipov]

Rename DEPLOY_DIR_APT, DEPLOY_DIR_APT to REPO_ISAR_DIR and
REPO_ISAR_DB_DIR correspondingly to unify with cached base
repository names REPO_BASE_DIR and REPO_BASE_DB_DIR.

Suggested-by: Claudius Heine <c...@denx.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta-isar/conf/layer.conf | 8 +++-----
meta/classes/base-apt-helper.bbclass | 2 +-
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/dpkg-base.bbclass | 10 +++++-----
meta/classes/dpkg.bbclass | 2 +-
meta/classes/image.bbclass | 4 ++--
meta/classes/isar-bootstrap-helper.bbclass | 4 ++--
meta/recipes-devtools/isar-apt/isar-apt.bb | 8 ++++----
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
9 files changed, 20 insertions(+), 22 deletions(-)

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 22b2ff2..4aa1cf1 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -20,11 +20,9 @@ LAYERDIR_isar = "${LAYERDIR}"

# Codename of the repository created by the caching class
DEBDISTRONAME = "isar"

-# Path to the Isar apt repository
-DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"

-

-# Path to the Isar databases used by `reprepro`
-DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"

+# Isar apt repository paths
+REPO_ISAR_DIR ?= "${DEPLOY_DIR}/isar-apt/apt"
+REPO_ISAR_DB_DIR ?= "${DEPLOY_DIR}/isar-apt/db"

# Base apt repository paths
REPO_BASE_DIR ?= "${DEPLOY_DIR}/base-apt/apt"
diff --git a/meta/classes/base-apt-helper.bbclass b/meta/classes/base-apt-helper.bbclass
index 9c03a7e..ff4a3d5 100644
--- a/meta/classes/base-apt-helper.bbclass
+++ b/meta/classes/base-apt-helper.bbclass
@@ -16,7 +16,7 @@ populate_base_apt() {

# Check if this package is taken from Isar-apt, if so - ingore it.

base_name=${package##*/}
- isar_package=$(find ${DEPLOY_DIR_APT}/${DISTRO} -name $base_name)
+ isar_package=$(find ${REPO_ISAR_DIR}/${DISTRO} -name $base_name)
if [ -n "$isar_package" ]; then

# Check if MD5 sums are identical. This helps to avoid the case

# when packages is overridden from another repo.

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index f67335e..870d27c 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -25,7 +25,7 @@ buildchroot_do_mounts() {
sudo flock ${MOUNT_LOCKFILE} -c ' \
set -e
if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then
- mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+ mount --bind ${REPO_ISAR_DIR}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads
mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
mount -t proc none ${BUILDCHROOT_DIR}/proc
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index edd111b..f1b127c 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -76,8 +76,8 @@ CLEANFUNCS += "repo_clean"
repo_clean() {
PACKAGES=$(cd ${S}/..; ls *.deb | sed 's/\([^_]*\).*/\1/')
if [ -n "${PACKAGES}" ]; then
- reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
- --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
+ --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \

-C main -A ${DISTRO_ARCH} \

remove ${DEBDISTRONAME} \
${PACKAGES}
@@ -87,8 +87,8 @@ repo_clean() {
# Install package to Isar-apt
do_deploy_deb() {
repo_clean
- reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
- --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+ reprepro -b ${REPO_ISAR_DIR}/${DISTRO} \
+ --dbdir ${REPO_ISAR_DB_DIR}/${DISTRO} \
-C main \
includedeb ${DEBDISTRONAME} \
${S}/../*.deb
@@ -96,6 +96,6 @@ do_deploy_deb() {

addtask deploy_deb after do_build
do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_deploy_deb[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[depends] = "isar-apt:do_cache_config"
do_deploy_deb[dirs] = "${S}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index da0b40d..f74c9c9 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -13,7 +13,7 @@ do_install_builddeps() {

addtask install_builddeps after do_prepare_build before do_build
# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

# Build package from sources using build script
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ea8cbf5..e948dea 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -102,7 +102,7 @@ SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO

do_populate_sdk() {
# Copy isar-apt with deployed Isar packages
- sudo cp -Trpfx ${DEPLOY_DIR_APT}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt

# Purge apt cache to make image slimmer
sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/*
@@ -158,7 +158,7 @@ do_install_imager_deps() {

do_install_imager_deps[depends] = "buildchroot-target:do_build"
do_install_imager_deps[deptask] = "do_deploy_deb"
-do_install_imager_deps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_imager_deps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_imager_deps[stamp-extra-info] = "${DISTRO}-${MACHINE}"

addtask install_imager_deps before do_build
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 597f6b1..f046216 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -93,9 +93,9 @@ setup_root_file_system() {
sudo tee "$ROOTFSDIR/etc/apt/preferences.d/isar" >/dev/null

if [ ${COPYISARAPT} ]; then
- sudo cp -Trpfx ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
else
- sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} $ROOTFSDIR/isar-apt
+ sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
fi

sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc

diff --git a/meta/recipes-devtools/isar-apt/isar-apt.bb b/meta/recipes-devtools/isar-apt/isar-apt.bb
index 9c31d12..a959691 100644
--- a/meta/recipes-devtools/isar-apt/isar-apt.bb
+++ b/meta/recipes-devtools/isar-apt/isar-apt.bb
@@ -3,10 +3,10 @@

SRC_URI = "file://distributions.in"

-CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+CACHE_CONF_DIR = "${REPO_ISAR_DIR}/${DISTRO}/conf"
do_cache_config[dirs] = "${CACHE_CONF_DIR}"
do_cache_config[stamp-extra-info] = "${DISTRO}"
-do_cache_config[lockfiles] = "${DEPLOY_DIR_APT}/isar.lock"
+do_cache_config[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"

# Generate reprepro config for current distro if it doesn't exist. Once it's

# generated, this task should do nothing.

@@ -16,8 +16,8 @@ do_cache_config() {

${WORKDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions

fi

- path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
- path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+ path_cache="${REPO_ISAR_DIR}/${DISTRO}"
+ path_databases="${REPO_ISAR_DB_DIR}/${DISTRO}"

if [ ! -d "${path_databases}" ]; then

reprepro -b ${path_cache} \
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 7c6b624..8c89637 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -36,7 +36,7 @@ do_install_builddeps() {

addtask install_builddeps after do_prepare_build before do_build
# apt and reprepro may not run in parallel, acquire the Isar lock
-do_install_builddeps[lockfiles] += "${DEPLOY_DIR_APT}/isar.lock"
+do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

dpkg_runbuild() {
--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

.../isar-bootstrap/files/base-apt-sources | 1 +

.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32 ++++++++++++++++------
2 files changed, 24 insertions(+), 9 deletions(-)
create mode 100644 meta/recipes-core/isar-bootstrap/files/base-apt-sources

diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources b/meta/recipes-core/isar-bootstrap/files/base-apt-sources

new file mode 100644

index 0000000..594db56
--- /dev/null
+++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
@@ -0,0 +1 @@
+deb file:///base-apt/debian {DISTRO} main
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc

index cfad136..cc1791c 100644

--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -12,18 +12,24 @@ SRC_URI = " \
file://isar-apt.conf \
file://isar-apt-fallback.conf \
file://locale \
- file://chroot-setup.sh"
+ file://chroot-setup.sh \
+ file://base-apt-sources"
PV = "1.0"

DEBOOTSTRAP ?= "qemu-debootstrap"
ROOTFSDIR = "${WORKDIR}/rootfs"
APTPREFS = "${WORKDIR}/apt-preferences"
APTSRCS = "${WORKDIR}/apt-sources"
+BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
APTKEYFILES = ""
APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""

+DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"

+
+inherit base-apt-helper
+
python () {
from urllib.parse import urlparse
distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
@@ -171,6 +177,10 @@ isar_bootstrap() {
esac
shift
done
+ debootstrap_args="--verbose --variant=minbase --include=locales "

+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

+ sed -i -e "s#{DISTRO}#"${DISTRO_SUITE}"#g" ${BASEAPTSRCS}
+ mkdir -p ${ROOTFSDIR}/base-apt

+ sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

The default ISAR behavior assumes to remove all the packages
from apt cache, but we need them to put into base-apt.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index f046216..b7cd64a 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

@@ -70,6 +70,7 @@ setup_root_file_system() {
--fstab) FSTAB=$2; shift ;;
--host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
--host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
+ --keep-apt-cache) KEEP_APT_CACHE=1 ;;
-*) bbfatal "$0: invalid option specified: $1" ;;
*) break ;;
esac
@@ -131,6 +132,11 @@ setup_root_file_system() {
/usr/bin/apt-get purge --yes $pkg
done
if [ ${CLEAN} ]; then
+ if [ ${KEEP_APT_CACHE} ]; then
+ mkdir -p ${WORKDIR}/apt_cache
+ sudo mv $(find $ROOTFSDIR/var/cache/apt -name '*.deb') ${WORKDIR}/apt_cache
+ sudo chown $USER ${WORKDIR}/apt_cache/*
+ fi
sudo -E chroot "$ROOTFSDIR" \
/usr/bin/apt-get autoremove --purge --yes
sudo -E chroot "$ROOTFSDIR" \
--

2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

This task puts all the packages using in build to base-apt.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta-isar/recipes-core/images/isar-image-base.bb | 3 ++-
meta/classes/image.bbclass | 21 +++++++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 4899593..bf606cc 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -41,7 +41,8 @@ devtmpfs /dev devtmpfs mode=0755,nosuid 0 0
# End /etc/fstab
EOF

- setup_root_file_system --clean --fstab "${WORKDIR}/fstab" \
+ setup_root_file_system --clean --keep-apt-cache \
+ --fstab "${WORKDIR}/fstab" \
"${IMAGE_ROOTFS}" ${IMAGE_PREINSTALL} ${IMAGE_INSTALL}

# Configure root filesystem

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e948dea..05ff06f 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass

@@ -123,6 +123,27 @@ do_populate_sdk[depends] = "sdkchroot:do_build"

addtask populate_sdk after do_rootfs

+inherit base-apt-helper
+
+do_cache_base_repo[depends] = "base-apt:do_cache_config"
+do_cache_base_repo[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_cache_base_repo() {
+ if [ -d ${WORKDIR}/apt_cache ]; then
+ populate_base_apt ${WORKDIR}/apt_cache
+ fi
+
+ if [ -d ${BUILDCHROOT_HOST_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_HOST_DIR}/var/cache/apt
+ fi
+
+ if [ -d ${BUILDCHROOT_TARGET_DIR}/var/cache/apt ]; then
+ populate_base_apt ${BUILDCHROOT_TARGET_DIR}/var/cache/apt

+ fi
+}
+

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta/classes/isar-bootstrap-helper.bbclass | 5 +++++
1 file changed, 5 insertions(+)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index b7cd64a..8f5e583 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

@@ -98,6 +98,11 @@ setup_root_file_system() {
else

sudo mount --bind ${REPO_ISAR_DIR}/${DISTRO} $ROOTFSDIR/isar-apt
fi

+

+ if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then

+ sudo mount --bind ${REPO_BASE_DIR} ${ROOTFSDIR}/base-apt

+ fi
+

sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs $ROOTFSDIR/dev
sudo mount -t proc none $ROOTFSDIR/proc

--
2.11.0

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

---
meta/classes/isar-bootstrap-helper.bbclass | 2 +-
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 8f5e583..24e1157 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

[Maxim Yu. Osipov]

From: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 4 ++++
1 file changed, 4 insertions(+)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index b8b8495..f3a960a 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -165,3 +165,7 @@ IMAGE_INSTALL = "example-hello example-raw example-module enable-fsck"
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "0"
+
+#

+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"

--
2.11.0

[Maxim Yu. Osipov]

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
---

doc/user_manual.md | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index db58cf1..d69dd0d 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -19,6 +19,8 @@ Copyright (C) 2016-2017, ilbers GmbH
- [Add a Custom Application](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#add-a-custom-application)
- [Enabling Cross-compilation](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#isar-cross-compilation)
- [Create an ISAR SDK root filesystem](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#create-an-isar-sdk-root-filesystem)

+ - [Creation of local apt repo caching upstream Debian packages](https://github.com/ilbers/isar/blob/master/doc/user_manual.md#creation-repo-caching-upstream-debian)

+

## Introduction

@@ -686,3 +688,40 @@ ii crossbuild-essential-armhf 12.3 all Inf
/usr/share/doc/libhello-dev/copyright
~#
```
+
+## Creation of local apt repo caching upstream Debian packages
+
+### Motivation
+
+Cache upstream debian packages to reduce time for further downloads and to be able to work offline.
+
+### Solution
+
+ - Trigger creation of local apt caching Debian packages during image generation.
+
+```
+bitbake -c cache_base_repo multiconfig:qemuarm-stretch:isar-image-base
+```
+
+ - Set `ISAR_USE_CACHED_BASE_REPO` in `conf/local.conf`:
+
+```

+# Uncomment this to enable use of cached base repository
+#ISAR_USE_CACHED_BASE_REPO ?= "1"

+```
+ - Remove build artifacts to use only local base-apt:
+
+```
+sudo rm -rf tmp/stamps/ tmp/work/ tmp/deploy/isar-apt/ tmp/deploy/images
+
+```
+
+ - Trigger again generation of image (now using local caching repo):
+
+```
+bitbake multiconfig:qemuarm-stretch:isar-image-base
+```
+
+### Limitation
+
+So far the local base-apt repo is not gpg signed.
--

2.11.0

[Maxim Yu. Osipov]

On 10/11/18 7:52 PM, Maxim Yu. Osipov wrote:
> Hello everybody,
>
> This series contains fixes/improvements suggested by Claudius Heine
> during v1 series review.
>
> Changes to v2:
> - Fixes/improvements found in base-apt-helper class
> - Unifies path names to local repositories
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>
> 2) Caches in base-apt repo upstream Debian packages during image generation.
>
> 3) After this step, image can be built offline using only base-apt repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg

Applied to the 'next'.

Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mos...@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[Jan Kiszka]

Could it be we have some regression here? After updating to current
next, a build without the cache enabled fails during do_rootfs with

| (Reading database ... 9600 files and directories currently installed.)
| Removing isar-cfg-localepurge (1.0+isar) ...
| find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
| chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)

I do not have the time to bisect, will just jump back for now.

[Maxim Yu. Osipov]

Hmmm... Of course, I've run CI build before applying to the 'next'
(see http://isar-build.org:8080/job/isar_mosipov_next/61).

I'll check if your problem is reproduced when build is not started from
the scratch.

Regards,
Maxim.

[Maxim Yu. Osipov]

Hi Jan,

I've built (cross compiled from the scratch) the
multiconfig:qemuarm-stretch:isar-image-base, after that I've checkouted
'next' branch and ran again 'bitbake
multiconfig:qemuarm-stretch:isar-image-base' in the same build
directory. Build was OK. Did you have another test case?


Kind regards,

[Henning Schild]

Am Thu, 11 Oct 2018 18:52:56 +0200
schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

This block should maybe be a function we reuse ...

> + fi
> +
> + # Check if this package is already in base-apt
> + isar_package=$(find ${REPO_BASE_DIR}/${DISTRO_NAME} -name
> $base_name)
> + if [ -n "$isar_package" ]; then
> + md1=$(md5sum $package | cut -d ' ' -f 1)
> + md2=$(md5sum $isar_package | cut -d ' ' -f 1)
> + if [ "$md1" = "$md2" ]; then
> + continue
> + fi

here.

Henning

[Henning Schild]

Am Thu, 11 Oct 2018 18:52:59 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

> From: Alexander Smirnov <asmi...@ilbers.de>
>
> The default ISAR behavior assumes to remove all the packages
> from apt cache, but we need them to put into base-apt.
>
> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index f046216..b7cd64a
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -70,6 +70,7 @@ setup_root_file_system() {
> --fstab) FSTAB=$2; shift ;;
> --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
> --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
> + --keep-apt-cache) KEEP_APT_CACHE=1 ;;
> -*) bbfatal "$0: invalid option specified: $1" ;;
> *) break ;;
> esac
> @@ -131,6 +132,11 @@ setup_root_file_system() {
> /usr/bin/apt-get purge --yes $pkg
> done
> if [ ${CLEAN} ]; then
> + if [ ${KEEP_APT_CACHE} ]; then

if [ ${KEEP_APT_CACHE} -eq 1 ]; then

[Henning Schild]

Am Thu, 11 Oct 2018 18:53:03 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

This should only be done when using the cache ... in fact it should not
be done at all!

Henning

[Henning Schild]

Hey,

i am kind of missing some context here. Is this really just a cache to
remove deb fetching from the build and allow offline builds? Or is this
somehow targeting reproducable builds?

If the latter is true i would like to see a summary of the earlier
discussions, a detailed description of what is happening and why. All
the commit messages of the series are far from verbose and the doc does
not make it much better.

Henning

Am Thu, 11 Oct 2018 18:52:55 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

[Henning Schild]

Am Thu, 11 Oct 2018 18:53:01 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

> From: Alexander Smirnov <asmi...@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
> Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
> ---
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> ++++++++++++++++------ 2 files changed, 24 insertions(+), 9
> deletions(-) create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources
>
> diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> b/meta/recipes-core/isar-bootstrap/files/base-apt-sources new file
> mode 100644 index 0000000..594db56
> --- /dev/null
> +++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> @@ -0,0 +1 @@
> +deb file:///base-apt/debian {DISTRO} main

Try "[trusted=yes]" in here and drop the workaround patch if possible.

Henning

[Henning Schild]

Am Thu, 11 Oct 2018 18:52:59 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

In my build USER is not set. So we fail with

>> chown: invalid user:
>> '/out/build/tmp/work/debian-stretch-amd64/isar-image-base/apt_cache/adduser_3.115_all.deb'

Where adduser is the first of *. Should be `whoami`.

Henning

[Henning Schild]

Am Tue, 23 Oct 2018 18:09:31 +0200
schrieb "[ext] Henning Schild" <henning...@siemens.com>:

What i was trying to say is, revert this because it breaks integrity
checking of all of Isar. If that breaks the series revert the whole
thing.

Henning

> Henning
>

[Baurzhan Ismagulov]

On Tue, Oct 23, 2018 at 06:14:56PM +0200, Henning Schild wrote:
> i am kind of missing some context here. Is this really just a cache to
> remove deb fetching from the build and allow offline builds? Or is this
> somehow targeting reproducable builds?
>
> If the latter is true i would like to see a summary of the earlier
> discussions, a detailed description of what is happening and why.

Yes, this is just a cache to allow offline builds. And yes, this is a small
step 0 towards reproducibility.

As I see it, previous discussions boil down to:

1. Doing automatic reproducibility right is complex and should be done in
steps.

2. Initial caching is a low-hanging fruit that can be addressed now and is
better than nothing.

This series provides (2).

For (1), I suggest to introduce a mechanism first. It should be usable
manually. It should start from minimal essential use cases (initial creation)
and be possibly extended to further use cases (update all, update single,
update single with deps, remove, etc.) as necessary. If desired, we could look
at optionally building it into the workflow of Isar after the mechanism
stabilizes (with policies such as automatically fetch on the first build,
update on every build, etc.). We can provide more details along the way.

On a related note, we've discussed these issues at ELCE. libapt and python-apt
seem to be what we need for the implementation.

With kind regards,
Baurzhan.

[Henning Schild]

Am Tue, 2 Oct 2018 14:19:03 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

> From: Alexander Smirnov <asmi...@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>

> Signed-off-by: Maxim Yu. Osipov <mos...@ilbers.de>
> ---
> .../isar-bootstrap/files/base-apt-sources | 1 +
> .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32
> ++++++++++++++++------ 2 files changed, 24 insertions(+), 9
> deletions(-) create mode 100644
> meta/recipes-core/isar-bootstrap/files/base-apt-sources
>
> diff --git a/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> b/meta/recipes-core/isar-bootstrap/files/base-apt-sources new file
> mode 100644 index 0000000..594db56
> --- /dev/null
> +++ b/meta/recipes-core/isar-bootstrap/files/base-apt-sources
> @@ -0,0 +1 @@
> +deb file:///base-apt/debian {DISTRO} main

> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index

> cfad136..9b5a894 100644 ---

> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -12,18
> +12,24 @@ SRC_URI = " \ file://isar-apt.conf \
> file://isar-apt-fallback.conf \
> file://locale \
> - file://chroot-setup.sh"
> + file://chroot-setup.sh \
> + file://base-apt-sources"
> PV = "1.0"
>
> DEBOOTSTRAP ?= "qemu-debootstrap"
> ROOTFSDIR = "${WORKDIR}/rootfs"
> APTPREFS = "${WORKDIR}/apt-preferences"
> APTSRCS = "${WORKDIR}/apt-sources"
> +BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
> APTKEYFILES = ""
> APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> DEBOOTSTRAP_KEYRING = ""
> DEPLOY_ISAR_BOOTSTRAP ?= ""
>
> +DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org

> file:///${BASE_APT_DIR} \n" if

> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> + +inherit base-apt-helper +
> python () {
> from urllib.parse import urlparse
> distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> @@ -171,6 +177,10 @@ isar_bootstrap() {
> esac
> shift
> done
> + debootstrap_args="--verbose --variant=minbase --include=locales "
> + if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
> + debootstrap_args="$debootstrap_args --no-check-gpg"

Can this be removed now that the one problematic repo is marked
trusted? Say someone has more upstream repos and uses offline just for
debian, they probably still want gpg verification.

Henning

> + sudo mount --bind ${BASE_APT_DIR}

[Henning Schild]

Hi,

how does series compare to "apt-cacher-ng"? Looks to me like that tool
does something very similar, and it most likely deals with gpg and
other details with more care.
I might be wrong, but if that tool was never considered ... i think it
should be before inventing it again.

Henning

Am Tue, 2 Oct 2018 14:18:58 +0200

schrieb "Maxim Yu. Osipov" <mos...@ilbers.de>:

> Hello everybody,
>
> This series by Alexander Smirnov,

>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian
> packages
>
> 2) Caches in base-apt repo upstream Debian packages during image
> generation.
>
> 3) After this step, image can be built offline using only base-apt
> repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg
>
> Kind regards,
> Maxim.
>
> Alexander Smirnov (8):
> base-apt: Add helper class

> base-apt: Introduce base implementaiton

> isar-boot-strap: Add option to keep cache
> image: Add cache_base_repo task
> isar-bootstrap: Make possible to reuse the cache
> buildchroot: Make it buildable from base-apt
> workaround: Use --allow-unauthenticated working with base-apt
> local.conf: Add option to use cached base repository
>

> Maxim Yu. Osipov (1):

> doc: Creation of local apt repo caching upstream Debian packages
>
> doc/user_manual.md | 39
> ++++++++++++++++ meta-isar/conf/layer.conf |
> 10 ++-- meta-isar/conf/local.conf.sample | 4 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 3 +-

> meta/classes/base-apt-helper.bbclass | 53
> ++++++++++++++++++++++
> meta/classes/image.bbclass | 21 +++++++++
> meta/classes/isar-bootstrap-helper.bbclass | 13
> +++++- .../isar-bootstrap/files/base-apt-sources | 1
> + .../recipes-core/isar-bootstrap/isar-bootstrap.inc | 32

> +++++++++---- meta/recipes-devtools/base-apt/base-apt.bb | 31
> +++++++++++++ .../base-apt/files/distributions.in | 3

> ++ meta/recipes-devtools/buildchroot/files/deps.sh | 2 +- 12
> files changed, 196 insertions(+), 16 deletions(-) create mode 100644

[Jan Kiszka]

On 02.10.18 14:18, Maxim Yu. Osipov wrote:
> Hello everybody,
>
> This series by Alexander Smirnov,
>
> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>
> 2) Caches in base-apt repo upstream Debian packages during image generation.
>
> 3) After this step, image can be built offline using only base-apt repo.
>
> Usage instructions can be found in the last patch.
>
> TODO:
> Sign the repo with gpg

There are more, in fact:

- enable support for cross-build: dependencies of buildchroot-target are
not cached which breaks the rebuild - probably related to the strange
way of creating the repo via a task. Maybe apt-cacher can help.

- move base_apt out of tmp - it belongs to downloads. That will also
make it easier to purge tmp completely in order trigger a rebuild.

- get rid of the control knobs, at least make them opt-out instead of
opt-in. I see no reason why the cache can't be created by default and
why it can't be used by default if it's there - we do the same with
other downloads.

I'm also curious if the apt-cacher-ng Henning mentioned can help to make things
simpler.

Thanks,

[Jan Kiszka]

On 04.11.18 11:07, Jan Kiszka wrote:> On 02.10.18 14:18, Maxim Yu. Osipov wrote:
>> Hello everybody,
>>
>> This series by Alexander Smirnov,
>>
>> 1) Introduces dedicated local apt (base-apt) repo for upstream Debian packages
>>
>> 2) Caches in base-apt repo upstream Debian packages during image generation.
>>
>> 3) After this step, image can be built offline using only base-apt repo.
>>
>> Usage instructions can be found in the last patch.
>> TODO:
>> Sign the repo with gpg
>
> There are more, in fact:
>
> - enable support for cross-build: dependencies of buildchroot-target are
> not cached which breaks the rebuild - probably related to the strange
> way of creating the repo via a task. Maybe apt-cacher can help.

Played a bit more with it, and it's not cross-build:

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ad19f3d..d033cf5 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -143,7 +143,7 @@ do_cache_base_repo() {
fi
}

-addtask cache_base_repo after do_rootfs
+addtask cache_base_repo after do_rootfs do_install_imager_deps

# Imager are expected to run natively, thus will use the target buildchroot.
ISAR_CROSS_COMPILE = "0"

I.e., we simply ignored the dependencies of the imager. Cross-building
is fine now as well.

>
> - move base_apt out of tmp - it belongs to downloads. That will also
> make it easier to purge tmp completely in order trigger a rebuild.
>
> - get rid of the control knobs, at least make them opt-out instead of
> opt-in. I see no reason why the cache can't be created by default and
> why it can't be used by default if it's there - we do the same with
> other downloads.

- make sure you can switch between cached and non-cached build on-the-
fly, without requiring to purge all build results. That's important
when you want to add packages that are not yet cached, at least as
long as we allow to control the mode manually.

[Jan Kiszka]

On 02.11.18 13:00, [ext] Henning Schild wrote:
> Hi,
>
> how does series compare to "apt-cacher-ng"? Looks to me like that tool
> does something very similar, and it most likely deals with gpg and
> other details with more care.
> I might be wrong, but if that tool was never considered ... i think it
> should be before inventing it again.

Could someone comment on this?

Thanks,

[Baurzhan Ismagulov]

On Fri, Nov 09, 2018 at 09:04:43AM +0100, Jan Kiszka wrote:
> > how does series compare to "apt-cacher-ng"? Looks to me like that tool
> > does something very similar, and it most likely deals with gpg and
> > other details with more care.
> > I might be wrong, but if that tool was never considered ... i think it
> > should be before inventing it again.

We had discussed the proxy approach last year. It doesn't support apt's
pluggable fetch schemes and is, being a pure network tool, completely
apt-agnostic. One can get the initial mirroring done with it, but implementing
further use cases like updating the cache without deleting it or selectively
picking just a few packages and, optionally, their dependencies requires apt
introspection. I think adding and maintaining that in a C++ tool designed for a
different purpose is not the way to go. That is why we should be reusing apt
code to stay up-to-date with Debian. libapt + python-apt provide what we need,
and other projects have good experience with that. aptly claims selective
update support, so that is also on our list.

With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

Hello Jan,

On Tue, Oct 23, 2018 at 12:56:11PM +0300, Maxim Yu. Osipov wrote:
> > > Could it be we have some regression here? After updating to current
> > > next, a build without the cache enabled fails during do_rootfs with
> > >
> > > | (Reading database ... 9600 files and directories currently installed.)
> > > | Removing isar-cfg-localepurge (1.0+isar) ...
> > > | find: '/build/tmp/work/debian-stretch-armhf/isar-image-base/rootfs/var/cache/apt/archives/partial': Permission denied
> > > | chown: invalid user: '/build/tmp/work/debian-stretch-armhf/isar-image-base/apt_cache/adduser_3.115_all.deb'
> > > | WARNING: exit code 1 from a shell command.
> > > | ERROR: Function failed: do_rootfs (log file is located at /build/tmp/work/debian-stretch-armhf/isar-image-base/temp/log.do_rootfs.5627)

...

> I've built (cross compiled from the scratch) the
> multiconfig:qemuarm-stretch:isar-image-base, after that I've checkouted
> 'next' branch and ran again 'bitbake
> multiconfig:qemuarm-stretch:isar-image-base' in the same build directory.
> Build was OK. Did you have another test case?

This issue blocks reproducibility. How can we reproduce it?

With kind regards,
Baurzhan.

[Jan Kiszka]

Might have been a glitch related to artifacts left over from builds with older
versions. A lot of things changed since then, specifically regarding
permissions. I currently do not see this issue, I rather see others
(bitbake...). Let's just move on.

[Henning Schild]

Am Fri, 9 Nov 2018 10:14:18 +0100
schrieb Baurzhan Ismagulov <i...@radix50.net>:

> On Fri, Nov 09, 2018 at 09:04:43AM +0100, Jan Kiszka wrote:
> > > how does series compare to "apt-cacher-ng"? Looks to me like that
> > > tool does something very similar, and it most likely deals with
> > > gpg and other details with more care.
> > > I might be wrong, but if that tool was never considered ... i
> > > think it should be before inventing it again.
>
> We had discussed the proxy approach last year. It doesn't support
> apt's pluggable fetch schemes and is, being a pure network tool,
> completely apt-agnostic.

That can also be seen as a benefit. It can in fact be replaced by any
caching proxy, squid ... you name it.

> One can get the initial mirroring done with
> it, but implementing further use cases like updating the cache
> without deleting it or selectively picking just a few packages and,
> optionally, their dependencies requires apt introspection.

Valid point, but is that a strong enough feature to roll our own?
Flushing the cache and rewarming it from scratch does not take too much
space and time. And you do not risk making a mistake. And the
implementation we have in Isar at the moment does not support that
either.

> I think
> adding and maintaining that in a C++ tool designed for a different
> purpose is not the way to go.

We would be using that tool, or any other http/ftp proxy. The language
it is written in or who maintains it does not matter all. If we do not
like it we change .. easy. Implementing our own, we have to maintain
it ...

> That is why we should be reusing apt
> code to stay up-to-date with Debian. libapt + python-apt provide what
> we need, and other projects have good experience with that. aptly
> claims selective update support, so that is also on our list.

Ok maybe aptly. If Isar itself was completely agnostic, people could
choose their cache. Some might go for apt-agnostic some for plain http.
Isar itself could carry documentation and suggestions on how to set
that up.

Henning

> With kind regards,
> Baurzhan.
>