[RFC][PATCH 0/6] Isar build reproducibility

[Alexander Smirnov]

Hello all,

this series proposes the way how build reproducibility could be implemented
in Isar. General idea is to get the list of all the necessary packages for
build, fetch them and create local repo, that will be used for further
builds/

Briefly speeking, it works like the following:

1. User sets the list of images that should be 'reproducible' in
BASE_APT_IMAGES variable in local.conf file.

2. Based on the list of images above, Isar will derive all the run-time and
build dependencies for these images.

3. Using multistrap, Isar will fetch the list of packages and create base-apt
local repository.

4. Now buildchroot and image root filesystems are generated using base-apt.

Some notes:

1. base-apt repository is mounted to buildchroot, so Isar packages are
able to install necessary deps via apt-get.

2. bitbake events are used to clean up buildchroot. I haven't found another
way how base-apt could be unmounted. So it's mounted once before any
package starts building and unmounted by bitbake event:
bb.event.BuildCompleted.

This series works good with latest next. Any comments are welcome.

Happy New Year 2018! :-)

With best regards,
Alex

Alexander Smirnov (6):
base-apt: Introduce fetching upstream apt
base-apt: Add to pipeline
buildchroot: Switch to base-apt
buildchroot: Add mount/umount for 'base-apt'
image: Switch to base-apt
base-apt: Add possibility to reuse

meta-isar/conf/local.conf.sample | 8 ++
.../recipes-core/images/files/multistrap.conf.in | 18 +---
meta-isar/recipes-core/images/isar-image-base.bb | 5 +-
meta/classes/dpkg-base.bbclass | 16 +++-
meta/classes/image.bbclass | 13 ++-
meta/classes/isar-events.bbclass | 21 +++++
meta/conf/isar-bitbake.conf | 2 +
meta/recipes-devtools/base-apt/base-apt.bb | 97 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +
.../base-apt/files/multistrap.conf.in | 28 +++++++
meta/recipes-devtools/buildchroot/buildchroot.bb | 28 ++++++-
.../buildchroot/files/configscript.sh | 1 -
.../buildchroot/files/multistrap.conf.in | 18 +---
13 files changed, 217 insertions(+), 41 deletions(-)
create mode 100644 meta/classes/isar-events.bbclass
create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in

--
2.1.4

[Alexander Smirnov]

This patch introduces mechanism how to fetch deb packages from dedicated
upstream apt repos and store them localy. Local repository is called
'base-apt' and it will be used to generate buildchroot and image root
filesystems. Using 'base-apt' will guarantee build reproducibility between
builds.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---
meta/conf/isar-bitbake.conf | 2 +
meta/recipes-devtools/base-apt/base-apt.bb | 79 ++++++++++++++++++++++
.../base-apt/files/distributions.in | 3 +
.../base-apt/files/multistrap.conf.in | 28 ++++++++
4 files changed, 112 insertions(+)

create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index 5a26743..df54399 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"

+BASE_APT_DIR ?= "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt"
+
# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "noop"

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
new file mode 100644
index 0000000..b292d89
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -0,0 +1,79 @@
+# Caching upstream apt repository to local one.
+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+DESCRIPTION = "Upstream apt caching"
+
+LICENSE = "gpl-2.0"
+LIC_FILES_CHKSUM = "file://${LAYERDIR_isar}/licenses/COPYING.GPLv2;md5=751419260aa954499f7abaabaa882bbe"
+
+FILESPATH =. "${LAYERDIR_core}/recipes-devtools/base-apt/files:"
+SRC_URI = "file://distributions.in \
+ file://multistrap.conf.in \
+ "
+
+BASE_PREINSTALL ?= ""
+
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
+
+BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf"
+do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}"
+do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
+do_get_base_apt() {
+ # Adjust multistrap config
+ sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \
+ -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
+ -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
+ -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
+ -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
+ -e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
+ -e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \
+ -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
+ -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
+ -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+ "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
+
+ # Fetch deb packages
+ sudo -E multistrap \
+ -a ${DISTRO_ARCH} \
+ -d "${WORKDIR}/download" \
+ -f "${WORKDIR}/multistrap.conf" \
+ > ${WORKDIR}/multistrap.log 2>&1
+
+ if [ ! -e "${BASE_APT_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DISTRO_SUITE}"#g" \
+ ${WORKDIR}/distributions.in > ${BASE_APT_CONF_DIR}/distributions
+ fi
+
+ # Create reprepro cache
+ if [ ! -d "${BASE_APT_DIR}/apt" ]; then
+ reprepro -b ${BASE_APT_DIR}/apt \
+ --dbdir ${BASE_APT_DIR}/db \
+ export ${DISTRO_SUITE}
+ fi
+
+ # Process all the packages fetched by multistrap
+ for deb in $(ls ${WORKDIR}/download/var/cache/apt/archives/*.deb);
+ do
+ pn=$(dpkg-deb -I $deb | grep 'Package:' | cut -d ' ' -f 3)
+ pv=$(dpkg-deb -I $deb | grep 'Version:' | cut -d ' ' -f 3)
+ line=$(cat ${WORKDIR}/multistrap.log | grep -E '^(Get:)' | grep " $pn ")
+ url=$(echo $line | cut -d ' ' -f 2)
+ component=$(basename $(echo $line | cut -d ' ' -f 3))
+
+ # Store download history
+ echo $pn $pv $component $url >> ${WORKDIR}/deb.list
+
+ reprepro -b ${BASE_APT_DIR}/apt \
+ --dbdir ${BASE_APT_DIR}/db \
+ -C $component \
+ includedeb ${DISTRO_SUITE} \
+ $deb
+ done
+
+ sudo rm -rf ${WORKDIR}/download
+}
+
+addtask get_base_apt after do_unpack before do_build
diff --git a/meta/recipes-devtools/base-apt/files/distributions.in b/meta/recipes-devtools/base-apt/files/distributions.in
new file mode 100644
index 0000000..44e9513
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main contrib non-free firmware
diff --git a/meta/recipes-devtools/base-apt/files/multistrap.conf.in b/meta/recipes-devtools/base-apt/files/multistrap.conf.in
new file mode 100644
index 0000000..27bf985
--- /dev/null
+++ b/meta/recipes-devtools/base-apt/files/multistrap.conf.in
@@ -0,0 +1,28 @@
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+
+[General]
+noauth=true
+unpack=false
+ignorenativearch=true
+bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
+aptsources=##DISTRO_MULTICONF_APTSOURCES##
+
+[base]
+source=##DISTRO_APT_SOURCE##
+suite=##DISTRO_SUITE##
+components=##DISTRO_COMPONENTS##
+packages=##BASE_PREINSTALL##
+omitdebsrc=true
+
+[updates]
+source=##DISTRO_APT_SOURCE##
+suite=##DISTRO_SUITE##-updates
+components=##DISTRO_COMPONENTS##
+omitdebsrc=true
+
+[security]
+source=##DISTRO_APT_SOURCE_SEC##
+suite=##DISTRO_SUITE##/updates
+components=##DISTRO_COMPONENTS##
+omitdebsrc=true
--
2.1.4

[Alexander Smirnov]

Add 'base-apt' recipe to build pipeline.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 5 +++++
meta/classes/dpkg-base.bbclass | 14 ++++++++++++++
meta/classes/image.bbclass | 13 ++++++++++++-
meta/recipes-devtools/base-apt/base-apt.bb | 5 ++++-
meta/recipes-devtools/buildchroot/buildchroot.bb | 12 ++++++++++++
5 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 660958f..2da803b 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -162,3 +162,8 @@ BB_NUMBER_THREADS = "4"
#
# Number of attempts to try to get reprepro lock for access to apt cache
REPREPRO_LOCK_ATTEMPTS = "16"
+
+#
+# List of images, wich dependencies will be pre-fetched and put in base-apt.
+# This will guarantee image building reproducibility.
+BASE_APT_IMAGES ?= "isar-image-base isar-image-debug"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ad58f0b..c61f9a6 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -8,6 +8,20 @@ do_build[depends] = "buildchroot:do_build"
# recipe name as identifier
PP = "/home/builder/${PN}"

+do_get_deps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
+S = "${WORKDIR}/${SRC_DIR}"
+
+do_get_deps() {
+ if [ -e ${S}/debian/control ]; then
+ mkdir -p ${BASE_APT_DIR}/deps
+ DEPS=$(perl -ne 'next if /^#/; $p=(s/^Build-Depends:\s*/ / or (/^ / and $p)); s/,|\n|\([^)]+\)|\[[^]]+\]//mg; print if $p' < ${S}/debian/control)
+ echo $DEPS > ${BASE_APT_DIR}/deps/${PN}
+ fi
+}
+
+addtask get_deps after do_unpack before do_build
+
BUILDROOT = "${BUILDCHROOT_DIR}/${PP}"
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index e2cb01b..457a525 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -66,6 +66,17 @@ do_cache_config() {

addtask cache_config before do_populate

+IMAGE_PREINSTALL ?= ""
+do_get_deps[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+
+do_get_deps() {
+ mkdir -p ${BASE_APT_DIR}/deps
+ echo ${IMAGE_PREINSTALL} > ${BASE_APT_DIR}/deps/${PN}
+}
+
+addtask get_deps before do_build after do_unpack
+do_get_deps[deptask] = "do_get_deps"
+
do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"

# Populate Isar apt repository by newly built packages
@@ -79,7 +90,7 @@ do_populate() {
fi
}

-addtask populate before do_build after do_unpack
+addtask populate before do_build after do_get_deps
do_populate[deptask] = "do_deploy_deb"

do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index b292d89..108d501 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -13,7 +13,10 @@ SRC_URI = "file://distributions.in \
file://multistrap.conf.in \
"

-BASE_PREINSTALL ?= ""
+DEPENDS = "${BASE_APT_IMAGES} buildchroot"
+do_get_base_apt[deptask] = "do_get_deps"
+
+BASE_PREINSTALL ?= "apt"

WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 59ad0e0..098f5c7 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -29,8 +29,20 @@ BUILDCHROOT_PREINSTALL ?= "gcc \

WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"

+do_get_deps[stamp-extra-info] = "${DISTRO}-${MACHINE}"
+
+do_get_deps() {
+ mkdir -p ${BASE_APT_DIR}/deps
+ echo ${BUILDCHROOT_PREINSTALL} > ${BASE_APT_DIR}/deps/${PN}
+}
+
+addtask get_deps before do_build after do_unpack
+do_get_deps[deptask] = "do_get_deps"
+
+
do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_build[dirs] = "${WORKDIR}/hooks_multistrap"
+do_build[depends] = "base-apt:do_get_base_apt"

do_build() {
E="${@ bb.utils.export_proxies(d)}"
--
2.1.4

[Alexander Smirnov]

Switch image generation from upstream repos to 'base-apt'.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/recipes-core/images/files/multistrap.conf.in | 18 +++---------------
meta-isar/recipes-core/images/isar-image-base.bb | 5 +----
2 files changed, 4 insertions(+), 19 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 432b6af..6ba535f 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -5,31 +5,19 @@
noauth=true
unpack=true
ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP## Isar
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+bootstrap=base-apt Isar
+aptsources=base-apt
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##

-[base]
+[base-apt]
source=##DISTRO_APT_SOURCE##
suite=##DISTRO_SUITE##
components=##DISTRO_COMPONENTS##
packages=##IMAGE_PREINSTALL##
omitdebsrc=true

-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
[Isar]
packages=##IMAGE_INSTALL##
source=##DEPLOY_DIR_APT##
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index e359ac3..ec14a98 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -41,10 +41,7 @@ do_rootfs() {

# Adjust multistrap config
sed -e 's|##IMAGE_PREINSTALL##|${IMAGE_PREINSTALL}|g' \
- -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
- -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
- -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
- -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
+ -e 's|##DISTRO_APT_SOURCE##|copy:///${BASE_APT_DIR}/apt|g' \

-e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \

-e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \

-e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
--
2.1.4

[Alexander Smirnov]

Switch buildchroot generation from upstream repos to 'base-apt'.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta/recipes-devtools/base-apt/base-apt.bb | 9 ++++++++-
meta/recipes-devtools/buildchroot/buildchroot.bb | 5 +----
.../recipes-devtools/buildchroot/files/configscript.sh | 1 -
.../buildchroot/files/multistrap.conf.in | 18 +++---------------
4 files changed, 12 insertions(+), 21 deletions(-)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 108d501..741ebca 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -25,8 +25,15 @@ do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}"
do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

do_get_base_apt() {
+ for package in `ls ${WORKDIR}/deps`; do
+ DEPS=$(cat ${WORKDIR}/deps/$package | xargs)
+ PACKAGES="$PACKAGES $DEPS"
+ done
+
+ PACKAGES="$PACKAGES $BASE_PREINSTALL"

+
# Adjust multistrap config

- sed -e 's|##BASE_PREINSTALL##|${BASE_PREINSTALL}|g' \
+ sed -e 's|##BASE_PREINSTALL##|'"$PACKAGES"'|g' \

-e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \

-e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \

-e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 098f5c7..4d773f9 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -57,10 +57,7 @@ do_build() {

# Adjust multistrap config
sed -e 's|##BUILDCHROOT_PREINSTALL##|${BUILDCHROOT_PREINSTALL}|g' \

- -e 's|##DISTRO_MULTICONF_BOOTSTRAP##|${DISTRO_MULTICONF_BOOTSTRAP}|g' \
- -e 's|##DISTRO_MULTICONF_APTSOURCES##|${DISTRO_MULTICONF_APTSOURCES}|g' \
- -e 's|##DISTRO_APT_SOURCE##|${DISTRO_APT_SOURCE}|g' \
- -e 's|##DISTRO_APT_SOURCE_SEC##|${DISTRO_APT_SOURCE_SEC}|g' \
+ -e 's|##DISTRO_APT_SOURCE##|copy:///${BASE_APT_DIR}/apt|g' \
-e 's|##DISTRO_SUITE##|${DISTRO_SUITE}|g' \
-e 's|##DISTRO_COMPONENTS##|${DISTRO_COMPONENTS}|g' \

-e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index 9813c9a..458c94b 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -44,5 +44,4 @@ mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev

#configuring packages
dpkg --configure -a
-apt-get update
umount /dev
diff --git a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
index a0b28e3..622209c 100644
--- a/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
+++ b/meta/recipes-devtools/buildchroot/files/multistrap.conf.in
@@ -5,27 +5,15 @@
noauth=true
unpack=true
ignorenativearch=true
-bootstrap=##DISTRO_MULTICONF_BOOTSTRAP##
-aptsources=##DISTRO_MULTICONF_APTSOURCES##
+bootstrap=base-apt

+aptsources=base-apt
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
hookdir=##DIR_HOOKS##

-[base]
+[base-apt]
source=##DISTRO_APT_SOURCE##
suite=##DISTRO_SUITE##
components=##DISTRO_COMPONENTS##

packages=##BUILDCHROOT_PREINSTALL##
omitdebsrc=true
-

-[updates]
-source=##DISTRO_APT_SOURCE##
-suite=##DISTRO_SUITE##-updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true
-
-[security]
-source=##DISTRO_APT_SOURCE_SEC##
-suite=##DISTRO_SUITE##/updates
-components=##DISTRO_COMPONENTS##
-omitdebsrc=true

--
2.1.4

[Alexander Smirnov]

'base-apt' should be avaialble in buildchroot to have possibility
to install build dependencies for packages being built. Mounting is
performed in dedicated task buildchroot:do_prepare. To implement
umounting, the bitbake events are used. This patch registers handler
for build complete event which performs cleanup.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta-isar/conf/local.conf.sample | 3 +++
meta/classes/dpkg-base.bbclass | 2 +-
meta/classes/isar-events.bbclass | 21 +++++++++++++++++++++
meta/recipes-devtools/buildchroot/buildchroot.bb | 11 +++++++++++
4 files changed, 36 insertions(+), 1 deletion(-)
create mode 100644 meta/classes/isar-events.bbclass

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 2da803b..acced81 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -167,3 +167,6 @@ REPREPRO_LOCK_ATTEMPTS = "16"

# List of images, wich dependencies will be pre-fetched and put in base-apt.

# This will guarantee image building reproducibility.

BASE_APT_IMAGES ?= "isar-image-base isar-image-debug"
+
+# Add event handlers for bitbake
+INHERIT += "isar-events"
diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index c61f9a6..bc8fab8 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
# Copyright (C) 2017 Siemens AG

# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_prepare"

# Each package should have its own unique build folder, so use

# recipe name as identifier

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
new file mode 100644
index 0000000..6ccda99
--- /dev/null
+++ b/meta/classes/isar-events.bbclass
@@ -0,0 +1,21 @@
+# Isar event handlers.

+#
+# This software is a part of ISAR.
+# Copyright (C) 2015-2017 ilbers GmbH
+

+addhandler isar_handler
+
+python isar_handler () {
+ import subprocess
+
+ devnull = open(os.devnull, 'w')
+
+ if isinstance(e, bb.event.BuildCompleted):
+ bchroot = d.getVar('BUILDCHROOT_DIR', True)
+
+ # Clean up buildchroot
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/proc || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+ subprocess.call('/usr/bin/sudo /bin/umount ' + bchroot + '/apt || /bin/true', stdout=devnull, stderr=devnull, shell=True)
+
+ devnull.close()
+}
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 4d773f9..1fad2f8 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -80,7 +80,18 @@ do_build() {
# Install package builder script
sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}

+ # Create share point for apt
+ sudo install -d ${BUILDCHROOT_DIR}/apt
+
# Configure root filesystem
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
_do_build_cleanup
}
+
+do_prepare[nostamp] = "1"
+
+do_prepare() {
+ sudo mount --bind ${BASE_APT_DIR}/apt ${BUILDCHROOT_DIR}/apt
+}
+
+addtask prepare after do_build
--
2.1.4

[Alexander Smirnov]

This patch adds possibility to reuse 'base-apt' from any folder. So user
could create 'base-apt' once and then use it for all the next builds.

Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
---

meta/recipes-devtools/base-apt/base-apt.bb | 8 ++++++++
1 file changed, 8 insertions(+)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 741ebca..7cf4ba0 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -20,9 +20,17 @@ BASE_PREINSTALL ?= "apt"

WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"

+python __anonymous() {
+ aptdir = d.getVar('BASE_APT_DIR', True) + '/apt/pool'
+
+ if not os.path.exists(aptdir):
+ d.delVarFlag('do_get_base_apt', 'noexec')
+}
+
BASE_APT_CONF_DIR = "${BASE_APT_DIR}/apt/conf"

do_get_base_apt[dirs] = "${BASE_APT_CONF_DIR}"
do_get_base_apt[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

+do_get_base_apt[noexec] = "true"

do_get_base_apt() {

for package in `ls ${WORKDIR}/deps`; do

--
2.1.4

[Jan Kiszka]

Hi Alex,

On 2018-01-02 15:57, Alexander Smirnov wrote:
> Hello all,
>
> this series proposes the way how build reproducibility could be implemented
> in Isar. General idea is to get the list of all the necessary packages for
> build, fetch them and create local repo, that will be used for further
> builds/
>
> Briefly speeking, it works like the following:
>
> 1. User sets the list of images that should be 'reproducible' in
> BASE_APT_IMAGES variable in local.conf file.

What speaks against making every image reproducible? I strongly assume
this is the common case. There should be a way to disable that, and that
could be local.conf controlled.

Is there a technical reason the core needs to know the name of the
reproducible image? Or can be control this on a per-build basis,
including all images of that build?

And will all images of the same bitbake run (thinking of multiconfig...)
share also the same base-apt repo, or is that per image?

>
> 2. Based on the list of images above, Isar will derive all the run-time and
> build dependencies for these images.
>
> 3. Using multistrap, Isar will fetch the list of packages and create base-apt
> local repository.
>
> 4. Now buildchroot and image root filesystems are generated using base-apt.
>
> Some notes:
>
> 1. base-apt repository is mounted to buildchroot, so Isar packages are
> able to install necessary deps via apt-get.

This does not your provide the packages a way to feed their own build
results into the same repo, does it? I'm referring to the issue that
Christian described
(https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
and which I ran into as well meanwhile.

>
> 2. bitbake events are used to clean up buildchroot. I haven't found another
> way how base-apt could be unmounted. So it's mounted once before any
> package starts building and unmounted by bitbake event:
> bb.event.BuildCompleted.

What is the implication of that? At least to me (without detailed
understanding of bitbake), this remains unclear.

>
> This series works good with latest next. Any comments are welcome.
>
> Happy New Year 2018! :-)

Same to you.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Jan Kiszka]

On 2018-01-02 15:57, Alexander Smirnov wrote:

> This patch introduces mechanism how to fetch deb packages from dedicated
> upstream apt repos and store them localy. Local repository is called
> 'base-apt' and it will be used to generate buildchroot and image root
> filesystems. Using 'base-apt' will guarantee build reproducibility between
> builds.
>
> Signed-off-by: Alexander Smirnov <asmi...@ilbers.de>
> ---
> meta/conf/isar-bitbake.conf | 2 +
> meta/recipes-devtools/base-apt/base-apt.bb | 79 ++++++++++++++++++++++
> .../base-apt/files/distributions.in | 3 +
> .../base-apt/files/multistrap.conf.in | 28 ++++++++
> 4 files changed, 112 insertions(+)
> create mode 100644 meta/recipes-devtools/base-apt/base-apt.bb
> create mode 100644 meta/recipes-devtools/base-apt/files/distributions.in
> create mode 100644 meta/recipes-devtools/base-apt/files/multistrap.conf.in
>
> diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
> index 5a26743..df54399 100644
> --- a/meta/conf/isar-bitbake.conf
> +++ b/meta/conf/isar-bitbake.conf
> @@ -23,6 +23,8 @@ DEPLOY_DIR_DEB = "${TMPDIR}/deploy/deb/${MACHINE}"
> SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
> BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
>
> +BASE_APT_DIR ?= "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/base-apt"

That should also go into TOPDIR, analogously to my patch for DL_DIR.

Hmm, is there really no way to share the common bits with
meta/recipes-devtools/buildchroot/files/multistrap.conf.in? I suspect
the want to be kept in sync, except for the General section.

[Jan Kiszka]

On 2018-01-02 15:57, Alexander Smirnov wrote:

> This patch introduces mechanism how to fetch deb packages from dedicated
> upstream apt repos and store them localy. Local repository is called
> 'base-apt' and it will be used to generate buildchroot and image root
> filesystems. Using 'base-apt' will guarantee build reproducibility between
> builds.

BTW, the name "base-apt" is not intuitive for me. Why "-apt"? Why not
"local-base" or "base-mirror"?

[Alexander Smirnov]

Hi Jan,

On 01/02/2018 07:09 PM, Jan Kiszka wrote:
> Hi Alex,
>
> On 2018-01-02 15:57, Alexander Smirnov wrote:
>> Hello all,
>>
>> this series proposes the way how build reproducibility could be implemented
>> in Isar. General idea is to get the list of all the necessary packages for
>> build, fetch them and create local repo, that will be used for further
>> builds/
>>
>> Briefly speeking, it works like the following:
>>
>> 1. User sets the list of images that should be 'reproducible' in
>> BASE_APT_IMAGES variable in local.conf file.
>
> What speaks against making every image reproducible? I strongly assume
> this is the common case. There should be a way to disable that, and that
> could be local.conf controlled.

I assume, that there could be lots of overlays used for building custom
product. So I'm not sure if it makes sense to include all the image
recipes from all the overlays.

For example, in Yocto, there is image sato, which provides graphical UI
and adds lots of tasks to build pipeline. In my experience I don't
remember anybody who used it in commercial products, so for they "sato"
reproducibility will be excessive.

>
> Is there a technical reason the core needs to know the name of the
> reproducible image? Or can be control this on a per-build basis,
> including all images of that build?

Yes, unfortunately it's a technical reason, at least I haven't found any
other way for now. Bitbake uses "waterfall one direction" dependencies
processing. Roughly speaking, if you imagine all the deps as tree with
head on the top, bitbake could derive only deps in the bottom for each
node in the tree.

isar-image-base
/ \
hello example-raw
\ /
buildchroot
|
base-apt


So, if your recipe, for example 'hello.bb' is building, you can't get
within it the information who exactly added him to build pipeline, what
is the final target/image bitbake is building etc...

That's why I need to know the top nodes of the dependency tree to get
the list of all the packages being used.

>
> And will all images of the same bitbake run (thinking of multiconfig...)
> share also the same base-apt repo, or is that per image?
>
>>
>> 2. Based on the list of images above, Isar will derive all the run-time and
>> build dependencies for these images.
>>
>> 3. Using multistrap, Isar will fetch the list of packages and create base-apt
>> local repository.
>>
>> 4. Now buildchroot and image root filesystems are generated using base-apt.
>>
>> Some notes:
>>
>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>> able to install necessary deps via apt-get.
>
> This does not your provide the packages a way to feed their own build
> results into the same repo, does it? I'm referring to the issue that
> Christian described
> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
> and which I ran into as well meanwhile.

The idea of base-apt - to be a partial mirror of upstream Debian. In my
understanding, for all the build results Isar apt should be used.

>>
>> 2. bitbake events are used to clean up buildchroot. I haven't found another
>> way how base-apt could be unmounted. So it's mounted once before any
>> package starts building and unmounted by bitbake event:
>> bb.event.BuildCompleted.
>
> What is the implication of that? At least to me (without detailed
> understanding of bitbake), this remains unclear.

No implications. Bitbake has various events that are generated at
certain points of operations:

https://www.yoctoproject.org/docs/1.6/bitbake-user-manual/bitbake-user-manual.html#events

These events for example are used in Yocto to generate build statistics,
in toaster UI.

The key problem here is to implement the following model:

- buildchroot:do_prepare (mount /proc, /dev, base-apt)
- build custom packages
- buildchroot:do_cleanup (unmount everything)

I could add dependency from "buildchroot:do_prepare" to any package
being build, it's not a problem.

But I can't add dependency from "buildchroot:do_cleanup". This task
should be executed when all the packages are built, but the list of
packages being build is generated dynamically, while bitbake supports
only static deps.

One way could be add "do_cleanup" to image recipe, because it has build
dependency from all the recipes, but this will work only if you build
image. The command:

$ bitbake example-raw

will not call "do_cleanup" in this case.

Alex

[Alexander Smirnov]

Hi,

No problem, have included this to fix-list for next series version.

multistrap config for buildchroot and image will not use these sections
anymore, they are dropped later in this series.

Alex

[Jan Kiszka]

What are the costs again? Please specify, at latest when you provide
some (user) documentation for this new feature.

>
>>
>> Is there a technical reason the core needs to know the name of the
>> reproducible image? Or can be control this on a per-build basis,
>> including all images of that build?
>
> Yes, unfortunately it's a technical reason, at least I haven't found any
> other way for now. Bitbake uses "waterfall one direction" dependencies
> processing. Roughly speaking, if you imagine all the deps as tree with
> head on the top, bitbake could derive only deps in the bottom for each
> node in the tree.
>
>              isar-image-base
>                /       \
>             hello    example-raw
>               \        /
>              buildchroot
>                  |
>               base-apt
>
>
> So, if your recipe, for example 'hello.bb' is building, you can't get
> within it the information who exactly added him to build pipeline, what
> is the final target/image bitbake is building etc...
>
> That's why I need to know the top nodes of the dependency tree to get
> the list of all the packages being used.

The top nodes are what is provided to bitbake as targets, no? Can't we
access that information?

>
>>
>> And will all images of the same bitbake run (thinking of multiconfig...)
>> share also the same base-apt repo, or is that per image?
>>
>>>
>>> 2. Based on the list of images above, Isar will derive all the
>>> run-time and
>>>     build dependencies for these images.
>>>
>>> 3. Using multistrap, Isar will fetch the list of packages and create
>>> base-apt
>>>     local repository.
>>>
>>> 4. Now buildchroot and image root filesystems are generated using
>>> base-apt.
>>>
>>> Some notes:
>>>
>>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>>>     able to install necessary deps via apt-get.
>>
>> This does not your provide the packages a way to feed their own build
>> results into the same repo, does it? I'm referring to the issue that
>> Christian described
>> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
>>
>> and which I ran into as well meanwhile.
>
> The idea of base-apt - to be a partial mirror of upstream Debian. In my
> understanding, for all the build results Isar apt should be used.

So, recipes would then deploy their debs into base-apt, making them
immediately available to dependent recipes?

Is there no "bitbake is exiting" event that you could hook up to? Then
the cleanups would happen at the very end of everything.

[Jan Kiszka]

On 2018-01-02 18:07, Jan Kiszka wrote:
>>>> Some notes:
>>>>
>>>> 1. base-apt repository is mounted to buildchroot, so Isar packages are
>>>>     able to install necessary deps via apt-get.
>>>
>>> This does not your provide the packages a way to feed their own build
>>> results into the same repo, does it? I'm referring to the issue that
>>> Christian described
>>> (https://groups.google.com/forum/#!msg/isar-users/efer3RF989o/r-zfUNNDAgAJ)
>>>
>>> and which I ran into as well meanwhile.
>>
>> The idea of base-apt - to be a partial mirror of upstream Debian. In my
>> understanding, for all the build results Isar apt should be used.
>
> So, recipes would then deploy their debs into base-apt, making them
> immediately available to dependent recipes?

BTW, if do that, we probably want to have a separate repo for the
self-built packages, just to simplify the cleanup of artifacts. This
repo here is actually something we could consider the download folder for.

[Henning Schild]

Am Tue, 2 Jan 2018 20:02:26 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

I think Jan was not just talking about the config. This would be the
third copy of a multistrap-task in Isar. Consolidating the present two
is on my whishlist for some time, i would not like to see yet another
similar task being introduced.

The task get_base_apt() should probably be split into at least two
seperate tasks. multistrap and reprepro + fill
But that will happen naturally if all multistrap-users use a multistrap
base-class.

Henning

> Alex
>

[Henning Schild]

Am Tue, 2 Jan 2018 17:57:40 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

This code comes from buildchroot/files/build.sh and should be in the
codebase only once.

Furthermore this version of do_get_deps should go into dpkg.bbclass and
we need another implementation for dpdk-raw.bbclass that takes care of
DEBIAN_DEPENDS

Henning

[Henning Schild]

Am Tue, 2 Jan 2018 17:57:38 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

> Hello all,
>
> this series proposes the way how build reproducibility could be
> implemented in Isar. General idea is to get the list of all the
> necessary packages for build, fetch them and create local repo, that
> will be used for further builds/
>
> Briefly speeking, it works like the following:
>
> 1. User sets the list of images that should be 'reproducible' in
> BASE_APT_IMAGES variable in local.conf file.

I am with Jan here, i would prefer an all or nothing approach.

> 2. Based on the list of images above, Isar will derive all the
> run-time and build dependencies for these images.
>
> 3. Using multistrap, Isar will fetch the list of packages and create
> base-apt local repository.
>
> 4. Now buildchroot and image root filesystems are generated using
> base-apt.

Patch5 makes base-apt and Isar the only repos for a rootfs/buildchroot.
How do images, not using BASE_APT, get packages that are not cached?

> Some notes:
>
> 1. base-apt repository is mounted to buildchroot, so Isar packages are
> able to install necessary deps via apt-get.
>
> 2. bitbake events are used to clean up buildchroot. I haven't found
> another way how base-apt could be unmounted. So it's mounted once
> before any package starts building and unmounted by bitbake event:
> bb.event.BuildCompleted.
>
> This series works good with latest next. Any comments are welcome.
>
> Happy New Year 2018! :-)

Same from me!

Henning

[Jan Kiszka]

On 2018-01-03 14:49, [ext] Henning Schild wrote:
> Am Tue, 2 Jan 2018 17:57:38 +0300
> schrieb Alexander Smirnov <asmi...@ilbers.de>:
>
>> Hello all,
>>
>> this series proposes the way how build reproducibility could be
>> implemented in Isar. General idea is to get the list of all the
>> necessary packages for build, fetch them and create local repo, that
>> will be used for further builds/
>>
>> Briefly speeking, it works like the following:
>>
>> 1. User sets the list of images that should be 'reproducible' in
>> BASE_APT_IMAGES variable in local.conf file.
>
> I am with Jan here, i would prefer an all or nothing approach.
>
>> 2. Based on the list of images above, Isar will derive all the
>> run-time and build dependencies for these images.
>>
>> 3. Using multistrap, Isar will fetch the list of packages and create
>> base-apt local repository.
>>
>> 4. Now buildchroot and image root filesystems are generated using
>> base-apt.
>
> Patch5 makes base-apt and Isar the only repos for a rootfs/buildchroot.
> How do images, not using BASE_APT, get packages that are not cached?
>

Another reason to split the caching and local redistribution repo that
will be filled by local builds from this mirroring and pinning repo. For
archiving purposes, there than later be some tool/task to export both
into a single repo, but we need a separation during builds.

[Henning Schild]

Am Wed, 3 Jan 2018 14:54:10 +0100
schrieb Jan Kiszka <jan.k...@siemens.com>:

Not sure what you are talking about, but there will be two repos.
"base-apt" which will contain all .debs from repos we fed into the
first multistrap - presumably official Debian mirrors. But all of the
partitial mirrors merged into one repo. And the second one "Isar" which
will contain what came out of dpdk.bbclass or dpdk-raw.bblass recipes.

Henning

> Jan
>

[Jan Kiszka]

Ah, good, it's already split. But then, while base-apt might be opt-in
only, the isar repo cannot be optional, i.e. dependent on that BASE_APT
control (or however it may be called in the end).

[Henning Schild]

Am Tue, 2 Jan 2018 17:57:40 +0300
schrieb Alexander Smirnov <asmi...@ilbers.de>:

This perl expression has another issue, it does not handle the
|-operator, which is also valid in Build-Depends.
https://www.debian.org/doc/debian-policy/ (7.1)

In fact this code fails for a lot of valid rules-files. It can also not
handle the architecure selection operator [] and it ignores
Build-Depends-Indep.

To sum up, this is very broken and should be replaced with something
official that hopefully works.

"mk-build-deps" looks promising, it generates a meta-package out of a
given control-file. I will look into replacing the perl with
mk-build-deps.

Henning

[Alexander Smirnov]

Hi all,

On 01/03/2018 04:49 PM, Henning Schild wrote:
> Am Tue, 2 Jan 2018 17:57:38 +0300
> schrieb Alexander Smirnov <asmi...@ilbers.de>:
>
>> Hello all,
>>
>> this series proposes the way how build reproducibility could be
>> implemented in Isar. General idea is to get the list of all the
>> necessary packages for build, fetch them and create local repo, that
>> will be used for further builds/
>>
>> Briefly speeking, it works like the following:
>>
>> 1. User sets the list of images that should be 'reproducible' in
>> BASE_APT_IMAGES variable in local.conf file.
>
> I am with Jan here, i would prefer an all or nothing approach.
>

Still thinking if it's possible to implement this with bitbake. The key
problem is that Isar doesn't have static list of Debian dependencies,
i.e. it needs to fetch source code and parse 'debian/conrtol' to get
list of required packages to install.

So some kind of the following chain should be implemented:

1. Identify image recipes.
2. Get list of packages in IMAGE_INSTALL for all images.
3. Make do_fetch/do_unpack and parse 'debian/control' for packages in
the list above.

In this case base-apt recipe should depend from *all* the available images.

Anonymous tasks can't help to derive dependencies, because package's
deps could be derived after 'do_unpack' only. In theory, we could define
some bitbake variable like: DEBIAN_DEPENDENCY that contains the same
data as in 'debian/control'. Moreover it could make sense to generate
'debian/control' on the fly, like Henning does it for 'example-raw'.

Having variable DEBIAN_DEPENDENCY makes possible to get list of packages
that are possibly used in whole current Isar tree by using anonymous
task in bbclass.

Opinions? :-)

>> 2. Based on the list of images above, Isar will derive all the
>> run-time and build dependencies for these images.
>>
>> 3. Using multistrap, Isar will fetch the list of packages and create
>> base-apt local repository.
>>
>> 4. Now buildchroot and image root filesystems are generated using
>> base-apt.
>
> Patch5 makes base-apt and Isar the only repos for a rootfs/buildchroot.
> How do images, not using BASE_APT, get packages that are not cached?
>

In my opinion, in case of base-apt, image should not use upstream
anymore, otherwise build reproducibility can't be guaranteed. apt will
try to fetch the freshest package.

So if new package appears in the build, the base-apt should be
updated/regenerated.

Alex