[PATCH 01/26] isar-bootstrap: Consider DISTRO_BOOTSTRAP_KEYS for gnupg installation

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

We also need gnupg when using a custom bootstrap key.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 1eb2ac6..bcd644b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -169,8 +169,9 @@ def get_distro_needs_https_support(d, is_host=False):
return ""

def get_distro_needs_gpg_support(d):
- apt_keys = d.getVar("THIRD_PARTY_APT_KEYS")
- if apt_keys and apt_keys != "":
+ apt_keys = d.getVar("DISTRO_BOOTSTRAP_KEYS") or ""
+ apt_keys += " " + (d.getVar("THIRD_PARTY_APT_KEYS") or "")
+ if apt_keys != " ":
return "gnupg"
return ""

--
2.16.4

[Jan Kiszka]

Highlights:
- fixed custom DISTRO_BOOTSTRAP_KEYS
- fixed shared lock across dpkg_runbuild (nasty stuff...)
- reworked STAMPing
- fixed follow-up errors on build terminations (error or ctrl-c)
- fixed two false sharings
- detection of false task sharing
- stop mangling PF
- no more tasks beyond do_build
- fixed isar-bootstrap rebuild
- deploy logs on gitlab-ci errors

The queue can also be found at

https://github.com/siemens/isar jan/queue

It passed ci_build.sh -q -f and builds jailhouse-images fine. But these
are many significant changes, so please review carefully.

Jan


CC: Cedric Hombourger <Cedric_H...@mentor.com>

Jan Kiszka (26):
isar-bootstrap: Consider DISTRO_BOOTSTRAP_KEYS for gnupg installation
isar-bootstrap: Fix keyring generation
isar-bootstrap: Install DISTRO_BOOTSTRAP_KEYS
Rework STAMP layout and cleaning
dpkg: Fix shared-locking of build task
dpkg-base: Free do_build task
base: Clean up build task
isar-events: Do not start cleanup early
meta-isar: Move bananapi to buster, remove bogus example
bitbake.conf: Reorder PV definition
example-module: Fix build with mipsel target
buildchroot: Ignore ISAR_CROSS_COMPILE for i386 on stretch
buildchroot: Fix downgrades when installing dependencies
buildchroot: Properly deploy the build result
sdk: Use clean-package-cache rather than open-coded cleanup
sdk: Deploy earlier
Remove unneeded PF tweakings
image: Stop changing PF, tune WORKDIR and STAMP directly
image: Make WORKDIR and STAMPs IMAGE_TYPE-specific
Adjust STAMP variables to use PN, rather than PF
Align WORKDIR structure with OE
isar-bootstrap: Clean up and fix rebuild
gitlab-ci: Keep logs as artifacts on failure
Detect false sharing of recipes
dpkg-base: Move do_deploy_deb before do_build
base-apt: Move do_cache_config before do_build

.gitlab-ci.yml | 6 +++++
doc/user_manual.md | 2 +-
.../conf/distro/debian-stretch-backports.list | 2 --
meta-isar/conf/local.conf.sample | 2 +-
meta-isar/conf/machine/bananapi.conf | 3 ---
...{bananapi-stretch.conf => bananapi-buster.conf} | 2 +-
.../conf/multiconfig/preferences.bananapi.conf | 7 -----
.../example-module/example-module.bb | 1 +
meta/classes/base.bbclass | 15 +++++------
meta/classes/buildchroot.bbclass | 6 +++--
meta/classes/dpkg-base.bbclass | 25 +++++++++--------
meta/classes/dpkg-raw.bbclass | 1 -
meta/classes/dpkg.bbclass | 8 +++---
meta/classes/image-sdk-extension.bbclass | 23 ++++++----------
meta/classes/image.bbclass | 5 +++-
meta/classes/isar-events.bbclass | 28 ++++++++++++++++---
meta/classes/patch.bbclass | 2 --
meta/classes/template.bbclass | 1 -
meta/conf/bitbake.conf | 15 +++++++----
.../isar-bootstrap/isar-bootstrap-host.bb | 31 ++++++++++------------
.../isar-bootstrap/isar-bootstrap-target.bb | 28 +++++++++----------
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 15 +++++------
meta/recipes-devtools/base-apt/base-apt.bb | 2 +-
.../buildchroot/buildchroot-host.bb | 3 ++-
.../buildchroot/buildchroot-target.bb | 3 ++-
meta/recipes-devtools/buildchroot/buildchroot.inc | 13 +++++++++
meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 14 ++++++++--
meta/recipes-kernel/linux/linux-custom.inc | 3 +--
29 files changed, 146 insertions(+), 122 deletions(-)
delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
were set to invalid WORKDIR, rather than the value relevant when running
do_generate_keyrings in recipe build context.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index bcd644b..d2210d7 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -190,8 +190,8 @@ def get_distro_components_argument(d, is_host):
else:
return ""

-APT_KEYS_DIR := "${WORKDIR}/aptkeys"
-DISTRO_BOOTSTRAP_KEYRING := "${WORKDIR}/distro-keyring.gpg"
+APT_KEYS_DIR = "${WORKDIR}/aptkeys"
+DISTRO_BOOTSTRAP_KEYRING = "${WORKDIR}/distro-keyring.gpg"

do_generate_keyrings[cleandirs] = "${APT_KEYS_DIR}"
do_generate_keyrings[dirs] = "${DL_DIR}"
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Rename do_build to do_dpkg_build so that the build task remains virtual,
just like it's common practice in OE.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 10 ++++++----
meta/classes/dpkg.bbclass | 2 +-
meta/recipes-kernel/linux/linux-custom.inc | 2 +-
3 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 8146328..50df06d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -18,7 +18,7 @@ do_adjust_git() {
addtask adjust_git after do_unpack before do_patch

inherit patch
-addtask patch after do_adjust_git before do_build
+addtask patch after do_adjust_git before do_dpkg_build

SRC_APT ?= ""

@@ -63,7 +63,7 @@ do_prepare_build() {
true
}

-addtask prepare_build after do_patch do_transform_template before do_build
+addtask prepare_build after do_patch do_transform_template before do_dpkg_build
# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"
@@ -87,7 +87,7 @@ dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"
}

-python do_build() {
+python do_dpkg_build() {
lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
shared=True)
bb.build.exec_func("dpkg_do_mounts", d)
@@ -96,6 +96,8 @@ python do_build() {
bb.utils.unlockfile(lock)
}

+addtask dpkg_build before do_build
+
CLEANFUNCS += "repo_clean"

repo_clean() {
@@ -129,7 +131,7 @@ do_deploy_deb() {
${S}/../*.deb
}

-addtask deploy_deb after do_build
+addtask deploy_deb after do_dpkg_build
do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[depends] = "isar-apt:do_cache_config"
do_deploy_deb[dirs] = "${S}"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index df47ced..f5396cb 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -11,7 +11,7 @@ do_install_builddeps() {
dpkg_undo_mounts
}

-addtask install_builddeps after do_prepare_build before do_build
+addtask install_builddeps after do_prepare_build before do_dpkg_build
# apt and reprepro may not run in parallel, acquire the Isar lock
do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index 12d8d9c..c045b89 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -41,7 +41,7 @@ do_install_builddeps() {
dpkg_undo_mounts
}

-addtask install_builddeps after do_prepare_build before do_build
+addtask install_builddeps after do_prepare_build before do_dpkg_build
# apt and reprepro may not run in parallel, acquire the Isar lock
do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

We use the DISTRO_BOOTSTRAP_KEYS already for bootstrapping - but we fail
to install them on the generated rootfs for later use in buildchroots or
target rootfs.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
1 file changed, 1 insertion(+)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index d2210d7..aea89a9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -206,6 +206,7 @@ do_generate_keyrings() {
if [ -n "${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True) or ""}" ]; then
for keyfile in ${@d.getVar("DISTRO_BOOTSTRAP_KEYFILES", True)}; do
sudo apt-key --keyring "${DISTRO_BOOTSTRAP_KEYRING}" add $keyfile
+ cp "$keyfile" "${APT_KEYS_DIR}"/"$(basename "$keyfile")"
done
fi
}
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Bitbake file locks are not just exclusive flock calls. For whatever
reason, bitbake prefers to delete the lock file before unlocking them.
That obviously creates a nasty race with other locker requests: Those
already waiting will get access to the removed lock, those coming later
will create a new one. To resolve that race, bitbakes lockfile() tests
if the inode of the acquired lock file is identical to the one that is
currently visible via the lock name. If not, it retries the lock. What a
pointless mess...

Our open-coded "flock -s" does not do that, thus will often acquire only
a stale lock. Fix that by using bitbake's services for locking. That
requires to move the lock acquisition at the top level of the task, ie.
do_build. Thus, also dpkg-raw and kernel builds are now under the shared
(read) lock. The better fix for this issue would actually be private
buildchroots for each build, but that will come later.

CC: Cedric Hombourger <Cedric_H...@mentor.com>
Fixes: 568072765b08 ("dpkg: acquire a read (shared) locks while building packages")

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 11 +++++++----
meta/classes/dpkg.bbclass | 5 ++---
2 files changed, 9 insertions(+), 7 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2d26418..8146328 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -87,10 +87,13 @@ dpkg_runbuild() {

die "This should never be called, overwrite it in your derived class"
}

-do_build() {
- dpkg_do_mounts
- dpkg_runbuild
- dpkg_undo_mounts
+python do_build() {
+ lock = bb.utils.lockfile(d.getVar("REPO_ISAR_DIR") + "/isar.lock",
+ shared=True)
+ bb.build.exec_func("dpkg_do_mounts", d)
+ bb.build.exec_func("dpkg_runbuild", d)
+ bb.build.exec_func("dpkg_undo_mounts", d)
+ bb.utils.unlockfile(lock)
}

CLEANFUNCS += "repo_clean"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index dd123c3..df47ced 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
# Build package from sources using build script
dpkg_runbuild() {
E="${@ bb.utils.export_proxies(d)}"
- flock -s "${REPO_ISAR_DIR}/isar.lock" \
- sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
- /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
+ sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
+ /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
}
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Follow OE more closely by aligning the stamps folder structure to
workdir. This may resolve false sharing of stamps between recipes of
common PF but different DISTRO/DISTRO_ARCH without the otherwise
required stamp-extra-info. That extra-info can now be removed.

As long as recipes mess with PF, we can't use OE's pattern as-is which
is PN-based.

Furthermore add proper STAMPCLEAN. This ensures that stamp files are
cleaned when a task is rebuilt. Not providing this pattern likely caused
a lot of failures when doing partial rebuilds.

As we are at it, switch do_clean to STAMPCLEAN as well.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/base.bbclass | 5 ++---
meta/classes/dpkg-base.bbclass | 6 ------
meta/classes/dpkg-raw.bbclass | 1 -
meta/classes/dpkg.bbclass | 1 -

meta/classes/patch.bbclass | 2 --
meta/classes/template.bbclass | 1 -

meta/conf/bitbake.conf | 4 +++-
meta/recipes-kernel/linux/linux-custom.inc | 1 -
8 files changed, 5 insertions(+), 16 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index ca80bf8..1d2852a 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -163,7 +163,6 @@ python do_fetch() {
addtask fetch before do_build

do_unpack[dirs] = "${WORKDIR}"
-do_unpack[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

# Unpack package and put it into working directory
python do_unpack() {
@@ -206,8 +205,8 @@ python do_clean() {
workdir = d.expand("${WORKDIR}")
subprocess.check_call(["sudo", "rm", "-rf", workdir])

- stamppath = bb.data.expand(d.getVar('STAMP', False), d)
- stampdirs = glob.glob(stamppath + ".*")
+ stampclean = bb.data.expand(d.getVar('STAMPCLEAN', False), d)
+ stampdirs = glob.glob(stampclean)
subprocess.check_call(["sudo", "rm", "-rf"] + stampdirs)
}

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 39c8acb..2d26418 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -16,7 +16,6 @@ do_adjust_git() {

}

addtask adjust_git after do_unpack before do_patch

-do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

inherit patch

addtask patch after do_adjust_git before do_build

@@ -42,7 +41,6 @@ do_apt_fetch() {

addtask apt_fetch after do_unpack before do_patch
do_apt_fetch[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
-do_apt_fetch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

def get_package_srcdir(d):
s = os.path.abspath(d.getVar("S", True))
@@ -60,15 +58,12 @@ def get_package_srcdir(d):
PP = "/home/builder/${PN}"
PPS ?= "${@get_package_srcdir(d)}"

-do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
# Empty do_prepare_build() implementation, to be overwritten if needed
do_prepare_build() {
true

}

addtask prepare_build after do_patch do_transform_template before do_build

-do_prepare_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

# If Isar recipes depend on each other, they typically need the package
# deployed to isar-apt
do_prepare_build[deptask] = "do_deploy_deb"

@@ -132,7 +127,6 @@ do_deploy_deb() {
}

addtask deploy_deb after do_build
-do_deploy_deb[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[depends] = "isar-apt:do_cache_config"
do_deploy_deb[dirs] = "${S}"

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index ea03ea4..f8dfb2d 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -16,7 +16,6 @@ do_install() {
}

do_install[cleandirs] = "${D}"
-do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
addtask install after do_unpack before do_prepare_build

do_prepare_build[cleandirs] += "${D}/debian"
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 997f2f3..dd123c3 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -14,7 +14,6 @@ do_install_builddeps() {

addtask install_builddeps after do_prepare_build before do_build

# apt and reprepro may not run in parallel, acquire the Isar lock

do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

-do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

# Build package from sources using build script
dpkg_runbuild() {

diff --git a/meta/classes/patch.bbclass b/meta/classes/patch.bbclass
index 2662702..c5ba463 100644
--- a/meta/classes/patch.bbclass
+++ b/meta/classes/patch.bbclass
@@ -78,5 +78,3 @@ python do_patch() {
except bb.fetch2.BBFetchException as e:
raise bb.build.FuncFailed(e)
}
-
-do_patch[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
diff --git a/meta/classes/template.bbclass b/meta/classes/template.bbclass
index f3b5f80..fb9d118 100644
--- a/meta/classes/template.bbclass
+++ b/meta/classes/template.bbclass
@@ -7,7 +7,6 @@ TEMPLATE_FILES ?= ""
TEMPLATE_VARS ?= "PN PV DESCRIPTION HOMEPAGE MAINTAINER DISTRO_ARCH"

do_transform_template[vardeps] = "TEMPLATE_FILES ${TEMPLATE_VARS}"
-do_transform_template[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
python do_transform_template() {
import subprocess, contextlib

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 3d18223..c41f294 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -35,7 +35,9 @@ PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0
S = "${WORKDIR}/${P}"
AUTOREV = "${@bb.fetch2.get_autorev(d)}"
SRC_URI = "file://${FILE}"
-STAMP = "${TMPDIR}/stamps/${PF}"
+STAMPS_DIR ?= "${TMPDIR}/stamps"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
SVNDIR = "${DL_DIR}/svn"
T = "${WORKDIR}/temp"
TMPDIR = "${TOPDIR}/tmp"
diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
index ee5f20c..12d8d9c 100644
--- a/meta/recipes-kernel/linux/linux-custom.inc
+++ b/meta/recipes-kernel/linux/linux-custom.inc
@@ -44,7 +44,6 @@ do_install_builddeps() {

addtask install_builddeps after do_prepare_build before do_build

# apt and reprepro may not run in parallel, acquire the Isar lock

do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

-do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"

dpkg_runbuild() {
chmod +x ${WORKDIR}/build-kernel.sh
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

No functional change, just logical ordering.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/conf/bitbake.conf | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index c41f294..1b9f92b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -29,9 +29,9 @@ GITDIR = "${DL_DIR}/git"
P = "${PN}-${PV}"
PF = "${PN}-${PV}-${PR}"
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
+PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"
PR = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[2] or 'r0'}"
PROVIDES = ""
-PV = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[1] or '1.0'}"

S = "${WORKDIR}/${P}"
AUTOREV = "${@bb.fetch2.get_autorev(d)}"
SRC_URI = "file://${FILE}"

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This is now an empty one that should no longer be executed. Align with
OE in this regard.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/base.bbclass | 10 ++++------
1 file changed, 4 insertions(+), 6 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 1d2852a..8c7b021 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -181,14 +181,12 @@ python do_unpack() {

addtask unpack after do_fetch before do_build

-addtask build
-do_build[dirs] = "${TOPDIR}"
-python base_do_build () {
- bb.note("The included, default BB base.bbclass does not define a useful default task.")
- bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
+do_build[noexec] = "1"
+do_build () {
+ :
}

-EXPORT_FUNCTIONS do_build
+addtask build

CLEANFUNCS ?= ""

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

The bananapi machine config contained a "nice" anti-pattern that made
multiconfig explode: By tuning DISTRO_APT_PREFERENCES, it modified the
dependencies of isar-bootstrap-target for all other stretch-armhf
targets. And that nicely exploded when those were built in parallel.

Rather than trying to fix this pattern towards multiconfig
compatibility, which would have been rather complex, simply remove it.
We have buster now, and the buster kernel is what this target want. So
give it to him.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta-isar/conf/distro/debian-stretch-backports.list | 2 --

meta-isar/conf/local.conf.sample | 2 +-
meta-isar/conf/machine/bananapi.conf | 3 ---

.../multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} | 2 +-
meta-isar/conf/multiconfig/preferences.bananapi.conf | 7 -------
5 files changed, 2 insertions(+), 14 deletions(-)

delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
deleted file mode 100644
index 6b744f3..0000000
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ /dev/null
@@ -1,2 +0,0 @@
-deb http://ftp.debian.org/debian stretch-backports main
-deb-src http://ftp.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 5b3a0a1..d9379b1 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -47,9 +47,9 @@ BBMULTICONFIG = " \
qemuamd64-stretch \
qemumipsel-stretch \
qemumipsel-buster \
- bananapi-stretch \
de0-nano-soc-stretch \
hikey-stretch \
+ bananapi-buster \
qemuamd64-buster \
qemuamd64-buster-tgz \
nand-ubi-demo-buster \
diff --git a/meta-isar/conf/machine/bananapi.conf b/meta-isar/conf/machine/bananapi.conf
index 7b380c1..cec8057 100644
--- a/meta-isar/conf/machine/bananapi.conf
+++ b/meta-isar/conf/machine/bananapi.conf
@@ -7,9 +7,6 @@ DISTRO_ARCH ?= "armhf"

KERNEL_NAME ?= "armmp"

-DISTRO_APT_SOURCES_append = " conf/distro/debian-stretch-backports.list"
-DISTRO_APT_PREFERENCES += "conf/multiconfig/preferences.bananapi.conf"
-
IMAGE_INSTALL += "u-boot-script"

IMAGE_TYPE ?= "wic-img"
diff --git a/meta-isar/conf/multiconfig/bananapi-stretch.conf b/meta-isar/conf/multiconfig/bananapi-buster.conf
similarity index 82%
rename from meta-isar/conf/multiconfig/bananapi-stretch.conf
rename to meta-isar/conf/multiconfig/bananapi-buster.conf
index 6c80020..6262a07 100644
--- a/meta-isar/conf/multiconfig/bananapi-stretch.conf
+++ b/meta-isar/conf/multiconfig/bananapi-buster.conf
@@ -4,4 +4,4 @@
# SPDX-License-Identifier: MIT

MACHINE = "bananapi"
-DISTRO = "debian-stretch"
+DISTRO = "debian-buster"
diff --git a/meta-isar/conf/multiconfig/preferences.bananapi.conf b/meta-isar/conf/multiconfig/preferences.bananapi.conf
deleted file mode 100644
index 9f5c196..0000000
--- a/meta-isar/conf/multiconfig/preferences.bananapi.conf
+++ /dev/null
@@ -1,7 +0,0 @@
-Package: linux-image* linux-headers* linux-kbuild*
-Pin: release n=stretch-backports
-Pin-Priority: 501
-
-Package: *
-Pin: release n=stretch-backports
-Pin-Priority: -1
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

That uses the Debian kernel, thus can't build the module cross as well.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta-isar/recipes-kernel/example-module/example-module.bb | 1 +

1 file changed, 1 insertion(+)

diff --git a/meta-isar/recipes-kernel/example-module/example-module.bb b/meta-isar/recipes-kernel/example-module/example-module.bb
index a08a6c5..90db80e 100644
--- a/meta-isar/recipes-kernel/example-module/example-module.bb
+++ b/meta-isar/recipes-kernel/example-module/example-module.bb
@@ -17,6 +17,7 @@ python() {
'rpi-rpfv',
'amd64',
'686-pae',
+ '4kc-malta',
]:
d.setVar('ISAR_CROSS_COMPILE', '0')
}
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Umounting already on runQueueExitWait just pulls the carpet under the
feet of still running builds, causing follow-up errors when just one
recipe failed. Even worse, terminating builds with ctrl-c does not allow
to finish most of the running tasks because they will stumble over
disappearing mounts.

While at it, give the handler a clearer name, namely that of its
trigger.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/isar-events.bbclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 7dd1ed5..62eb25d 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,9 +4,9 @@
# Copyright (C) 2015-2017 ilbers GmbH
# Copyright (c) Siemens AG, 2018

-addhandler isar_handler
+addhandler build_completed

-python isar_handler() {
+python build_completed() {
import subprocess

tmpdir = d.getVar('TMPDIR', True)
@@ -25,4 +25,4 @@ python isar_handler() {
)
}

-isar_handler[eventmask] = "bb.runqueue.runQueueExitWait bb.event.BuildCompleted"
+build_completed[eventmask] = "bb.event.BuildCompleted"
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Commit 77e62c1e9bbd didn't fully address the issue: There can still be
failing build dependency installations due to "apt-get upgrade" refusing
the downgrade without explicit --allow-downgrades.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-devtools/buildchroot/files/deps.sh | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/buildchroot/files/deps.sh b/meta/recipes-devtools/buildchroot/files/deps.sh
index 1963943..19eb9b7 100644
--- a/meta/recipes-devtools/buildchroot/files/deps.sh
+++ b/meta/recipes-devtools/buildchroot/files/deps.sh
@@ -29,4 +29,4 @@ apt-get update \
mk-build-deps $set_arch -t "${install_cmd}" -i -r debian/control

# Upgrade any already installed packages in case we are partially rebuilding
-apt-get upgrade -y
+apt-get upgrade -y --allow-downgrades
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Stretch has no crossbuild-essential-i386, while buster has. That means
we need to use native build for stretch, even if crossbuild was
requested.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/buildchroot.bbclass | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index b7a8ea5..2ba2cdd 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,9 @@ ISAR_CROSS_COMPILE ??= "0"
# Add dependency from the correct buildchroot: host or target
python __anonymous() {
mode = d.getVar('ISAR_CROSS_COMPILE', True)
- if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
+ distro_arch = d.getVar('DISTRO_ARCH')
+ if mode == "0" or d.getVar('HOST_ARCH') == distro_arch or \
+ (d.getVar('HOST_DISTRO') == "debian-stretch" and distro_arch == "i386"):
dep = "buildchroot-target:do_build"
rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
else:
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This follows the pattern used for isar-bootstrap already: Establish a
stable link to the rootfs in the workdir from a folder in DEPLOY_DIR.
This is cleaner than pointing directly into the workdir via
BUILDCHROOT_HOST/TARGET_DIR, and it avoid having to mess with PF.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/buildchroot.bbclass | 2 +-
meta/conf/bitbake.conf | 5 +++--
meta/recipes-devtools/buildchroot/buildchroot-host.bb | 2 ++
meta/recipes-devtools/buildchroot/buildchroot-target.bb | 2 ++
meta/recipes-devtools/buildchroot/buildchroot.inc | 13 +++++++++++++
5 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 2ba2cdd..a87e144 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -21,7 +21,7 @@ python __anonymous() {
d.setVar('BUILDCHROOT_DIR', rootfs)
}

-MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"
+MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}.lock"

buildchroot_do_mounts() {
sudo -s <<'EOSUDO'
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 1b9f92b..f25b20f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -47,11 +47,12 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
# isar specific config
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
+DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"
+BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"
CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 121992c..40a37ea 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,8 @@
DESCRIPTION = "Isar development filesystem for host"
PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

+BUILDCHROOT_VARIANT = "host"
+
require buildchroot.inc
ROOTFS_ARCH = "${HOST_ARCH}"
ROOTFS_DISTRO = "${HOST_DISTRO}"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 5a01258..3b96977 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -6,6 +6,8 @@
DESCRIPTION = "Isar development filesystem for target"
PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

+BUILDCHROOT_VARIANT = "target"
+
require buildchroot.inc

BUILDCHROOT_PREINSTALL ?= "gcc \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 40fc8b0..e704137 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -46,3 +46,16 @@ buildchroot_install_files() {
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh $USER_ID $GROUP_ID

}
+
+DEPLOY_BUILDCHROOT = "${@d.getVar('BUILDCHROOT_' + d.getVar('BUILDCHROOT_VARIANT').upper() + '_DIR')}"
+
+do_buildchroot_deploy[dirs] = "${DEPLOY_DIR_BUILDCHROOT}-${BUILDCHROOT_VARIANT}"
+do_buildchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_BUILDCHROOT}"
+}
+addtask buildchroot_deploy before do_build after do_rootfs
+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {
+ rm -f "${DEPLOY_BUILDCHROOT}"
+}
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

All these recipes now have unique WORKDIRs and STAMPs, and all deploy
their shared artifacts via DEPLOY_DIR.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb | 1 -
meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb | 1 -
meta/recipes-devtools/buildchroot/buildchroot-host.bb | 1 -
meta/recipes-devtools/buildchroot/buildchroot-target.bb | 1 -
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 1 -
5 files changed, 5 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 8363465..b2dded5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,7 +6,6 @@
# SPDX-License-Identifier: MIT

Description = "Minimal host Debian root file system"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 77276e9..d7fcdbf 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,7 +6,6 @@
# SPDX-License-Identifier: MIT

Description = "Minimal target Debian root file system"
-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index 40a37ea..795b77a 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -4,7 +4,6 @@
# Copyright (C) 2015-2018 ilbers GmbH

DESCRIPTION = "Isar development filesystem for host"

-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

BUILDCHROOT_VARIANT = "host"

diff --git a/meta/recipes-devtools/buildchroot/buildchroot-target.bb b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
index 3b96977..68aedaa 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-target.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-target.bb
@@ -4,7 +4,6 @@
# Copyright (C) 2015-2018 ilbers GmbH

DESCRIPTION = "Isar development filesystem for target"

-PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

BUILDCHROOT_VARIANT = "target"

diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 84b8808..e658122 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -12,7 +12,6 @@ SRC_URI = " \
file://configscript.sh \
file://README.sdk"
PV = "0.1"
-PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

TOOLCHAIN = "crossbuild-essential-${DISTRO_ARCH}"
TOOLCHAIN_${HOST_ARCH} = "build-essential"
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This fixes false sharing of parallel builds like
multiconfig:qemuamd64-buster:isar-image-base plus
multiconfig:qemuamd64-buster-tgz:isar-image-base.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/image.bbclass | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 36e9875..89cc5fe 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,9 +2,9 @@

# Copyright (C) 2015-2017 ilbers GmbH

# Make workdir and stamps machine-specific without changing common PN target
-WORKDIR .= "-${MACHINE}"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"
+WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"

IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This removes the last fiddling with PF, addressing the issue - a common
image PN name for multiconfig targets while the results are actually
machine-specific - by tuning the critical paths directly. This are just
WORKDIR as well as STAMP and STAMPCLEAN.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/image.bbclass | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index ec6bd39..36e9875 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,7 +1,10 @@
# This software is a part of ISAR.

# Copyright (C) 2015-2017 ilbers GmbH

-PF = "${PN}-${DISTRO}-${MACHINE}"
+# Make workdir and stamps machine-specific without changing common PN target
+WORKDIR .= "-${MACHINE}"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/*-*"

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

There is nothing that the sdkchroot will gain after its bootstrap.
Therefore we can clean up the apt cache already there, reusing common
infrastructure.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/image-sdk-extension.bbclass | 3 ---
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 2 +-
2 files changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 8df3ed1..a53d954 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -13,9 +13,6 @@ do_populate_sdk() {
# Copy isar-apt with deployed Isar packages
sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt

- # Purge apt cache to make image slimmer
- sudo rm -rf ${SDKCHROOT_DIR}/rootfs/var/cache/apt/*
-
sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true
sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 3cfe05c..d2cc92e 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -23,7 +23,7 @@ ROOTFS_ARCH = "${HOST_ARCH}"
ROOTFS_DISTRO = "${HOST_DISTRO}"
ROOTFSDIR = "${S}"
ROOTFS_PACKAGES = "${SDKCHROOT_PREINSTALL} ${TOOLCHAIN}"
-ROOTFS_FEATURES += ""
+ROOTFS_FEATURES += "clean-package-cache"

python() {
if d.getVar("HOST_ARCH") not in ['i386', 'amd64']:
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This avoids having to point from image-sdk-extension directly into the
sdkchroot target workdir. It also aligns the sdk with isar-bootstrap and
buildchroot in this regard.

As we now have a deployment target of the sdk rootfs (SDKCHROOT_DIR), we
can drop the previous link in DEPLOY_DIR_IMAGE.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/image-sdk-extension.bbclass | 20 ++++++++------------
meta/conf/bitbake.conf | 2 ++
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 11 +++++++++++
3 files changed, 21 insertions(+), 12 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index a53d954..b9f2cf4 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,29 +5,25 @@
#
# This class extends the image.bbclass to supply the creation of a sdk

-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-
do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"

do_populate_sdk() {
# Copy isar-apt with deployed Isar packages

- sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/rootfs/isar-apt
+ sudo cp -Trpfx ${REPO_ISAR_DIR}/${DISTRO} ${SDKCHROOT_DIR}/isar-apt

- sudo umount -R ${SDKCHROOT_DIR}/rootfs/dev || true

- sudo umount ${SDKCHROOT_DIR}/rootfs/proc || true
- sudo umount -R ${SDKCHROOT_DIR}/rootfs/sys || true
+ sudo umount -R ${SDKCHROOT_DIR}/dev || true
+ sudo umount ${SDKCHROOT_DIR}/proc || true
+ sudo umount -R ${SDKCHROOT_DIR}/sys || true

# Remove setup scripts
- sudo rm -f ${SDKCHROOT_DIR}/rootfs/chroot-setup.sh ${SDKCHROOT_DIR}/rootfs/configscript.sh
+ sudo rm -f ${SDKCHROOT_DIR}/chroot-setup.sh ${SDKCHROOT_DIR}/configscript.sh

# Copy mount_chroot.sh for convenience
- sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}/rootfs
+ sudo cp ${ISARROOT}/scripts/mount_chroot.sh ${SDKCHROOT_DIR}

# Create SDK archive
- sudo tar -C ${SDKCHROOT_DIR} --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
+ cd -P ${SDKCHROOT_DIR}/..
+ sudo tar --transform="s|^rootfs|sdk-${DISTRO}-${DISTRO_ARCH}|" \
-c rootfs | xz -T0 > ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}.tar.xz
-
- # Install deployment link for local use
- ln -Tfsr ${SDKCHROOT_DIR}/rootfs ${DEPLOY_DIR_IMAGE}/sdk-${DISTRO}-${DISTRO_ARCH}
}
addtask populate_sdk after do_rootfs
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index f25b20f..e5681bf 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -48,11 +48,13 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"

WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"

DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
+DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"

DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"

BUILDCHROOT_HOST_DIR = "${DEPLOY_DIR_BUILDCHROOT}-host/${HOST_DISTRO}-${HOST_ARCH}_${DISTRO}-${DISTRO_ARCH}"

BUILDCHROOT_TARGET_DIR = "${DEPLOY_DIR_BUILDCHROOT}-target/${DISTRO}-${DISTRO_ARCH}"

+SDKCHROOT_DIR = "${DEPLOY_DIR_SDKCHROOT}/${DISTRO}-${DISTRO_ARCH}"

CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index d2cc92e..84b8808 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -56,3 +56,14 @@ sdkchroot_install_files() {
sudo install -m 755 ${WORKDIR}/configscript.sh ${S}
sudo chroot ${S} /configscript.sh ${DISTRO_ARCH}
}
+
+do_sdkchroot_deploy[dirs] = "${DEPLOY_DIR_SDKCHROOT}"
+do_sdkchroot_deploy() {
+ ln -Tfsr "${ROOTFSDIR}" "${SDKCHROOT_DIR}"
+}
+addtask sdkchroot_deploy before do_build after do_rootfs

+
+CLEANFUNCS = "clean_deploy"
+clean_deploy() {

+ rm -f "${SDKCHROOT_DIR}"
+}
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.gitlab-ci.yml | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 1c6fa84..5c64adc 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -11,3 +11,9 @@ all:
- export ftp_proxy=$FTP_PROXY
- export no_proxy=$NO_PROXY
- scripts/ci_build.sh -q -f
+ artifacts:
+ name: "logs-$CI_JOB_ID"
+ paths:
+ - build/tmp/work/*/*/*/temp
+ when: on_failure
+ expire_in: 1 week
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Pick up the OE pattern to have a per-PV-PR workdir under PN. It also
aligns this with our STAMP setting.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

doc/user_manual.md | 2 +-
meta/classes/image.bbclass | 2 +-
meta/conf/bitbake.conf | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/doc/user_manual.md b/doc/user_manual.md
index 289fd1d..eba5f0f 100644
--- a/doc/user_manual.md
+++ b/doc/user_manual.md
@@ -452,7 +452,7 @@ The user may use `met-isar/recipes-core-images` as a template for new image reci
### General Information
The image recipe in Isar creates a folder with target root filesystem. Its default location is:
```
-tmp/work/${DISTRO}-${DISTRO_ARCH}/${MACHINE}/${IMAGE}/rootfs
+tmp/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}/rootfs
```
Every image type in Isar is implemented as a `bitbake` class. The goal of these classes is to pack root filesystem folder to appropriate format.

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89cc5fe..0965f26 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -2,7 +2,7 @@

# Copyright (C) 2015-2017 ilbers GmbH

# Make workdir and stamps machine-specific without changing common PN target

-WORKDIR .= "-${MACHINE}-${IMAGE_TYPE}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"

STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/${PV}-${PR}"

STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}-${IMAGE_TYPE}/*-*"

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9c9c7fe..bca1114 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -45,7 +45,7 @@ PERSISTENT_DIR = "${TMPDIR}/cache"

GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"

# isar specific config
-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
+WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"

DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"

DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

The last misuse of PF is gone, we can adjust the defaults to OE.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/conf/bitbake.conf | 4 ++--

1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index e5681bf..9c9c7fe 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -36,8 +36,8 @@ S = "${WORKDIR}/${P}"

AUTOREV = "${@bb.fetch2.get_autorev(d)}"
SRC_URI = "file://${FILE}"

STAMPS_DIR ?= "${TMPDIR}/stamps"
-STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/${PV}-${PR}"
-STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
+STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"

SVNDIR = "${DL_DIR}/svn"
T = "${WORKDIR}/temp"
TMPDIR = "${TOPDIR}/tmp"

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe. This also aligns base-apt with isar-apt.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-devtools/base-apt/base-apt.bb | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-devtools/base-apt/base-apt.bb b/meta/recipes-devtools/base-apt/base-apt.bb
index 488a44d..f1a0e68 100644
--- a/meta/recipes-devtools/base-apt/base-apt.bb
+++ b/meta/recipes-devtools/base-apt/base-apt.bb
@@ -37,4 +37,4 @@ do_cache_config() {
fi
}

-addtask cache_config after do_build
+addtask cache_config after do_unpack before do_build
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

When a task is run more than once per build, this indicates a bug,
usually (though not only) related to incorrect multiconfig use. Such
bugs are hard to find in practice, so they could benefit from some
tooling.

This adds a very simple but effective check for such re-run scenarios
by putting a "once.<taskname>" stamp file into each workdir during a
build. If such a file already exists, an error is thrown. The workdirs
are purged after the parsing phase on the beginning of each build so
that no false positives are raised.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/isar-events.bbclass | 22 ++++++++++++++++++++++
1 file changed, 22 insertions(+)

diff --git a/meta/classes/isar-events.bbclass b/meta/classes/isar-events.bbclass
index 62eb25d..1335356 100644
--- a/meta/classes/isar-events.bbclass
+++ b/meta/classes/isar-events.bbclass
@@ -4,6 +4,28 @@

# Copyright (C) 2015-2017 ilbers GmbH

# Copyright (c) Siemens AG, 2018

+addhandler parse_completed
+
+python parse_completed() {
+ bb.utils.remove(d.getVar('TMPDIR') + "/work/*/*/*/temp/once.*")
+}
+parse_completed[eventmask] = "bb.event.ParseCompleted"
+
+addhandler task_started
+
+python task_started() {
+ task = d.getVar('BB_RUNTASK')
+ try:
+ f = open("%s/once.%s" % (d.getVar('T'), task), "x")
+ f.close()
+ except FileExistsError:
+ bb.error("Detect multiple executions of %s in %s" %
+ (task, d.getVar('WORKDIR')))
+ bb.error("Rerun a clean build with empty STAMPCLEAN " \
+ "and compare the sigdata files")
+}
+task_started[eventmask] = "bb.build.TaskStarted"
+
addhandler build_completed

python build_completed() {
--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

There is not good reason to move anything beyond the final do_build task
in a recipe.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 50df06d..d4541a4 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -131,7 +131,7 @@ do_deploy_deb() {
${S}/../*.deb
}

-addtask deploy_deb after do_dpkg_build
+addtask deploy_deb after do_dpkg_build before do_build

do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
do_deploy_deb[depends] = "isar-apt:do_cache_config"
do_deploy_deb[dirs] = "${S}"

--
2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

There is no reason for using a lock here. All bootstraps for the same
distro/arch are detected by bitbake as identical targets in multiconfig
scenarios and not run multiple times. If they are run multiple times,
there is another bug, such as with the recently fixed bananapi machine.

Also remove the equally bogus re-execution test in
do_apt_config_prepare. This also fixes rebuilds of the isar-bootstrap
target in case some parameter has changed.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

.../isar-bootstrap/isar-bootstrap-host.bb | 30 ++++++++++------------
.../isar-bootstrap/isar-bootstrap-target.bb | 27 +++++++++----------
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ----
3 files changed, 26 insertions(+), 36 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index b2dded5..8c211a9 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -8,7 +8,6 @@

Description = "Minimal host Debian root file system"

DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc

@@ -25,21 +24,20 @@ do_apt_config_prepare[vardeps] += "\
DEPLOY_ISAR_BOOTSTRAP \
"
python do_apt_config_prepare() {
- if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
- apt_preferences_out = d.getVar("APTPREFS", True)
- apt_preferences_list = (
- d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
- ).split()
- aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
- apt_sources_out = d.getVar("APTSRCS", True)
- apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
- apt_sources_list = (
- d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
- ).split()
-
- aggregate_files(d, apt_sources_list, apt_sources_init_out)
- aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+ apt_preferences_out = d.getVar("APTPREFS", True)
+ apt_preferences_list = (
+ d.getVar("HOST_DISTRO_APT_PREFERENCES", True) or ""
+ ).split()
+ aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+ apt_sources_out = d.getVar("APTSRCS", True)
+ apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+ apt_sources_list = (
+ d.getVar("HOST_DISTRO_APT_SOURCES", True) or ""
+ ).split()
+
+ aggregate_files(d, apt_sources_list, apt_sources_init_out)
+ aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
}
addtask apt_config_prepare before do_bootstrap after do_unpack

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index d7fcdbf..80e7f40 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -8,7 +8,6 @@

Description = "Minimal target Debian root file system"

DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"

-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc

@@ -22,19 +21,18 @@ do_apt_config_prepare[vardeps] += "\
DEPLOY_ISAR_BOOTSTRAP \
"
python do_apt_config_prepare() {
- if not os.path.islink(d.getVar("DEPLOY_ISAR_BOOTSTRAP", True)):
- apt_preferences_out = d.getVar("APTPREFS", True)
- apt_preferences_list = (
- d.getVar("DISTRO_APT_PREFERENCES", True) or ""
- ).split()
- aggregate_files(d, apt_preferences_list, apt_preferences_out)
-
- apt_sources_out = d.getVar("APTSRCS", True)
- apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
- apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
-
- aggregate_files(d, apt_sources_list, apt_sources_init_out)
- aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
+ apt_preferences_out = d.getVar("APTPREFS", True)
+ apt_preferences_list = (
+ d.getVar("DISTRO_APT_PREFERENCES", True) or ""
+ ).split()
+ aggregate_files(d, apt_preferences_list, apt_preferences_out)
+
+ apt_sources_out = d.getVar("APTSRCS", True)
+ apt_sources_init_out = d.getVar("APTSRCS_INIT", True)
+ apt_sources_list = (d.getVar("DISTRO_APT_SOURCES", True) or "").split()
+
+ aggregate_files(d, apt_sources_list, apt_sources_init_out)
+ aggregate_aptsources_list(d, apt_sources_list, apt_sources_out)
}
addtask apt_config_prepare before do_bootstrap after do_unpack

@@ -45,4 +43,3 @@ do_bootstrap() {
isar_bootstrap
}
addtask bootstrap before do_build after do_generate_keyrings
-
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index aea89a9..36cefd2 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -212,14 +212,11 @@ do_generate_keyrings() {
}
addtask generate_keyrings before do_build after do_unpack

-
-
def get_host_release():
import platform
rel = platform.release()
return rel

-
do_bootstrap[vardeps] += "DISTRO_APT_PREMIRRORS"
do_bootstrap[dirs] = "${DEPLOY_DIR_BOOTSTRAP}"

@@ -246,7 +243,6 @@ isar_bootstrap() {
E="${@bb.utils.export_proxies(d)}"
export IS_HOST debootstrap_args E
sudo -E -s <<'EOSUDO'
- ( flock 9
set -e
if [ ! -e "${DEPLOY_ISAR_BOOTSTRAP}" ]; then
rm -rf "${ROOTFSDIR}"
@@ -340,7 +336,6 @@ isar_bootstrap() {
# Finalize debootstrap by setting the link in deploy
ln -Tfsr "${ROOTFSDIR}" "${DEPLOY_ISAR_BOOTSTRAP}"
fi
- ) 9>'${ISAR_BOOTSTRAP_LOCK}'
EOSUDO
}

--
2.16.4

[Cedric Hombourger]

On 8/26/2019 8:26 AM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> Bitbake file locks are not just exclusive flock calls. For whatever
> reason, bitbake prefers to delete the lock file before unlocking them.
> That obviously creates a nasty race with other locker requests: Those
> already waiting will get access to the removed lock, those coming later
> will create a new one. To resolve that race, bitbakes lockfile() tests
> if the inode of the acquired lock file is identical to the one that is
> currently visible via the lock name. If not, it retries the lock. What a
> pointless mess...
>
> Our open-coded "flock -s" does not do that, thus will often acquire only
> a stale lock. Fix that by using bitbake's services for locking. That
> requires to move the lock acquisition at the top level of the task, ie.
> do_build. Thus, also dpkg-raw and kernel builds are now under the shared
> (read) lock. The better fix for this issue would actually be private
> buildchroots for each build, but that will come later.

Very good catch! ack

[Henning Schild]

Am Mon, 26 Aug 2019 08:26:08 +0200
schrieb "[ext] Jan Kiszka" <jan.k...@siemens.com>:

This actually once came from bitbake/classes/base.bbclass

You might want to compare with that.

the default does not have noexec but it does have nostamp

> -EXPORT_FUNCTIONS do_build
> +addtask build

The default bitbake one has both of these lines.

Henning

> CLEANFUNCS ?= ""
>

[Claudius Heine]

Hi Jan,

Can you explain the issue of the PN/PF usage a bit more. I did not
understand:

> As long as recipes mess with PF, we can't use OE's pattern as-is which
> is PN-based.

The recipes that mess with PF are: buildchroot-*.bb sdkchroot.bb
isar-bootstrap-*.bb and image.bbclass. All of which do not really have a
version.

So I would think that:

STAMP = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/${PF}"
STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}/*-*"

should work. Or am I missing something?

regards,
Claudius

> +STAMPCLEAN = "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PF}/*-*"
> SVNDIR = "${DL_DIR}/svn"
> T = "${WORKDIR}/temp"
> TMPDIR = "${TOPDIR}/tmp"
> diff --git a/meta/recipes-kernel/linux/linux-custom.inc b/meta/recipes-kernel/linux/linux-custom.inc
> index ee5f20c..12d8d9c 100644
> --- a/meta/recipes-kernel/linux/linux-custom.inc
> +++ b/meta/recipes-kernel/linux/linux-custom.inc
> @@ -44,7 +44,6 @@ do_install_builddeps() {
> addtask install_builddeps after do_prepare_build before do_build
> # apt and reprepro may not run in parallel, acquire the Isar lock
> do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> -do_install_builddeps[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>
> dpkg_runbuild() {
> chmod +x ${WORKDIR}/build-kernel.sh
>

--

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

[Claudius Heine]

Hi Jan,

On 26/08/2019 08.26, [ext] Jan Kiszka wrote:

What is the difference between this and

do_build[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"

?

regards,
Claudius

> }
>
> CLEANFUNCS += "repo_clean"
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index dd123c3..df47ced 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -18,7 +18,6 @@ do_install_builddeps[lockfiles] += "${REPO_ISAR_DIR}/isar.lock"
> # Build package from sources using build script
> dpkg_runbuild() {
> E="${@ bb.utils.export_proxies(d)}"
> - flock -s "${REPO_ISAR_DIR}/isar.lock" \
> - sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> - /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
> + sudo -E chroot --userspec=$( id -u ):$( id -g ) ${BUILDCHROOT_DIR} \
> + /isar/build.sh ${PP}/${PPS} ${DISTRO_ARCH}
> }
>

--

[Henning Schild]

Am Mon, 26 Aug 2019 08:26:19 +0200

schrieb "[ext] Jan Kiszka" <jan.k...@siemens.com>:

> From: Jan Kiszka <jan.k...@siemens.com>
>
> This removes the last fiddling with PF, addressing the issue - a
> common image PN name for multiconfig targets while the results are
> actually machine-specific - by tuning the critical paths directly.
> This are just WORKDIR as well as STAMP and STAMPCLEAN.
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
> meta/classes/image.bbclass | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index ec6bd39..36e9875 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -1,7 +1,10 @@
> # This software is a part of ISAR.
> # Copyright (C) 2015-2017 ilbers GmbH
>
> -PF = "${PN}-${DISTRO}-${MACHINE}"
> +# Make workdir and stamps machine-specific without changing common
> PN target +WORKDIR .= "-${MACHINE}"
> +STAMP =
> "${STAMPS_DIR}/${DISTRO}-${DISTRO_ARCH}/${PN}-${MACHINE}/${PV}-${PR}"

.../${PF} ?

Henning

[Jan Kiszka]

PF is a effectively read-only. This is fixed later on this series. Then there is
no need to deviate from OE any more.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Jan Kiszka]

On 26.08.19 10:30, Claudius Heine wrote:

shared=True.

[Jan Kiszka]

I compared to OE and took its version. The bitbake one is a template that makes
no sense here.

>
> the default does not have noexec but it does have nostamp
>
>> -EXPORT_FUNCTIONS do_build
>> +addtask build
>
> The default bitbake one has both of these lines.

Again, OE is closer to our use case.

[Jan Kiszka]

Nope, that's already included in PN, PV and PR.

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

The buildchroot is shared between images of the same target arch. Just
using /tmp/<image-name> can cause conflicts and build failures.

Seen with jailhouse-images.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/wic-img.bbclass | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 1ccbe55..43e8fd5 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -144,14 +144,18 @@ EOSUDO
export BUILDDIR=${BUILDDIR}
export MTOOLS_SKIP_CHECK=1

+ # create the temp dir in the buildchroot to ensure uniqueness
+ BUILDCHROOT_WICTMP=$(mktemp -d -p ${BUILDCHROOT_DIR}/tmp)
+ WICTMP=$(echo $BUILDCHROOT_WICTMP | sed 's|^${BUILDCHROOT_DIR}||')
+
sudo -E chroot ${BUILDCHROOT_DIR} \
${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
--vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
- -o /tmp/${IMAGE_FULLNAME}.wic/ \
+ -o $WICTMP/${IMAGE_FULLNAME}.wic/ \
--bmap \
-e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
- WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)
+ WIC_DIRECT=$(ls -t -1 $BUILDCHROOT_WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)
cp -f ${WIC_DIRECT} ${WIC_IMAGE_FILE}
cp -f ${WIC_DIRECT}.bmap ${WIC_IMAGE_FILE}.bmap
}
--

2.16.4

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

The buildchroot is shared between images of the same target arch. Just
using /tmp/<image-name> can cause conflicts and build failures.

Seen with jailhouse-images.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Changes in v2:
- avoid back and forth with BUILDCHROOT_WICTMP and WICTMP

meta/classes/wic-img.bbclass | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/wic-img.bbclass b/meta/classes/wic-img.bbclass
index 1ccbe55..795640f 100644
--- a/meta/classes/wic-img.bbclass
+++ b/meta/classes/wic-img.bbclass
@@ -144,14 +144,17 @@ EOSUDO

export BUILDDIR=${BUILDDIR}
export MTOOLS_SKIP_CHECK=1

+ # create the temp dir in the buildchroot to ensure uniqueness

+ WICTMP=$(cd ${BUILDCHROOT_DIR}; mktemp -d -p tmp)

+
sudo -E chroot ${BUILDCHROOT_DIR} \
${ISARROOT}/scripts/wic create ${WKS_FULL_PATH} \
--vars "${STAGING_DIR}/${MACHINE}/imgdata/" \
- -o /tmp/${IMAGE_FULLNAME}.wic/ \

+ -o /$WICTMP/${IMAGE_FULLNAME}.wic/ \

--bmap \
-e ${IMAGE_BASENAME} ${WIC_CREATE_EXTRA_ARGS}
sudo chown -R $(stat -c "%U" ${ISARROOT}) ${ISARROOT}/meta ${ISARROOT}/meta-isar ${ISARROOT}/scripts || true
- WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/tmp/${IMAGE_FULLNAME}.wic/*.direct | head -1)

+ WIC_DIRECT=$(ls -t -1 ${BUILDCHROOT_DIR}/$WICTMP/${IMAGE_FULLNAME}.wic/*.direct | head -1)

[Claudius Heine]

Hi Jan,

On 26/08/2019 08.26, [ext] Jan Kiszka wrote:

> From: Jan Kiszka <jan.k...@siemens.com>
>
> There is not good reason to move anything beyond the final do_build task
> in a recipe.

This is not exactly what this patch does though, so you might want to
update the description.

What you do here is add do_deploy_deb as a dependency to do_build,
meaning now deploying a debian package is part of the default build
target. The main difference is in the definition of when is a recipe
considered to be 'build': Either after the debian package was created
inside the work directory or when the debian package is added to a
repository. I would argue that there are reasons for both perspectives.

It is possible to consider 'do_depoy*' in packages more like 'make
install', and as such not part of the default target.

regards,
Claudius

>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
> meta/classes/dpkg-base.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 50df06d..d4541a4 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -131,7 +131,7 @@ do_deploy_deb() {
> ${S}/../*.deb
> }
>
> -addtask deploy_deb after do_dpkg_build
> +addtask deploy_deb after do_dpkg_build before do_build
> do_deploy_deb[lockfiles] = "${REPO_ISAR_DIR}/isar.lock"
> do_deploy_deb[depends] = "isar-apt:do_cache_config"
> do_deploy_deb[dirs] = "${S}"
>

--

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

The bananapi machine config contained a "nice" anti-pattern that made
multiconfig explode: By tuning DISTRO_APT_PREFERENCES, it modified the
dependencies of isar-bootstrap-target for all other stretch-armhf
targets. And that nicely exploded when those were built in parallel.

Rather than trying to fix this pattern towards multiconfig
compatibility, which would have been rather complex, simply remove it.
We have buster now, and the buster kernel is what this target want. So
give it to him.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Changes in v2:
- rebased over next

The complete series of my pending patches is still at
https://github.com/siemens/isar/commits/jan/queue

meta-isar/conf/distro/debian-stretch-backports.list | 2 --
meta-isar/conf/local.conf.sample | 2 +-
meta-isar/conf/machine/bananapi.conf | 3 ---
.../multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} | 2 +-
meta-isar/conf/multiconfig/preferences.bananapi.conf | 7 -------
5 files changed, 2 insertions(+), 14 deletions(-)
delete mode 100644 meta-isar/conf/distro/debian-stretch-backports.list
rename meta-isar/conf/multiconfig/{bananapi-stretch.conf => bananapi-buster.conf} (82%)
delete mode 100644 meta-isar/conf/multiconfig/preferences.bananapi.conf

diff --git a/meta-isar/conf/distro/debian-stretch-backports.list b/meta-isar/conf/distro/debian-stretch-backports.list
deleted file mode 100644
index 6b744f3..0000000
--- a/meta-isar/conf/distro/debian-stretch-backports.list
+++ /dev/null
@@ -1,2 +0,0 @@
-deb http://ftp.debian.org/debian stretch-backports main
-deb-src http://ftp.debian.org/debian stretch-backports main
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 60805eb..d73da1d 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -54,7 +54,7 @@ BBMULTICONFIG = " \
qemumipsel-stretch \
qemumipsel-buster \
qemumipsel-bullseye \
- bananapi-stretch \
+ bananapi-buster \
de0-nano-soc-stretch \
hikey-stretch \
nand-ubi-demo-buster \
diff --git a/meta-isar/conf/machine/bananapi.conf b/meta-isar/conf/machine/bananapi.conf
index 7b380c1..cec8057 100644
--- a/meta-isar/conf/machine/bananapi.conf
+++ b/meta-isar/conf/machine/bananapi.conf
@@ -7,9 +7,6 @@ DISTRO_ARCH ?= "armhf"

KERNEL_NAME ?= "armmp"

-DISTRO_APT_SOURCES_append = " conf/distro/debian-stretch-backports.list"
-DISTRO_APT_PREFERENCES += "conf/multiconfig/preferences.bananapi.conf"
-
IMAGE_INSTALL += "u-boot-script"

IMAGE_TYPE ?= "wic-img"
diff --git a/meta-isar/conf/multiconfig/bananapi-stretch.conf b/meta-isar/conf/multiconfig/bananapi-buster.conf
similarity index 82%
rename from meta-isar/conf/multiconfig/bananapi-stretch.conf
rename to meta-isar/conf/multiconfig/bananapi-buster.conf
index 6c80020..6262a07 100644
--- a/meta-isar/conf/multiconfig/bananapi-stretch.conf
+++ b/meta-isar/conf/multiconfig/bananapi-buster.conf
@@ -4,4 +4,4 @@
# SPDX-License-Identifier: MIT

MACHINE = "bananapi"
-DISTRO = "debian-stretch"
+DISTRO = "debian-buster"
diff --git a/meta-isar/conf/multiconfig/preferences.bananapi.conf b/meta-isar/conf/multiconfig/preferences.bananapi.conf
deleted file mode 100644
index 9f5c196..0000000
--- a/meta-isar/conf/multiconfig/preferences.bananapi.conf
+++ /dev/null
@@ -1,7 +0,0 @@
-Package: linux-image* linux-headers* linux-kbuild*
-Pin: release n=stretch-backports
-Pin-Priority: 501
-
-Package: *
-Pin: release n=stretch-backports
-Pin-Priority: -1
--
2.16.4

[Jan Kiszka]

I just understood (thanks to a rare CI failure) what this lock was papering
over: Raspbian and Debian builds for the same target distro/arch share their
host boostrap deployment. That needs to be fixed. Of course, that lock didn't
solve anything because it forced the second user to live with the bootstrap
settings of the first one.

Again the time to remind everyone to try harder understanding issues as they
occur, rather than throwing random stuff at them or even ignore them as "not my
use case". Those things always tend to bite us again at some point. I think we
are getting better here, though.

[Baurzhan Ismagulov]

On Mon, Aug 26, 2019 at 08:26:03AM +0200, Jan Kiszka wrote:
> Due to immediate expansion, APT_KEYS_DIR and DISTRO_BOOTSTRAP_KEYRING
> were set to invalid WORKDIR, rather than the value relevant when running
> do_generate_keyrings in recipe build context.

Thanks, merged 02-08, 10, 11 to next.

With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

On Wed, Aug 28, 2019 at 09:32:43AM +0200, Claudius Heine wrote:
> > There is not good reason to move anything beyond the final do_build task
> > in a recipe.
>
> This is not exactly what this patch does though, so you might want to update
> the description.
>
> What you do here is add do_deploy_deb as a dependency to do_build, meaning
> now deploying a debian package is part of the default build target. The main
> difference is in the definition of when is a recipe considered to be
> 'build': Either after the debian package was created inside the work
> directory or when the debian package is added to a repository. I would argue
> that there are reasons for both perspectives.
>
> It is possible to consider 'do_depoy*' in packages more like 'make install',
> and as such not part of the default target.

Thanks for the explanation. Yes, there are both perspectives. OE deploys by
default. The description doesn't expand on the implications, which I think is
ok. I'm going to apply this.

With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

On Mon, Aug 26, 2019 at 08:26:01AM +0200, Jan Kiszka wrote:
> Highlights:
> - fixed custom DISTRO_BOOTSTRAP_KEYS
> - fixed shared lock across dpkg_runbuild (nasty stuff...)
> - reworked STAMPing
> - fixed follow-up errors on build terminations (error or ctrl-c)
> - fixed two false sharings
> - detection of false task sharing
> - stop mangling PF
> - no more tasks beyond do_build
> - fixed isar-bootstrap rebuild
> - deploy logs on gitlab-ci errors

Applied to next, thanks.

With kind regards,
Baurzhan.