[PATCH 0/8] Cleanup rootfs creation

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Hi,

this patchset contains some patches that where developed while implementing
the preprocessing image pipeline. They are universally useful, but do prepare
for the next steps.

Cheers,
Claudius

Claudius Heine (8):
isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
base.bbclass
move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
bitbake.conf: remove unneeded and differently used variables
image.bbclass: make IMAGE_ROOTFS overwritable
bitbake.conf: set default QEMU_ARCH variables
buildchroot/configscript: make creation of builder uid/gid idempotent

meta/classes/base.bbclass | 9 +++++++++
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/image-sdk-extension.bbclass | 2 +-
meta/classes/image.bbclass | 2 +-
meta/classes/isar-bootstrap-helper.bbclass | 14 --------------
meta/conf/bitbake.conf | 13 ++++++++-----
.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----
.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
.../buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--
.../buildchroot/files/configscript.sh | 4 ++--
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 3 +--
13 files changed, 30 insertions(+), 41 deletions(-)

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
support was dropped.)

Signed-off-by: Claudius Heine <c...@denx.de>
---
meta/classes/base.bbclass | 9 +++++++++
meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 9a1837f..ca80bf8 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -20,6 +20,15 @@

THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"

+def get_deb_host_arch():
+ import subprocess
+ host_arch = subprocess.check_output(
+ ["dpkg", "--print-architecture"]
+ ).decode('utf-8').strip()
+ return host_arch
+HOST_ARCH ??= "${@get_deb_host_arch()}"
+HOST_DISTRO ??= "${DISTRO}"
+
die() {
bbfatal "$*"
}
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 38d90fb..59c447b 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -7,18 +7,6 @@

IMAGE_TRANSIENT_PACKAGES ??= ""

-def get_deb_host_arch():
- import subprocess
- host_arch = subprocess.check_output(
- ["dpkg", "--print-architecture"]
- ).decode('utf-8').strip()
- return host_arch
-
-#Debian Distribution for SDK host
-HOST_DISTRO ?= "debian-stretch"
-#Determine SDK host architecture if not explicitly set
-HOST_ARCH ?= "${@get_deb_host_arch()}"
-
HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"

def reverse_bb_array(d, varname):
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/recipes-devtools/buildchroot/files/configscript.sh | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index 25a21ef..c4d8cf8 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -10,6 +10,6 @@ locales locales/locales_to_be_generated multiselect en_US.UTF-8 UTF-8
locales locales/default_environment_locale select en_US.UTF-8
END

-groupadd --system builder -o --gid $2
-useradd --system -o --uid $1 --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
+grep -q "builder:x:$2" /etc/group || groupadd -f --system builder -o --gid $2
+grep -q "builder:x:$1" /etc/passwd || useradd --system -o --uid $1 --gid builder --no-create-home --home /home/builder --no-user-group --comment "Isar buildchroot build user" builder
chown -R builder:builder /home/builder
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

IMAGE_CMD and TARGET_ARCH are not used in Isar or any bitbake code.

IMAGE_ROOTFS is used as the directory where image recipes store their
root file system. Pointing that to a global path like '${TMPDIR}/rootfs'
does not make sense for Isar.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 3 ---
1 file changed, 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7c1c89f..21d13ef 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -26,8 +26,6 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
GITDIR = "${DL_DIR}/git"
-IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
-IMAGE_ROOTFS = "${TMPDIR}/rootfs"
P = "${PN}-${PV}"
PF = "${PN}-${PV}-${PR}"
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
@@ -39,7 +37,6 @@ SRC_URI = "file://${FILE}"
STAMP = "${TMPDIR}/stamps/${PF}"
SVNDIR = "${DL_DIR}/svn"
T = "${WORKDIR}/temp"
-TARGET_ARCH = "${BUILD_ARCH}"
TMPDIR = "${TOPDIR}/tmp"
PERSISTENT_DIR = "${TMPDIR}/cache"
GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This way recipes that inherit image.bbclass can set the this path
themselves. This was previously not possible, because the bitbake.conf
set this variable with '='.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f9a3052..6881d37 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"

IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
-IMAGE_ROOTFS = "${WORKDIR}/rootfs"
+IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"

IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 21d13ef..37f1963 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -53,6 +53,12 @@ CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

+# Setting default QEMU_ARCH variables for different DISTRO_ARCH:
+QEMU_ARCH_amd64 = "x86_64"
+QEMU_ARCH_i386 = "i386"
+QEMU_ARCH_armhf = "arm"
+QEMU_ARCH_arm64 = "aarch64"
+
# Codename of the repository created by the caching class
DEBDISTRONAME = "isar"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This variable is only used in 'isar-bootstrap-host.bb' and mentioned in
'isar-bootstrap.inc'. So it does not make sense to have it available
anywhere else.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/isar-bootstrap-helper.bbclass | 2 --
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 59c447b..0d77929 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -7,8 +7,6 @@

IMAGE_TRANSIENT_PACKAGES ??= ""

-HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
-
def reverse_bb_array(d, varname):
array = d.getVar(varname, True)
if array is None:
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index c1b571a..0ec2e2b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -27,6 +27,7 @@ APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"

DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/buildchroot.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a87bf4..ea1538a 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,7 @@ ISAR_CROSS_COMPILE ??= "0"
# Add dependency from the correct buildchroot: host or target
python __anonymous() {
mode = d.getVar('ISAR_CROSS_COMPILE', True)
- if mode == "0":
+ if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
dep = "buildchroot-target:do_build"
rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
else:
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Each of those are dependent on both HOST_ARCH and DISTRO_ARCH, so the
stamps etc. should not be shared.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image-sdk-extension.bbclass | 2 +-
meta/conf/bitbake.conf | 4 ++--

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

meta/recipes-devtools/buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 3 +--
7 files changed, 10 insertions(+), 20 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 1838080..8df3ed1 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,7 +5,7 @@
#
# This class extends the image.bbclass to supply the creation of a sdk

-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}"
+SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..7c1c89f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -50,8 +50,8 @@ DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target/rootfs"
+BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"

CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 08b068f..6d0f5f5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,17 +6,13 @@
# SPDX-License-Identifier: MIT

Description = "Minimal host Debian root file system"
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
-DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
+DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
+ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc
-inherit isar-bootstrap-helper

-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_apt_config_prepare[dirs] = "${WORKDIR}"
do_apt_config_prepare[vardeps] += "\
APTPREFS \
@@ -47,7 +43,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack

OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}"

-do_bootstrap[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap --host
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 79f3e34..552a317 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,16 +6,13 @@
# SPDX-License-Identifier: MIT

Description = "Minimal target Debian root file system"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc

-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_apt_config_prepare[dirs] = "${WORKDIR}"
do_apt_config_prepare[vardeps] += "\
APTPREFS \
@@ -44,7 +41,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack

OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}"

-do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index c8c10c0..e8f7f69 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,7 @@
DESCRIPTION = "Isar development filesystem for host"

require buildchroot.inc
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

BUILDCHROOT_PREINSTALL ?= "make \
debhelper \
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 764bc5e..cdb4c07 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -12,13 +12,12 @@ SRC_URI = "file://configscript.sh \
file://common.sh \
file://deps.sh"
PV = "1.0"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

inherit isar-bootstrap-helper

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
BUILDCHROOT_DIR = "${WORKDIR}/rootfs"

-do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
${BUILDCHROOT_DIR}/isar-apt \
${BUILDCHROOT_DIR}/downloads \
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 895a454..249fa94 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -14,6 +14,7 @@ SRC_URI = " \
PV = "0.1"

inherit isar-bootstrap-helper
+PF = "${PN}-${ROOTFS_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

SDKCHROOT_PREINSTALL := "debhelper \
autotools-dev \
@@ -25,11 +26,9 @@ SDKCHROOT_PREINSTALL := "debhelper \
devscripts \
equivs"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
S = "${WORKDIR}/rootfs"

do_build[dirs] = "${DEPLOY_DIR_IMAGE}"
-do_build[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
do_build[root_cleandirs] = "${S} \
${S}/isar-apt"

--
2.20.1

[Andreas Reichel]

maybe first define a variable for ${DISTRO}-${DISTRO_ARCH} and then
prepend this with ${HOST_ARCH}- where needed makes the code more
readable.

> CACHE = "${TMPDIR}/cache"
>
> OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> index 08b068f..6d0f5f5 100644
> --- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> +++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> @@ -6,17 +6,13 @@
> # SPDX-License-Identifier: MIT
>
> Description = "Minimal host Debian root file system"
> +PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
>
> -WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
> -DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> -ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
> +DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
> +ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"
>

Same here

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To post to this group, send email to isar-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20190416132647.3074-5-claudius.heine.ext%40siemens.com.
> For more options, visit https://groups.google.com/d/optout.

--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas...@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

[Andreas Reichel]

On Tue, Apr 16, 2019 at 03:26:44PM +0200, [ext] claudius....@siemens.com wrote:
> From: Claudius Heine <c...@denx.de>
>
> This way recipes that inherit image.bbclass can set the this path

the this

> themselves. This was previously not possible, because the bitbake.conf
> set this variable with '='.
>
> Signed-off-by: Claudius Heine <c...@denx.de>
> ---
> meta/classes/image.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index f9a3052..6881d37 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"
>
> IMAGE_INSTALL ?= ""
> IMAGE_TYPE ?= "ext4-img"
> -IMAGE_ROOTFS = "${WORKDIR}/rootfs"
> +IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"
>
> IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"
>
> --
> 2.20.1
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To post to this group, send email to isar-...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20190416132647.3074-7-claudius.heine.ext%40siemens.com.

[Claudius Heine]

Hi Andreas,

I understand your point, but this is the 'bitbake.conf' so any variable
will be available everywhere. So I would like to avoid creating any
helper variables that are of limited use anywhere else and create mostly
noise.

Also changing the order would lead to more inconsistencies IMO.
Everywhere the 'DISTRO' code word is named first, then followed by the
native architecture of that root file system. This change just adds the
architecture of the target machine to the last place for the
cross-tool-chain.

Thanks,
Claudius

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Hi,

this is the fixed up version.

regards,
Claudius

changes from v1:
- fixed typo in commit message

Claudius Heine (8):
isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
base.bbclass
move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
bitbake.conf: remove unneeded and differently used variables
image.bbclass: make IMAGE_ROOTFS overwritable
bitbake.conf: set default QEMU_ARCH variables
buildchroot/configscript: make creation of builder uid/gid idempotent

meta/classes/base.bbclass | 9 +++++++++
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/image-sdk-extension.bbclass | 2 +-
meta/classes/image.bbclass | 2 +-
meta/classes/isar-bootstrap-helper.bbclass | 14 --------------
meta/conf/bitbake.conf | 13 ++++++++-----

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
.../buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This variable is only used in 'isar-bootstrap-host.bb' and mentioned in
'isar-bootstrap.inc'. So it does not make sense to have it available
anywhere else.

Signed-off-by: Claudius Heine <c...@denx.de>
---

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

IMAGE_CMD and TARGET_ARCH are not used in Isar or any bitbake code.

IMAGE_ROOTFS is used as the directory where image recipes store their
root file system. Pointing that to a global path like '${TMPDIR}/rootfs'
does not make sense for Isar.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 3 ---
1 file changed, 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7c1c89f..21d13ef 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/buildchroot.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a87bf4..ea1538a 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,7 @@ ISAR_CROSS_COMPILE ??= "0"
# Add dependency from the correct buildchroot: host or target
python __anonymous() {
mode = d.getVar('ISAR_CROSS_COMPILE', True)
- if mode == "0":
+ if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
dep = "buildchroot-target:do_build"

rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
else:
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
support was dropped.)

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/base.bbclass | 9 +++++++++
meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 9a1837f..ca80bf8 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -20,6 +20,15 @@

THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"

+def get_deb_host_arch():
+ import subprocess
+ host_arch = subprocess.check_output(
+ ["dpkg", "--print-architecture"]
+ ).decode('utf-8').strip()
+ return host_arch
+HOST_ARCH ??= "${@get_deb_host_arch()}"
+HOST_DISTRO ??= "${DISTRO}"
+
die() {
bbfatal "$*"
}

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 38d90fb..59c447b 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This way recipes that inherit image.bbclass can set this path

themselves. This was previously not possible, because the bitbake.conf
set this variable with '='.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f9a3052..6881d37 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"

IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
-IMAGE_ROOTFS = "${WORKDIR}/rootfs"
+IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"

IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Each of those are dependent on both HOST_ARCH and DISTRO_ARCH, so the
stamps etc. should not be shared.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image-sdk-extension.bbclass | 2 +-
meta/conf/bitbake.conf | 4 ++--

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

meta/recipes-devtools/buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--

meta/recipes-devtools/sdkchroot/sdkchroot.bb | 3 +--
7 files changed, 10 insertions(+), 20 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 1838080..8df3ed1 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,7 +5,7 @@
#
# This class extends the image.bbclass to supply the creation of a sdk

-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}"
+SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..7c1c89f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf

@@ -50,8 +50,8 @@ DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target/rootfs"
+BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"

CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 08b068f..6d0f5f5 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,17 +6,13 @@
# SPDX-License-Identifier: MIT

Description = "Minimal host Debian root file system"
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
-DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
+DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
+ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 21d13ef..37f1963 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -53,6 +53,12 @@ CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

[Maxim Yu. Osipov]

Hi,

Tried the series against the current 'next'.

Creation of SDK has been failed (my host is Debian 9) (see log below).

I've double checked - in the current 'next' 'bitbake -c do_populate_sdk
multiconfig:qemuarm-stretch:isar-image-base' works fine.

My local.conf is attached for convenience.

Maxim.

===============
bitbake -c do_populate_sdk multiconfig:qemuarm-stretch:isar-image-base
Parsing recipes: 100%
|#######################################################################################################################################################################################################|
Time: 0:00:02
Parsing of 26 .bb files complete (0 cached, 26 parsed). 390 targets, 0
skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100%
|####################################################################################################################################################################################################|
Time: 0:00:03
NOTE: Executing RunQueue Tasks
ERROR: mc:qemuarm-stretch:sdkchroot-${ROOTFS_DISTRO}-amd64-armhf
do_build: Function failed: do_build (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/sdkchroot-${ROOTFS_DISTRO}-amd64-armhf/temp/log.do_build.24544)
ERROR: Logfile of failure stored in:
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/sdkchroot-${ROOTFS_DISTRO}-amd64-armhf/temp/log.do_build.24544
Log data follows:
| DEBUG: Executing shell function do_build
| cp: cannot stat
'/home/myo/work/isar/src/trunk/isar/build/tmp/deploy/bootstrap/debian-stretch-amd64/':
No such file or directory
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_build (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/sdkchroot-${ROOTFS_DISTRO}-amd64-armhf/temp/log.do_build.24544)
ERROR: Task
(multiconfig:qemuarm-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-devtools/sdkchroot/sdkchroot.bb:do_build)
failed with exit code '1'
ERROR: mc:qemuarm-stretch:buildchroot-host-debian-stretch-amd64-armhf
do_build: Function failed: do_build (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/buildchroot-host-debian-stretch-amd64-armhf/temp/log.do_build.24543)
ERROR: Logfile of failure stored in:
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/buildchroot-host-debian-stretch-amd64-armhf/temp/log.do_build.24543
Log data follows:
| DEBUG: Executing shell function do_build
| cp: cannot stat
'/home/myo/work/isar/src/trunk/isar/build/tmp/deploy/bootstrap/debian-stretch-amd64/':
No such file or directory
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_build (log file is located at
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-armhf/buildchroot-host-debian-stretch-amd64-armhf/temp/log.do_build.24543)
ERROR: Task
(multiconfig:qemuarm-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-devtools/buildchroot/buildchroot-host.bb:do_build)
failed with exit code '1'

=====

Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mos...@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[Maxim Yu. Osipov]

The same problem with SDK creation discovered in v1 series persists also
in v2 series.

Just recognized that "quick" CI build fails too. See for details:
http://isar-build.org:8080/job/isar_mosipov_develop/91/console

Please run CI scripts before submitting patches to the mailing list -
this will save maintainer's time.

Maxim.

[Claudius Heine]

Hi Maxim,

On 22/04/2019 15.48, Maxim Yu. Osipov wrote:
> The same problem with SDK creation discovered in v1 series persists also
> in v2 series.
>
> Just recognized that "quick" CI build fails too. See for details:
> http://isar-build.org:8080/job/isar_mosipov_develop/91/console
>
> Please run CI scripts before submitting patches to the mailing list -

I did:

http://isar-build.org:8080/job/isar_claudius_ilbers-ci/46/consoleText

That build just fails because of the qemu boot where loading of modules
fails for some reason, but you said that could be ignored.

> this will save maintainer's time.

Well if you are short on time, then I would suggest looking into better
tooling.

There are solutions that could automate most of the job you do as a
maintainer, meaning loading patchset from mailinglist, processing it in
CI and reporting results back to the mailinglist.

Probably something like this:
https://that.guru/blog/patchwork-and-ci-in-a-tree/

As I suggested months ago that patchwork can help safe time and avoid
leaving any patches behind. If you don't look into solutions like that
or write your own scripting, then I don't see much reason for you to
complain.

By doing that, you might even have enough time to start doing code
reviews and taking part in design discussions etc. where it is much
better invested that just pushing the "Run CI build" button and waiting
for that to finish.

To be fair I might do you injustice and you are performing thorough
review of every ones patches and every ones code is just that excellent
that only grammar and spelling mistakes in commit messages are left to
critic, but then it might just be a matter of improving communication.

Claudius

[Claudius Heine]

Hi Maxim,

On 22/04/2019 15.28, Maxim Yu. Osipov wrote:
> Hi,
>
> Tried the series against the current 'next'.
>
> Creation of SDK has been failed (my host is Debian 9) (see log below).
>
> I've double checked - in the current 'next' 'bitbake -c do_populate_sdk
> multiconfig:qemuarm-stretch:isar-image-base' works fine.

Thanks! That is a good find. That issue happened because I tried to
split up the pre-processing patchset.

Is that tested in CI?

Claudius

[Maxim Yu. Osipov]

Hi Claudius,

On 4/24/19 9:20 AM, Claudius Heine wrote:
> Hi Maxim,
>
> On 22/04/2019 15.28, Maxim Yu. Osipov wrote:
>> Hi,
>>
>> Tried the series against the current 'next'.
>>
>> Creation of SDK has been failed (my host is Debian 9) (see log below).
>>
>> I've double checked - in the current 'next' 'bitbake -c
>> do_populate_sdk multiconfig:qemuarm-stretch:isar-image-base' works fine.
>
> Thanks! That is a good find. That issue happened because I tried to
> split up the pre-processing patchset.
>
> Is that tested in CI?

Unfortunately at the moment no - I'll include it into CI.

Maxim.

[Maxim Yu. Osipov]

Hi Claudius,

On 4/24/19 8:53 AM, Claudius Heine wrote:
> Hi Maxim,
>
> On 22/04/2019 15.48, Maxim Yu. Osipov wrote:
>> The same problem with SDK creation discovered in v1 series persists
>> also in v2 series.
>>
>> Just recognized that "quick" CI build fails too. See for details:
>> http://isar-build.org:8080/job/isar_mosipov_develop/91/console
>>
>> Please run CI scripts before submitting patches to the mailing list -
>
> I did:
>
> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/46/consoleText
>
> That build just fails because of the qemu boot where loading of modules
> fails for some reason, but you said that could be ignored.

Well, I said that one may ignore problems with checking login prompt (CI
tests need to be improved at this point), but I don't remember that I
said that one may ignore problems with modules loading.

Actually, the problem with loading modules persists in the current
'next' starting from the build #291 (when I applied your patch set
Implement ROOTFS postprocess commands)
http://isar-build.org:8080/job/isar_next/291/console

I'm investigating the reason now - from the first glance the ARM stretch
kernel version has been changed from 4.9.0-8 to 4.19.0.4.

>
>> this will save maintainer's time.
>
> Well if you are short on time, then I would suggest looking into better
> tooling.
>
> There are solutions that could automate most of the job you do as a
> maintainer, meaning loading patchset from mailinglist, processing it in
> CI and reporting results back to the mailinglist.
>
> Probably something like this:
> https://that.guru/blog/patchwork-and-ci-in-a-tree/
>
> As I suggested months ago that patchwork can help safe time and avoid
> leaving any patches behind. If you don't look into solutions like that
> or write your own scripting, then I don't see much reason for you to
> complain.


> By doing that, you might even have enough time to start doing code
> reviews and taking part in design discussions etc. where it is much
> better invested that just pushing the "Run CI build" button and waiting
> for that to finish.


> To be fair I might do you injustice and you are performing thorough
> review of every ones patches and every ones code is just that excellent
> that only grammar and spelling mistakes in commit messages are left to
> critic, but then it might just be a matter of improving communication.

Thanks for the link/your opinion.

Maxim.

[Claudius Heine]

Hi Maxim,

On 24/04/2019 10.00, Maxim Yu. Osipov wrote:
> Hi Claudius,
>
> On 4/24/19 8:53 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 22/04/2019 15.48, Maxim Yu. Osipov wrote:
>>> The same problem with SDK creation discovered in v1 series persists
>>> also in v2 series.
>>>
>>> Just recognized that "quick" CI build fails too. See for details:
>>> http://isar-build.org:8080/job/isar_mosipov_develop/91/console
>>>
>>> Please run CI scripts before submitting patches to the mailing list -
>>
>> I did:
>>
>> http://isar-build.org:8080/job/isar_claudius_ilbers-ci/46/consoleText
>>
>> That build just fails because of the qemu boot where loading of
>> modules fails for some reason, but you said that could be ignored.
>
> Well, I said that one may ignore problems with checking login prompt (CI
> tests need to be improved at this point), but I don't remember that I
> said that one may ignore problems with modules loading.

I thought you meant that issue, since that occurred on my
post-processing patchset on my CI as well and you reviewed and merged
that. It did not happen on your CI run as far as I could see so I
thought it might be just something that occurred when I try to build
something on that machine. (different user account, priorities, what ever)

Well those two problems occur pretty intermittent. From what I have seen
there is no real pattern there. I also tried booting the resulting image
in qemu at one point locally as well and I could load the module from
the command line or was already loaded IIRC, so I thought it could be a
timing issue.

Do we have a different CI configuration? If that is the case can you set
my CI configuration to the one used when building next?

Claudius

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/buildchroot.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a87bf4..ea1538a 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,7 @@ ISAR_CROSS_COMPILE ??= "0"
# Add dependency from the correct buildchroot: host or target
python __anonymous() {
mode = d.getVar('ISAR_CROSS_COMPILE', True)
- if mode == "0":
+ if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
dep = "buildchroot-target:do_build"

rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
else:
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
support was dropped.)

Signed-off-by: Claudius Heine <c...@denx.de>
---

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Each of those are dependent on both HOST_ARCH and DISTRO_ARCH, so the
stamps etc. should not be shared.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image-sdk-extension.bbclass | 2 +-
meta/conf/bitbake.conf | 4 ++--

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

meta/recipes-devtools/buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--

meta/recipes-devtools/sdkchroot/sdkchroot.bb | 3 +--
7 files changed, 10 insertions(+), 20 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 1838080..8df3ed1 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,7 +5,7 @@
#
# This class extends the image.bbclass to supply the creation of a sdk

-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}"
+SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..7c1c89f 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf

@@ -50,8 +50,8 @@ DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target/rootfs"
+BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"

CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

index 895a454..822d4bb 100644

--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -14,6 +14,7 @@ SRC_URI = " \
PV = "0.1"

inherit isar-bootstrap-helper

+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This variable is only used in 'isar-bootstrap-host.bb' and mentioned in
'isar-bootstrap.inc'. So it does not make sense to have it available
anywhere else.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/isar-bootstrap-helper.bbclass | 2 --
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 59c447b..0d77929 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

IMAGE_CMD and TARGET_ARCH are not used in Isar or any bitbake code.

IMAGE_ROOTFS is used as the directory where image recipes store their
root file system. Pointing that to a global path like '${TMPDIR}/rootfs'
does not make sense for Isar.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 3 ---
1 file changed, 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 7c1c89f..21d13ef 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This way recipes that inherit image.bbclass can set this path
themselves. This was previously not possible, because the bitbake.conf
set this variable with '='.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f9a3052..6881d37 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"

IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
-IMAGE_ROOTFS = "${WORKDIR}/rootfs"
+IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"

IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 21d13ef..37f1963 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -53,6 +53,12 @@ CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Hi,

I could not reproduce the raspberrypi issue Maxim found in his CI build:
http://isar-build.org:8080/job/isar_mosipov_develop/91/consoleText

This is my build log:
http://isar-build.org:8080/job/isar_claudius_ilbers-ci/54/consoleText

Only the module loading fails, which Maxim stated he will look into.

This patchset uses still the 'old' style for refactoring/cleanup patchsets,
until I automated the official patch submission process as documented by the
CONTRIBUTERS policy.

You can find the whole patchset in under this url:

https://github.com/cmhe/isar/tree/ch/cleanup

regards,
Claudius

changes from v2:
- replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
populate_sdk issue

changes from v1:
- fixed typo in commit message

Claudius Heine (8):
isar-boostrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to
base.bbclass
move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH
isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR
bitbake.conf: remove unneeded and differently used variables
image.bbclass: make IMAGE_ROOTFS overwritable
bitbake.conf: set default QEMU_ARCH variables
buildchroot/configscript: make creation of builder uid/gid idempotent

meta/classes/base.bbclass | 9 +++++++++
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/image-sdk-extension.bbclass | 2 +-
meta/classes/image.bbclass | 2 +-
meta/classes/isar-bootstrap-helper.bbclass | 14 --------------
meta/conf/bitbake.conf | 13 ++++++++-----

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

.../recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
.../buildchroot/buildchroot-host.bb | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--

.../buildchroot/files/configscript.sh | 4 ++--
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 3 +--
13 files changed, 30 insertions(+), 41 deletions(-)

--

2.20.1

[Maxim Yu. Osipov]

Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
similar problems as before.
http://isar-build.org:8080/job/isar_mosipov_develop/98/console

I observe the same errors on my host.

Maxim.

[Claudius Heine]

Hi Maxim,

On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>
> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
> similar problems as before.
> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>
> I observe the same errors on my host.

So what do we do then when the CI is not reproducible?

I am at a loss at how to investigate an error that does not happen when
I build on isar-build or on my local host and only when you build it on
isar-build or at your host.

Any suggestions?

Claudius

[Maxim Yu. Osipov]

Hi Claudius,

On 4/25/19 8:40 AM, Claudius Heine wrote:
> Hi Maxim,
>
> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>
>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>> similar problems as before.
>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>
>> I observe the same errors on my host.
>
> So what do we do then when the CI is not reproducible?
>
> I am at a loss at how to investigate an error that does not happen when
> I build on isar-build or on my local host and only when you build it on
> isar-build or at your host.

Do you mean that this problem is not reproduced when you run "fast" CI
build (ci_build.sh -q -f) on your PC?

> Any suggestions?

Yes, sure. We will create an additional project for you on
isar-build.org to launch "fast" CI build.

Maxim.

[Claudius Heine]

Hi Maxim,

On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
> Hi Claudius,
>
> On 4/25/19 8:40 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>
>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>> similar problems as before.
>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>
>>> I observe the same errors on my host.
>>
>> So what do we do then when the CI is not reproducible?
>>
>> I am at a loss at how to investigate an error that does not happen
>> when I build on isar-build or on my local host and only when you build
>> it on isar-build or at your host.
>
> Do you mean that this problem is not reproduced when you run "fast" CI
> build (ci_build.sh -q -f) on your PC?

I don't do ci_builds locally, because that is the job of the CI. I just
build one target that might be affected of the specific change.

>
>> Any suggestions?
>
> Yes, sure. We will create an additional project for you on
> isar-build.org to launch "fast" CI build.

So you suggest that I should now test every patch I send two times, once
for each different CI configuration manually?

Claudius

[Maxim Yu. Osipov]

Hi Claudius,

On 4/25/19 10:01 AM, Claudius Heine wrote:
> Hi Maxim,
>
> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>> Hi Maxim,
>>>
>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>
>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>> similar problems as before.
>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>
>>>> I observe the same errors on my host.
>>>
>>> So what do we do then when the CI is not reproducible?
>>>
>>> I am at a loss at how to investigate an error that does not happen
>>> when I build on isar-build or on my local host and only when you
>>> build it on isar-build or at your host.
>>
>> Do you mean that this problem is not reproduced when you run "fast" CI
>> build (ci_build.sh -q -f) on your PC?
>
> I don't do ci_builds locally, because that is the job of the CI. I just
> build one target that might be affected of the specific change.

That's the reason why you can't reproduce the problem.

>>
>>> Any suggestions?
>>
>> Yes, sure. We will create an additional project for you on
>> isar-build.org to launch "fast" CI build.
>
> So you suggest that I should now test every patch I send two times, once
> for each different CI configuration manually?

There is another option. I may reconfigure your current project to fast
build. Should I do that?

Maxim.

[Claudius Heine]

On 25/04/2019 10.07, Maxim Yu. Osipov wrote:
> Hi Claudius,
>
> On 4/25/19 10:01 AM, Claudius Heine wrote:
>> Hi Maxim,
>>
>> On 25/04/2019 09.55, Maxim Yu. Osipov wrote:
>>> Hi Claudius,
>>>
>>> On 4/25/19 8:40 AM, Claudius Heine wrote:
>>>> Hi Maxim,
>>>>
>>>> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>>>>
>>>>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>>>>> similar problems as before.
>>>>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>>>>
>>>>> I observe the same errors on my host.
>>>>
>>>> So what do we do then when the CI is not reproducible?
>>>>
>>>> I am at a loss at how to investigate an error that does not happen
>>>> when I build on isar-build or on my local host and only when you
>>>> build it on isar-build or at your host.
>>>
>>> Do you mean that this problem is not reproduced when you run "fast"
>>> CI build (ci_build.sh -q -f) on your PC?
>>
>> I don't do ci_builds locally, because that is the job of the CI. I
>> just build one target that might be affected of the specific change.
>
> That's the reason why you can't reproduce the problem.

Well I did build the multiconfig+target where the issue occured IIRC.

>
>>>
>>>> Any suggestions?
>>>
>>> Yes, sure. We will create an additional project for you on
>>> isar-build.org to launch "fast" CI build.
>>
>> So you suggest that I should now test every patch I send two times,
>> once for each different CI configuration manually?
>
> There is another option. I may reconfigure your current project to fast
> build. Should I do that?

I don't care. I would like to have my CI configured in a way that allows
my patches to tested for upstream inclusion. If that is in a "fast",
"slow" or "totally unnoteworthy" velocity does not matter for that. CI
is an acceptability test and I want the strictest one. If I was mistaken
that "fast" is less strict than "slow", then switch to "fast". If I now
need to pass "fast" and "slow" for my patch to be acceptable, then I
would suggest introducing a "abysmal slow" CI run that combines "slow"
and "fast".

Claudius

[Maxim Yu. Osipov]

I've just sent the CI tests procedure in separate email.

The problem that if we include all the features into single CI run it
will takes hours to test single patch.

Anyway I pointed you to the problem and the way how to reproduce it.
You wrote that you don't want to call CI locally. It's up to you - I
don't care too - if the problem is not fixed the patchset will not be
applied.

I didn't understood from your emotional passage what configuration you
prefer.

I suggested two options

1) Add additional project for you for "fast" build. Yes. In such cases
you have to launch the tests in both projects but you will be sure that
integration will go more smoothly.

2) Reconfigure your existing project from "standard" to the "fast" one.

It is up to you to decide how you want to test your patch set.

Greetings,

[Jan Kiszka]

On 25.04.19 08:40, [ext] Claudius Heine wrote:
> Hi Maxim,
>
> On 25/04/2019 08.03, Maxim Yu. Osipov wrote:
>>
>> Tried this series with fast ci build ("scripts/ci_build.sh -q -f"-
>> similar problems as before.
>> http://isar-build.org:8080/job/isar_mosipov_develop/98/console
>>
>> I observe the same errors on my host.
>
> So what do we do then when the CI is not reproducible?

Then you likely see a race that you may only trigger by running sufficient
builds in parallel, and maybe only with the "right" number of cores. Otherwise,
you need to do offline analysis or push instrumented branches into the same CI
system

If you take the time to look at the logs, you will likely quickly spot that
there is some issue with mounting /dev and possibly also /dev/pts. Dig into your
changes /wrt what may have created a race here - or started to expose a sleeping
one.

In any case, we need to make CI a community resource. We will come up with a
proposal soon.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Claudius Heine]

I looked into the "fast" build now, and it cross-compiles
"multiconfig:rpi-stretch:isar-image-base". Now I tried that and the
build fails even on the current 'next' so the issue seems to be there.

>>
>>>
>>>>>
>>>>>> Any suggestions?
>>>>>
>>>>> Yes, sure. We will create an additional project for you on
>>>>> isar-build.org to launch "fast" CI build.
>>>>
>>>> So you suggest that I should now test every patch I send two times,
>>>> once for each different CI configuration manually?
>>>
>>> There is another option. I may reconfigure your current project to
>>> fast build. Should I do that?
>>
>> I don't care. I would like to have my CI configured in a way that
>> allows my patches to tested for upstream inclusion. If that is in a
>> "fast", "slow" or "totally unnoteworthy" velocity does not matter for
>> that. CI is an acceptability test and I want the strictest one. If I
>> was mistaken that "fast" is less strict than "slow", then switch to
>> "fast". If I now need to pass "fast" and "slow" for my patch to be
>> acceptable, then I would suggest introducing a "abysmal slow" CI run
>> that combines "slow" and "fast".
>
> I've just sent the CI tests procedure in separate email.

That is not a CI test procedure. That is a test procedure for local
testing, if someone wants to use the ci scripts to do so. I don't since
my environment is different from what you have on your CI server.

> The problem that if we include all the features into single CI run it
> will takes hours to test single patch.

Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that
everyone should do that anyway. Manually.

>
> Anyway I pointed you to the problem and the way how to reproduce it.
> You wrote that you don't want to call CI locally. It's up to you - I
> don't care too - if the problem is not fixed the patchset will not be
> applied.

Well I don't have the same environment as you on your CI server, so
running scripts build for your CI server is not something I can do. That
is not choice that is just how things are.

> I didn't understood from your emotional passage what configuration you
> prefer.

Well CI testing should be done reproducible and transparent. If my
branch on jenkins has a different configuration than some other branch
there and now the invisible configuration of those branches changes like
the weather, then this is not reproducible nor transparent.

Having to explain the basics of what CI is, is pretty nerve wreaking and
exhausting.

> I  suggested two options
>
> 1) Add additional project for you for "fast" build. Yes. In such cases
> you have to launch the tests in both projects but you will be sure that
> integration will go more smoothly.

Well the most acceptable way would be 3 branches. One for "fast", one
for "slow" and one for "acceptance" (which combines slow and fast).

>
> 2) Reconfigure your existing project from "standard" to the "fast" one.
>
> It is up to you to decide how you want to test your patch set.

No its not. It is up to you to test the patchset and decide if that
should be merged or not. So contributors have to jump through those
hoops in order for it to be merged reasonable fast.

Claudius

[Maxim Yu. Osipov]

No. "Fast" CI build (which in particular tests cross-compilation) in
'next' is OK - I've built it today morning when I tested your patch
"ubi-img/ubifs-img: change fatal error to skip recipe" and my patch
which includes SDK cretion test into "fast" build.

You may look at the log:
http://isar-build.org:8080/job/isar_mosipov_develop/97/console

>>>
>>>>
>>>>>>
>>>>>>> Any suggestions?
>>>>>>
>>>>>> Yes, sure. We will create an additional project for you on
>>>>>> isar-build.org to launch "fast" CI build.
>>>>>
>>>>> So you suggest that I should now test every patch I send two times,
>>>>> once for each different CI configuration manually?
>>>>
>>>> There is another option. I may reconfigure your current project to
>>>> fast build. Should I do that?
>>>
>>> I don't care. I would like to have my CI configured in a way that
>>> allows my patches to tested for upstream inclusion. If that is in a
>>> "fast", "slow" or "totally unnoteworthy" velocity does not matter for
>>> that. CI is an acceptability test and I want the strictest one. If I
>>> was mistaken that "fast" is less strict than "slow", then switch to
>>> "fast". If I now need to pass "fast" and "slow" for my patch to be
>>> acceptable, then I would suggest introducing a "abysmal slow" CI run
>>> that combines "slow" and "fast".
>>
>> I've just sent the CI tests procedure in separate email.
>
> That is not a CI test procedure. That is a test procedure for local
> testing, if someone wants to use the ci scripts to do so. I don't since
> my environment is different from what you have on your CI server.
>
>> The problem that if we include all the features into single CI run it
>> will takes hours to test single patch.
>
> Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that
> everyone should do that anyway. Manually.

It's highly suggested but its' not a must.
Current iteration of patch series just proves that it's worth to run
these scripts if you want to avoid multiple patches versions iterations.

>
>>
>> Anyway I pointed you to the problem and the way how to reproduce it.
>> You wrote that you don't want to call CI locally. It's up to you - I
>> don't care too - if the problem is not fixed the patchset will not be
>> applied.
>
> Well I don't have the same environment as you on your CI server, so
> running scripts build for your CI server is not something I can do. That
> is not choice that is just how things are.
>
>> I didn't understood from your emotional passage what configuration you
>> prefer.
>
> Well CI testing should be done reproducible and transparent. If my
> branch on jenkins has a different configuration than some other branch
> there and now the invisible configuration of those branches changes like
> the weather, then this is not reproducible nor transparent.
>
> Having to explain the basics of what CI is, is pretty nerve wreaking and
> exhausting.

Thank you for explaining what CI is - I'm sorry to make you nervous and
exhausted.

>> I  suggested two options
>>
>> 1) Add additional project for you for "fast" build. Yes. In such cases
>> you have to launch the tests in both projects but you will be sure that
>> integration will go more smoothly.
>
> Well the most acceptable way would be 3 branches. One for "fast", one
> for "slow" and one for "acceptance" (which combines slow and fast).
>
>>
>> 2) Reconfigure your existing project from "standard" to the "fast" one.
>>
>> It is up to you to decide how you want to test your patch set.
>
> No its not. It is up to you to test the patchset and decide if that
> should be merged or not. So contributors have to jump through those
> hoops in order for it to be merged reasonable fast.

And I decided to don't merge your patch set because tests were failed.
That's the only story.

Maxim.

[Baurzhan Ismagulov]

On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:
> That is not a CI test procedure. That is a test procedure for local testing,
> if someone wants to use the ci scripts to do so. I don't since my
> environment is different from what you have on your CI server.

The CI server doesn't have any magic environment. It's set up according to
doc/user_manual.md#getting-started. scripts/ci_build.sh and
scripts/vm_smoke_test are expected to be able to run locally or on the server.
Of course, you can run any part of that manually if you are not comfortable
running the whole. This is the motivation of moving to Avocado -- to break the
big script into more maintainable and short pieces, so one could directly see
which particular test case has failed and restart just that one.

> > The problem that if we include all the features into single CI run it
> > will takes hours to test single patch.
>
> Right. "abysmal slow". But in your 'CONTRIBUTERS' you state that everyone
> should do that anyway. Manually.

This is what the maintainer does for every series. Isar supports many features
on a number of platforms; testing all does cost time. We don't ask that every
developer does that routinely (though that is in your interest to get the patch
applied quickly); responding to a specific failure is enough.


With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

On Thu, Apr 25, 2019 at 11:11:21AM +0200, Claudius Heine wrote:

> I looked into the "fast" build now, and it cross-compiles
> "multiconfig:rpi-stretch:isar-image-base". Now I tried that and the build
> fails even on the current 'next' so the issue seems to be there.

How does the problem manifest itself? Please provide some relevant details
(e.g., bitbake output, failure logs, etc.).

With kind regards,
Baurzhan.

[Claudius Heine]

Hi Baurzhan,

Commit ID: 2396d4f44fe1f867ddae389b61cb7017cb2b908c

--- local.conf ---
# cross
ISAR_CROSS_COMPILE = "1"

# standard
CONF_VERSION = "1"

MACHINE ??= "qemu"
DISTRO ??= "raspbian-stretch"
BBMULTICONFIG ?= "rpi-stretch"
--- end local.conf ---

bitbake -k -c build multiconfig:rpi-stretch:isar-image-base

--- error log ---
ERROR: mc:rpi-stretch:isar-bootstrap-host-1.0-r0 do_bootstrap: Function
failed: do_bootstrap (log file is located at
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078)

ERROR: Logfile of failure stored in:

/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| W: Target architecture is the same as host architecture; disabling
QEMU support
| I: Running command: debootstrap --arch amd64 --verbose
--variant=minbase --include=locales --components=main,contrib,non-free
--keyring
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg
stretch
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs
http://ftp.debian.org/debian
| I: Retrieving InRelease
| I: Retrieving Release
| I: Retrieving Release.gpg
| I: Checking Release signature
| E: Release signed by unknown key (key id EF0F382A1A7B6500)
| WARNING:
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/run.do_bootstrap.27078:1
exit 1 from 'sudo -E -s <<'EOSUDO'
| ( flock 9
| set -e
| if [ ! -e
"/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64"
]; then
| rm -rf
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
|
| if [ ${IS_HOST} ]; then
| qemu-debootstrap $debootstrap_args \
| --components=main,contrib,non-free \
| --keyring
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg
\
| "stretch" \
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
\
| "http://ftp.debian.org/debian"
|
| else
| "qemu-debootstrap" $debootstrap_args \
| --arch="armhf" \
|
--components=main,contrib,non-free,firmware \
| --keyring
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-keyring.gpg
\
| "stretch" \
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
\
| "http://archive.raspbian.org/raspbian"
| fi
|
| # Install apt config
| mkdir -p
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d"
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-preferences"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/preferences.d/bootstrap"
| mkdir -p
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d"
| if [ "${ISAR_USE_CACHED_BASE_REPO}" = "1" ]; then
| sed -e "s#{BASE_DISTRO}#"raspbian"#g" \
| -e "s#{BASE_DISTRO_CODENAME}#"stretch"#g" \
| -i
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources
| mkdir -p
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt
| sudo mount --bind
/mnt/ssd/isar/upstream/build/downloads/base-apt/apt
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/base-apt-sources"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/base-apt.list"
| else
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list.d/bootstrap.list"
| fi
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/apt-sources-init"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources-list"
| rm -f
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/sources.list"
| mkdir -p
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d"
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt.conf"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/50isar.conf"
|
| if [ "stretch" = "stretch" ] && [ "4" -lt "4" ]; then
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/isar-apt-fallback.conf"
\
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/apt/apt.conf.d/55isar-fallback.conf"
| fi
|
| # Set locale
| install -v -m644
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/locale"
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale"
|
| sed -i '/en_US.UTF-8 UTF-8/s/^#//g'
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/etc/locale.gen"
| chroot
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
/usr/sbin/locale-gen
|
| # setup chroot
| install -v -m755
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/chroot-setup.sh"
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh"
|
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/chroot-setup.sh"
"setup"
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
|
| # update APT
| mount --rbind /dev
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev
| mount --make-rslave
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev
| mount -t proc none
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc
| mount --rbind /sys
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys
| mount --make-rslave
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys
|
| export DEBIAN_FRONTEND=noninteractive
| chroot
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
/usr/bin/apt-get update -y
| chroot
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
/usr/bin/apt-get dist-upgrade -y \
| -o Debug::pkgProblemResolver=yes
|
| umount -l
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/dev"
| umount -l
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/proc"
| umount -l
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/sys"
| umount -l
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs/base-apt"
|| true
|
| # Finalize debootstrap by setting the link in deploy
| ln -Tfsr
"/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/rootfs"
"/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64"
| fi
| )
9>'/mnt/ssd/isar/upstream/build/tmp/deploy/bootstrap/debian-stretch-amd64.lock'
| EOSUDO
| '
| ERROR: Function failed: do_bootstrap (log file is located at
/mnt/ssd/isar/upstream/build/tmp/work/raspbian-stretch-armhf/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.27078)
ERROR: Task
(multiconfig:rpi-stretch:/mnt/data/isar/layers/isar-master/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap)

failed with exit code '1'

--- end error log ---

So pretty much the same error I currently deal with.

Claudius

[Claudius Heine]

Just some speculation here. I find it strange that a normal debian
stretch bootstrap gets a '--keyring' parameter in that case. What is
bootstrapped here is a amd64 rootfs and the keyring contains the one for
raspbian.

But I don't know why this causes an error for me here, but not on the
CI. But only on the CI when I add my cleanup patches.

Claudius

[Claudius Heine]

Hi,

On 24/04/2019 11.27, [ext] claudius....@siemens.com wrote:
> From: Claudius Heine <c...@denx.de>
>
> This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
> support was dropped.)
>
> Signed-off-by: Claudius Heine <c...@denx.de>
> ---
> meta/classes/base.bbclass | 9 +++++++++
> meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
> 2 files changed, 9 insertions(+), 12 deletions(-)
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 9a1837f..ca80bf8 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -20,6 +20,15 @@
>
> THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"
>
> +def get_deb_host_arch():
> + import subprocess
> + host_arch = subprocess.check_output(
> + ["dpkg", "--print-architecture"]
> + ).decode('utf-8').strip()
> + return host_arch
> +HOST_ARCH ??= "${@get_deb_host_arch()}"
> +HOST_DISTRO ??= "${DISTRO}"

I did find in issue here when debugging.
In case of raspbian-stretch HOST_DISTRO needs to be 'debian-stretch'.

I corrected that for the next version of this patchset.

Currently the same issue in CI issue now reoccurs with the
'isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR' patch
though.

I am currently investigating that.

Claudius

> +
> die() {
> bbfatal "$*"
> }
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
> index 38d90fb..59c447b 100644
> --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -7,18 +7,6 @@
>
> IMAGE_TRANSIENT_PACKAGES ??= ""
>
> -def get_deb_host_arch():
> - import subprocess
> - host_arch = subprocess.check_output(
> - ["dpkg", "--print-architecture"]
> - ).decode('utf-8').strip()
> - return host_arch
> -
> -#Debian Distribution for SDK host
> -HOST_DISTRO ?= "debian-stretch"
> -#Determine SDK host architecture if not explicitly set
> -HOST_ARCH ?= "${@get_deb_host_arch()}"
> -
> HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
>
> def reverse_bb_array(d, varname):
>

--

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Hi,

so after some debugging 'fast' build now succeeds:
http://isar-build.org:8080/job/isar_claudius_ilbers_fast_ci/15/

I needed a patch that removes the DISTRO_APT_KEYS from the isar-bootstrap-host
as well as some fixes to the paths when cross-compiling. Also raspbian only
supports debian-stretch as HOST_DISTRO.

Well the lessons learned from this is that the 'standard' CI build does not
test cross-compiling as I would have assumed and that there is a 'fast' CI
build configuration that tests this.

Also some investigation into why rpi-stretch:isar-bootstrap-host accepted a
keyring with the raspberry pi keys for building a ordinary debian stretch
rootfs at some point needs to happen. There might be a task ordering issue
hidden there.

regards,
Claudius

P.S. The 'standard' CI build for this is still running. I got
'E: Couldn't download packages: libpam0g' while bootstraping, which
sound like a issue that might be fixed by restarting the build. Maybe using
an apt-cacher as proxy might help?


changes from v3:
- added patch for isar-bootstrap-host
- fixed HOST_DISTRO for raspbian-stretch
- fixed cross-compiling paths

changes from v2:
- replaced 'ROOTFS_ARCH' with 'HOST_ARCH' in sdkchroot.bb to avoid
populate_sdk issue

changes from v1:
- fixed typo in commit message

Claudius Heine (9):
isar-bootstrap-host: disable DISTRO_APT_KEYS usage
isar-bootstrap-helper: move 'HOST_ARCH' and 'HOST_DISTRO' to

base.bbclass
move 'HOST_DISTRO_APT_SOURCES' from bootstrap-helper to isar-bootstrap
buildchroot.bbclass: only cross build if HOST_ARCH != DISTRO_ARCH

isar-bootstrap/buildchroot/sdkchroot: refactor PF and WORKDIR

bitbake.conf: remove unneeded and differently used variables
image.bbclass: make IMAGE_ROOTFS overwritable
bitbake.conf: set default QEMU_ARCH variables
buildchroot/configscript: make creation of builder uid/gid idempotent

meta-isar/conf/multiconfig/rpi-stretch.conf | 1 +

meta/classes/base.bbclass | 9 +++++++++
meta/classes/buildchroot.bbclass | 2 +-
meta/classes/image-sdk-extension.bbclass | 2 +-
meta/classes/image.bbclass | 2 +-

meta/classes/isar-bootstrap-helper.bbclass | 18 ++----------------
meta/conf/bitbake.conf | 13 ++++++++-----
.../isar-bootstrap/isar-bootstrap-host.bb | 13 +++++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----
.../isar-bootstrap/isar-bootstrap.inc | 1 +
.../buildchroot/buildchroot-host.bb | 4 ++--
.../buildchroot/buildchroot.inc | 3 +--
.../buildchroot/files/configscript.sh | 4 ++--
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 5 ++---
14 files changed, 37 insertions(+), 46 deletions(-)

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/buildchroot.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/buildchroot.bbclass b/meta/classes/buildchroot.bbclass
index 1a87bf4..ea1538a 100644
--- a/meta/classes/buildchroot.bbclass
+++ b/meta/classes/buildchroot.bbclass
@@ -8,7 +8,7 @@ ISAR_CROSS_COMPILE ??= "0"
# Add dependency from the correct buildchroot: host or target
python __anonymous() {
mode = d.getVar('ISAR_CROSS_COMPILE', True)
- if mode == "0":
+ if mode == "0" or d.getVar('HOST_ARCH') == d.getVar('DISTRO_ARCH'):
dep = "buildchroot-target:do_build"

rootfs = d.getVar('BUILDCHROOT_TARGET_DIR', True)
else:
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 6 ++++++
1 file changed, 6 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf

index dda7bf1..be9c3a8 100644

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This variable is only used in 'isar-bootstrap-host.bb' and mentioned in
'isar-bootstrap.inc'. So it does not make sense to have it available
anywhere else.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/isar-bootstrap-helper.bbclass | 2 --
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 1 +
2 files changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 59c447b..0d77929 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This also sets the 'HOST_DISTRO' to 'DISTRO' per default. (jessie
support was dropped.)

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta-isar/conf/multiconfig/rpi-stretch.conf | 1 +
meta/classes/base.bbclass | 9 +++++++++

meta/classes/isar-bootstrap-helper.bbclass | 12 ------------
3 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/meta-isar/conf/multiconfig/rpi-stretch.conf b/meta-isar/conf/multiconfig/rpi-stretch.conf
index ec1f71d..f3fac38 100644
--- a/meta-isar/conf/multiconfig/rpi-stretch.conf
+++ b/meta-isar/conf/multiconfig/rpi-stretch.conf
@@ -12,6 +12,7 @@ MACHINE ?= "rpi"
# This sets the default distribution configuration:
DISTRO ?= "raspbian-stretch"
DISTRO_ARCH ?= "armhf"
+HOST_DISTRO ?= "debian-stretch"

KERNEL_NAME ?= "rpi-rpfv"

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 9a1837f..ca80bf8 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -20,6 +20,15 @@

THISDIR = "${@os.path.dirname(d.getVar('FILE', True))}"

+def get_deb_host_arch():
+ import subprocess
+ host_arch = subprocess.check_output(
+ ["dpkg", "--print-architecture"]
+ ).decode('utf-8').strip()
+ return host_arch
+HOST_ARCH ??= "${@get_deb_host_arch()}"
+HOST_DISTRO ??= "${DISTRO}"

+
die() {
bbfatal "$*"
}
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 38d90fb..59c447b 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass

@@ -7,18 +7,6 @@

IMAGE_TRANSIENT_PACKAGES ??= ""

-def get_deb_host_arch():
- import subprocess
- host_arch = subprocess.check_output(
- ["dpkg", "--print-architecture"]
- ).decode('utf-8').strip()
- return host_arch
-
-#Debian Distribution for SDK host
-HOST_DISTRO ?= "debian-stretch"
-#Determine SDK host architecture if not explicitly set
-HOST_ARCH ?= "${@get_deb_host_arch()}"
-
HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"

def reverse_bb_array(d, varname):
--

2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

This way recipes that inherit image.bbclass can set this path
themselves. This was previously not possible, because the bitbake.conf
set this variable with '='.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image.bbclass | 2 +-

1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f9a3052..6881d37 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -5,7 +5,7 @@ PF = "${PN}-${DISTRO}-${MACHINE}"

IMAGE_INSTALL ?= ""
IMAGE_TYPE ?= "ext4-img"
-IMAGE_ROOTFS = "${WORKDIR}/rootfs"
+IMAGE_ROOTFS ?= "${WORKDIR}/rootfs"

IMAGE_INSTALL += "${@ ("linux-image-" + d.getVar("KERNEL_NAME", True)) if d.getVar("KERNEL_NAME", True) else ""}"

--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Signed-off-by: Claudius Heine <c...@denx.de>
---

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

Each of those are dependent on both HOST_ARCH and DISTRO_ARCH, so the
stamps etc. should not be shared.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/classes/image-sdk-extension.bbclass | 2 +-
meta/classes/isar-bootstrap-helper.bbclass | 4 ++--
meta/conf/bitbake.conf | 4 ++--

.../isar-bootstrap/isar-bootstrap-host.bb | 11 +++--------
.../isar-bootstrap/isar-bootstrap-target.bb | 6 +-----

meta/recipes-devtools/buildchroot/buildchroot-host.bb | 4 ++--
meta/recipes-devtools/buildchroot/buildchroot.inc | 3 +--
meta/recipes-devtools/sdkchroot/sdkchroot.bb | 5 ++---
8 files changed, 14 insertions(+), 25 deletions(-)

diff --git a/meta/classes/image-sdk-extension.bbclass b/meta/classes/image-sdk-extension.bbclass
index 1838080..8df3ed1 100644
--- a/meta/classes/image-sdk-extension.bbclass
+++ b/meta/classes/image-sdk-extension.bbclass
@@ -5,7 +5,7 @@
#
# This class extends the image.bbclass to supply the creation of a sdk

-SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}"
+SDKCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/sdkchroot-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

do_populate_sdk[stamp-extra-info] = "${DISTRO}-${MACHINE}"
do_populate_sdk[depends] = "sdkchroot:do_build"
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 0d77929..8612be2 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -52,8 +52,8 @@ setup_root_file_system() {
--clean) CLEAN=1 ;;
--copyisarapt) COPYISARAPT=1 ;;
--fstab) FSTAB=$2; shift ;;
- --host-arch) ROOTFS_ARCH=${HOST_ARCH} ;;
- --host-distro) ROOTFS_DISTRO=${HOST_DISTRO} ;;
+ --host-arch) ROOTFS_ARCH="$2-$ROOTFS_ARCH"; shift ;;
+ --host-distro) ROOTFS_DISTRO="$2"; shift ;;
--keep-apt-cache) KEEP_APT_CACHE=1 ;;
-*) bbfatal "$0: invalid option specified: $1" ;;
*) break ;;
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 0e521bb..9eb14ac 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -50,8 +50,8 @@ DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
DEPLOY_DIR_IMAGE = "${DEPLOY_DIR}/images/${MACHINE}"
DL_DIR ?= "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
-BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host/rootfs"
-BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target/rootfs"
+BUILDCHROOT_HOST_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-host-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}/rootfs"
+BUILDCHROOT_TARGET_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot-target-${DISTRO}-${DISTRO_ARCH}/rootfs"

CACHE = "${TMPDIR}/cache"

OVERRIDES = "${DISTRO_ARCH}:${MACHINE}:${DISTRO}:forcevariable"

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index 3e96281..2e2e6d1 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -6,19 +6,15 @@
# SPDX-License-Identifier: MIT

Description = "Minimal host Debian root file system"
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
-DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
-ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
+DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"
+ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc
# We only build debian host buildchroot environments
DISTRO_APT_KEYS = ""
-inherit isar-bootstrap-helper

-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_apt_config_prepare[dirs] = "${WORKDIR}"
do_apt_config_prepare[vardeps] += "\
APTPREFS \
@@ -49,7 +45,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack

OVERRIDES_append = ":${@get_distro_needs_https_support(d, True)}"

-do_bootstrap[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
do_bootstrap[vardeps] += "HOST_DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap --host
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
index 79f3e34..552a317 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb
@@ -6,16 +6,13 @@
# SPDX-License-Identifier: MIT

Description = "Minimal target Debian root file system"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}"
ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${DISTRO}-${DISTRO_ARCH}.lock"

require isar-bootstrap.inc

-do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_apt_config_prepare[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_apt_config_prepare[dirs] = "${WORKDIR}"
do_apt_config_prepare[vardeps] += "\
APTPREFS \
@@ -44,7 +41,6 @@ addtask apt_config_prepare before do_bootstrap after do_unpack

OVERRIDES_append = ":${@get_distro_needs_https_support(d, False)}"

-do_bootstrap[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_bootstrap[vardeps] += "DISTRO_APT_SOURCES"
do_bootstrap() {
isar_bootstrap
diff --git a/meta/recipes-devtools/buildchroot/buildchroot-host.bb b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
index c8c10c0..12a1fdf 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot-host.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot-host.bb
@@ -6,6 +6,7 @@
DESCRIPTION = "Isar development filesystem for host"

require buildchroot.inc
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

BUILDCHROOT_PREINSTALL ?= "make \
debhelper \
@@ -26,6 +27,5 @@ BUILDCHROOT_PREINSTALL_append_armhf += "binutils-arm-linux-gnueabihf \
BUILDCHROOT_PREINSTALL_append_arm64 += "binutils-aarch64-linux-gnu \
crossbuild-essential-arm64"

-
-PARAMS = "--host-arch --host-distro"
+PARAMS = "--host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}'"
do_build[depends] = "isar-apt:do_cache_config isar-bootstrap-host:do_bootstrap"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 764bc5e..cdb4c07 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -12,13 +12,12 @@ SRC_URI = "file://configscript.sh \
file://common.sh \
file://deps.sh"
PV = "1.0"
+PF = "${PN}-${DISTRO}-${DISTRO_ARCH}"

inherit isar-bootstrap-helper

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
BUILDCHROOT_DIR = "${WORKDIR}/rootfs"

-do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
do_build[root_cleandirs] = "${BUILDCHROOT_DIR} \
${BUILDCHROOT_DIR}/isar-apt \
${BUILDCHROOT_DIR}/downloads \
diff --git a/meta/recipes-devtools/sdkchroot/sdkchroot.bb b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
index 895a454..49aeb47 100644
--- a/meta/recipes-devtools/sdkchroot/sdkchroot.bb
+++ b/meta/recipes-devtools/sdkchroot/sdkchroot.bb
@@ -14,6 +14,7 @@ SRC_URI = " \
PV = "0.1"

inherit isar-bootstrap-helper
+PF = "${PN}-${HOST_DISTRO}-${HOST_ARCH}-${DISTRO_ARCH}"

SDKCHROOT_PREINSTALL := "debhelper \
autotools-dev \
@@ -25,11 +26,9 @@ SDKCHROOT_PREINSTALL := "debhelper \
devscripts \
equivs"

-WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_ARCH}"
S = "${WORKDIR}/rootfs"

do_build[dirs] = "${DEPLOY_DIR_IMAGE}"
-do_build[stamp-extra-info] = "${HOST_DISTRO}-${HOST_ARCH}"
do_build[root_cleandirs] = "${S} \
${S}/isar-apt"

@@ -50,7 +49,7 @@ do_build() {
packages="${SDKCHROOT_PREINSTALL} crossbuild-essential-${DISTRO_ARCH}"
fi

- setup_root_file_system --copyisarapt --host-arch --host-distro "${S}" $packages
+ setup_root_file_system --copyisarapt --host-arch '${HOST_ARCH}' --host-distro '${HOST_DISTRO}' "${S}" $packages

# Configure root filesystem
sudo install -m 644 ${WORKDIR}/README.sdk ${S}
--
2.20.1

[claudius....@siemens.com]

From: Claudius Heine <c...@denx.de>

IMAGE_CMD and TARGET_ARCH are not used in Isar or any bitbake code.

IMAGE_ROOTFS is used as the directory where image recipes store their
root file system. Pointing that to a global path like '${TMPDIR}/rootfs'
does not make sense for Isar.

Signed-off-by: Claudius Heine <c...@denx.de>
---

meta/conf/bitbake.conf | 3 ---
1 file changed, 3 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9eb14ac..dda7bf1 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -26,8 +26,6 @@ DEPLOY_DIR = "${TMPDIR}/deploy"
FILESPATH = "${FILE_DIRNAME}/${PF}:${FILE_DIRNAME}/${P}:${FILE_DIRNAME}/${PN}:${FILE_DIRNAME}/files:${FILE_DIRNAME}"
FILE_DIRNAME = "${@os.path.dirname(d.getVar('FILE', False))}"
GITDIR = "${DL_DIR}/git"
-IMAGE_CMD = "_NO_DEFINED_IMAGE_TYPES_"
-IMAGE_ROOTFS = "${TMPDIR}/rootfs"
P = "${PN}-${PV}"
PF = "${PN}-${PV}-${PR}"
PN = "${@bb.parse.BBHandler.vars_from_file(d.getVar('FILE', False),d)[0] or 'defaultpkgname'}"
@@ -39,7 +37,6 @@ SRC_URI = "file://${FILE}"
STAMP = "${TMPDIR}/stamps/${PF}"
SVNDIR = "${DL_DIR}/svn"
T = "${WORKDIR}/temp"
-TARGET_ARCH = "${BUILD_ARCH}"
TMPDIR = "${TOPDIR}/tmp"
PERSISTENT_DIR = "${TMPDIR}/cache"
GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
--
2.20.1