[PATCH 5/6] dpkg-raw: Clean DEBIAN dir prior to filling it

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

When we run unprivileged, writing files in that directory will not work
after the first run because everything is root-owned at the end.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-raw.bbclass | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 44aa078..e3ae9e8 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -16,10 +16,9 @@ do_install() {
do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
addtask install after do_unpack before do_deb_package_prepare

-# so we can put hooks etc. in there already
-do_install[dirs] = "${D}/DEBIAN"
-
do_deb_package_prepare() {
+ sudo rm -rf ${D}/DEBIAN
+ mkdir ${D}/DEBIAN
cat<<-__EOF__ > ${D}/DEBIAN/control
Package: ${PN}
Architecture: ${DISTRO_ARCH}
--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Python code must not contain tabs.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/base.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index ae84677..0fa27db 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -106,8 +106,8 @@ addtask unpack after do_fetch before do_build
addtask build
do_build[dirs] = "${TOPDIR}"
python base_do_build () {
- bb.note("The included, default BB base.bbclass does not define a useful default task.")
- bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
+ bb.note("The included, default BB base.bbclass does not define a useful default task.")
+ bb.note("Try running the 'listtasks' task against a .bb to see what tasks are defined.")
}

EXPORT_FUNCTIONS do_clean do_mrproper do_build
--
2.13.6

[Jan Kiszka]

Yeah, finally Yocto/OE-like usability: This ensures for many cases that
changes to recipes lead to rebuilds of dependent recipes, including the
final image. Some extra measures are needed so that those rebuilds work
with clean dirs.

And if the change detection should not work, e.g. changes to file://
resources are not detected, then a clean or cleanall task is now
available and ensures a proper manual rebuild.

This massively increases the fun factor when developing Isar projects.

Jan

Jan Kiszka (6):
Fix indention of base_do_build
Add clean and cleanall tasks
Enable recipe caching
Enable proper rebuilds on dependency changes
dpkg-raw: Clean DEBIAN dir prior to filling it
isar-image-base: Clean rootfs folder prior to building

meta-isar/recipes-core/images/isar-image-base.bb | 2 ++
meta/classes/base.bbclass | 39 ++++++++++++++++++++++--
meta/classes/dpkg-base.bbclass | 14 +++++++++
meta/classes/dpkg-raw.bbclass | 5 ++-
meta/conf/isar-bitbake.conf | 18 +++++++++--
5 files changed, 70 insertions(+), 8 deletions(-)

--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Import BB_HASHBASE_WHITELIST and BB_HASHCONFIG_WHITELIST from OE,
removing some items that are surely OE-specific, and set CACHE. That's
all what's needed to enable bitbake's recipe parsing cache.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/conf/isar-bitbake.conf | 15 +++++++++++++++
1 file changed, 15 insertions(+)

diff --git a/meta/conf/isar-bitbake.conf b/meta/conf/isar-bitbake.conf
index f7f751c..660e0c0 100644
--- a/meta/conf/isar-bitbake.conf
+++ b/meta/conf/isar-bitbake.conf
@@ -22,6 +22,21 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PF}"
DL_DIR = "${TOPDIR}/downloads"
SSTATE_DIR ?= "${TMPDIR}/sstate-cache"
BUILDCHROOT_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/rootfs"
+CACHE = "${TMPDIR}/cache"
+
+BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH BBSERVER DL_DIR \
+ SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM \
+ USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST \
+ PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE \
+ CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE CCACHE_NOHASHDIR LICENSE_PATH SDKPKGSUFFIX \
+ WORKDIR STAMPCLEAN PKGDATA_DIR BUILD_ARCH SSTATE_PKGARCH \
+ BB_WORKERCONTEXT BB_LIMITEDDEPS DEPLOY_DIR"
+BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SSH_AGENT_PID \
+ SSH_AUTH_SOCK PSEUDO_BUILD BB_ENV_EXTRAWHITE DISABLE_SANITY_CHECKS \
+ BB_NUMBER_THREADS BB_ORIGENV BB_INVALIDCONF BBINCLUDED \
+ GIT_PROXY_COMMAND ALL_PROXY all_proxy NO_PROXY no_proxy FTP_PROXY ftp_proxy \
+ HTTP_PROXY http_proxy HTTPS_PROXY https_proxy SOCKS5_USER SOCKS5_PASSWD \
+ BB_SETSCENE_ENFORCE BB_CMDLINE BB_SERVER_TIMEOUT"

# Setup our default hash policy
BB_SIGNATURE_HANDLER ?= "noop"
--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This ensures we have a clean base when rerunning the task.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta-isar/recipes-core/images/isar-image-base.bb | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 2d16f8f..3d441cc 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -33,6 +33,8 @@ do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
do_rootfs() {
E="${@ bb.utils.export_proxies(d)}"

+ sudo rm -rf ${IMAGE_ROOTFS}
+
chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
chmod +x "${WORKDIR}/setup.sh"
install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"
--
2.13.6

[Jan Kiszka]

On 2018-02-12 08:59, [ext] Jan Kiszka wrote:
> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
>
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
>
> This massively increases the fun factor when developing Isar projects.
>

BTW, this series is independent of my kernel work. I just reordered both
in my internal queue so that the rework of the latter is more convenient
thanks to this one. Feel free to pull it first, unless you have remarks.

Jan

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Alexander Smirnov]

On 02/12/2018 10:59 AM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> This ensures we have a clean base when rerunning the task.
>

How do you do this?

$ bitbake -C do_rootfs ...?

If so, is it correct way to do this, I mean without do_clean before?

Alex

[Alexander Smirnov]

On 02/12/2018 10:59 AM, Jan Kiszka wrote:

> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
>
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
>
> This massively increases the fun factor when developing Isar projects.
>

Got error during build:

$ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
multiconfig:qemuarm-jessie:isar-image-base
multiconfig:qemuarm-stretch:isar-image-base
multiconfig:qemui386-jessie:isar-image-base
multiconfig:qemui386-stretch:isar-image-base
multiconfig:qemuamd64-jessie:isar-image-base
multiconfig:qemuamd64-stretch:isar-image-base
multiconfig:rpi-jessie:isar-image-base


.........

builder@zbook:~/isar$ cat
/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/temp/log.do_install.16775
DEBUG: Executing shell function do_install
NOTE: Creating example-raw binary
NOTE: Putting example-raw into package
install: creating directory
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image’
install: creating directory
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr’
install: creating directory
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr/local’
install: creating directory
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr/local/bin/’
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/example-raw’
->
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr/local/bin/example-raw’
NOTE: Now copy /README into package
install: creating directory
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr/local/doc/’
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/README’
->
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//usr/local/doc/README-example-raw-0.2’
NOTE: Now for a debian hook, see dpkg-deb
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/postinst’
->
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//DEBIAN/postinst’
install: cannot create regular file
‘/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/image//DEBIAN/postinst’:
No such file or directory
WARNING: exit code 1 from a shell command.
ERROR: Function failed: do_install (log file is located at
/home/builder/isar/build/tmp/work/raspbian-jessie-armhf/example-raw-0.2-r0/temp/log.do_install.16775)

Alex

[Alexander Smirnov]

On 02/12/2018 10:59 AM, Jan Kiszka wrote:

> From: Jan Kiszka <jan.k...@siemens.com>
>
> When we run unprivileged, writing files in that directory will not work
> after the first run because everything is root-owned at the end.
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
> meta/classes/dpkg-raw.bbclass | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index 44aa078..e3ae9e8 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -16,10 +16,9 @@ do_install() {
> do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> addtask install after do_unpack before do_deb_package_prepare
>
> -# so we can put hooks etc. in there already
> -do_install[dirs] = "${D}/DEBIAN"

Here you remove this folder, while it's used in example-raw recipe.

Alex

[Jan Kiszka]

It is created here now - nothing is lost. If you tell me a good reason
to keep it listed above as well, I'm fine with leaving that line alone.

Jan

[Jan Kiszka]

On 2018-02-13 14:49, Alexander Smirnov wrote:
> On 02/12/2018 10:59 AM, Jan Kiszka wrote:
>> From: Jan Kiszka <jan.k...@siemens.com>
>>
>> This ensures we have a clean base when rerunning the task.
>>
>
> How do you do this?
>
>  $ bitbake -C do_rootfs ...?
>
> If so, is it correct way to do this, I mean without do_clean before?

The task is rerun by bitbake now whenever it detects that some
dependency changed and had to be rerun as well. Or of the image recipe
itself got some changes. I do not need explicit "-c clean" in most cases
(see the other reply on patch 4).

[Jan Kiszka]

Oh! Yes, there is likely a dependency on "dpkg-raw: Install hooks
automatically" in this series, sorry. Please make sure that patch is
applied as well.

[Alexander Smirnov]

Jan Kiszka <jan.k...@siemens.com> 13 февраля 2018 г. 19:22:23 написал:

> On 2018-02-13 15:06, Alexander Smirnov wrote:
>>
>>
>> On 02/12/2018 10:59 AM, Jan Kiszka wrote:
>>> From: Jan Kiszka <jan.k...@siemens.com>
>>>
>>> When we run unprivileged, writing files in that directory will not work
>>> after the first run because everything is root-owned at the end.
>>>
>>> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
>>> ---
>>>   meta/classes/dpkg-raw.bbclass | 5 ++---
>>>   1 file changed, 2 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>> b/meta/classes/dpkg-raw.bbclass
>>> index 44aa078..e3ae9e8 100644
>>> --- a/meta/classes/dpkg-raw.bbclass
>>> +++ b/meta/classes/dpkg-raw.bbclass
>>> @@ -16,10 +16,9 @@ do_install() {
>>>   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>   addtask install after do_unpack before do_deb_package_prepare
>>>   -# so we can put hooks etc. in there already
>>> -do_install[dirs] = "${D}/DEBIAN"
>>
>> Here you remove this folder, while it's used in example-raw recipe.

I mean here you break the build for next and master.

Alex

>> > Alex
>>
>>> -
>>>   do_deb_package_prepare() {
>>> +    sudo rm -rf ${D}/DEBIAN
>>> +    mkdir ${D}/DEBIAN
>
> It is created here now - nothing is lost. If you tell me a good reason
> to keep it listed above as well, I'm fine with leaving that line alone.
>
> Jan
>
>>>       cat<<-__EOF__ > ${D}/DEBIAN/control
>>>           Package: ${PN}
>>>           Architecture: ${DISTRO_ARCH}
>>>
>

--
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

[Jan Kiszka]

On 2018-02-13 17:31, Alexander Smirnov wrote:
>
>
>
> Jan Kiszka <jan.k...@siemens.com> 13 февраля 2018 г. 19:22:23 написал:
>
>> On 2018-02-13 15:06, Alexander Smirnov wrote:
>>>
>>>
>>> On 02/12/2018 10:59 AM, Jan Kiszka wrote:
>>>> From: Jan Kiszka <jan.k...@siemens.com>
>>>>
>>>> When we run unprivileged, writing files in that directory will not work
>>>> after the first run because everything is root-owned at the end.
>>>>
>>>> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
>>>> ---
>>>>   meta/classes/dpkg-raw.bbclass | 5 ++---
>>>>   1 file changed, 2 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/meta/classes/dpkg-raw.bbclass
>>>> b/meta/classes/dpkg-raw.bbclass
>>>> index 44aa078..e3ae9e8 100644
>>>> --- a/meta/classes/dpkg-raw.bbclass
>>>> +++ b/meta/classes/dpkg-raw.bbclass
>>>> @@ -16,10 +16,9 @@ do_install() {
>>>>   do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>>>   addtask install after do_unpack before do_deb_package_prepare
>>>>   -# so we can put hooks etc. in there already
>>>> -do_install[dirs] = "${D}/DEBIAN"
>>>
>>> Here you remove this folder, while it's used in example-raw recipe.
>
> I mean here you break the build for next and master.

Yes, that is because my "dpkg-raw: Install hooks automatically" patch
has to be applied first (which prevents that the do_install step messes
with DEBIAN). Then the reasoning below applies.

Jan

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

When we run unprivileged, writing files in that directory will not work
after the first run because everything is root-owned at the end.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-raw.bbclass | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index 44aa078..e3ae9e8 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -16,10 +16,9 @@ do_install() {
do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
addtask install after do_unpack before do_deb_package_prepare

-# so we can put hooks etc. in there already
-do_install[dirs] = "${D}/DEBIAN"

-
do_deb_package_prepare() {
+ sudo rm -rf ${D}/DEBIAN
+ mkdir ${D}/DEBIAN

cat<<-__EOF__ > ${D}/DEBIAN/control
Package: ${PN}
Architecture: ${DISTRO_ARCH}
--

2.13.6

[Jan Kiszka]

Yeah, finally Yocto/OE-like usability: This ensures for many cases that
changes to recipes lead to rebuilds of dependent recipes, including the
final image. Some extra measures are needed so that those rebuilds work
with clean dirs.

And if the change detection should not work, e.g. changes to file://
resources are not detected, then a clean or cleanall task is now
available and ensures a proper manual rebuild.

This massively increases the fun factor when developing Isar projects.

Changes in v2:
- include patch 1 (dpkg-raw: Install hooks automatically) to clarify
that this is a dependency
- add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
broken mounts when doing rebuilds
- add patch 9 (buildchroot: Clean up rootfs prior to populating it),
analogously to patch 7 and 8

Jan

Jan Kiszka (9):
dpkg-raw: Install hooks automatically

Fix indention of base_do_build
Add clean and cleanall tasks
Enable recipe caching

Move buildchroot mounts into dpkg-base class

Enable proper rebuilds on dependency changes
dpkg-raw: Clean DEBIAN dir prior to filling it
isar-image-base: Clean rootfs folder prior to building

buildchroot: Clean up rootfs prior to populating it

.../recipes-app/example-raw/example-raw_0.2.bb | 8 -----
meta-isar/recipes-app/example-raw/files/postinst | 4 +++
meta-isar/recipes-core/images/isar-image-base.bb | 2 ++
meta/classes/base.bbclass | 39 +++++++++++++++++++--
meta/classes/dpkg-base.bbclass | 27 ++++++++++++++-
meta/classes/dpkg-raw.bbclass | 10 +++---
meta/conf/isar-bitbake.conf | 18 ++++++++--
meta/recipes-devtools/buildchroot/buildchroot.bb | 40 +++++-----------------
8 files changed, 99 insertions(+), 49 deletions(-)

--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Import BB_HASHBASE_WHITELIST and BB_HASHCONFIG_WHITELIST from OE,
removing some items that are surely OE-specific, and set CACHE. That's
all what's needed to enable bitbake's recipe parsing cache.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

Python code must not contain tabs.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This ensures we have a clean base when rerunning the task.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta-isar/recipes-core/images/isar-image-base.bb | 2 ++
1 file changed, 2 insertions(+)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb

index 8ddbabb..8678b51 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -31,6 +31,8 @@ do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"

do_rootfs() {
E="${@ bb.utils.export_proxies(d)}"

+ sudo rm -rf ${IMAGE_ROOTFS}
+
chmod +x "${WORKDIR}/${DISTRO_CONFIG_SCRIPT}"
chmod +x "${WORKDIR}/setup.sh"
install -m 755 "${WORKDIR}/download_dev-random" "${WORKDIR}/hooks_multistrap/"

--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This ensures we are always re-building a clean image.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index cd91228..62979f5 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -31,15 +31,17 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"

do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
- ${BUILDCHROOT_DIR}/dev \
- ${BUILDCHROOT_DIR}/proc \
- ${BUILDCHROOT_DIR}/sys"
do_build[depends] = "isar-apt:do_cache_config"

do_build() {
E="${@ bb.utils.export_proxies(d)}"

+ sudo rm -rf ${BUILDCHROOT_DIR}
+ mkdir -p ${BUILDCHROOT_DIR}/isar-apt
+ mkdir -p ${BUILDCHROOT_DIR}/dev
+ mkdir -p ${BUILDCHROOT_DIR}/proc
+ mkdir -p ${BUILDCHROOT_DIR}/sys
+
chmod +x "${WORKDIR}/setup.sh"
chmod +x "${WORKDIR}/configscript.sh"

--
2.13.6

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This avoids the unfortunate setup_mounts task which is either not
executed on rebuilds or causes excessive package rebuilds. We just need
ensure prior to switching into the buildchroot that all mount points are
populated.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-base.bbclass | 13 +++++++++-
meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
2 files changed, 15 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a45fbce..66132da 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
# Copyright (C) 2017 Siemens AG

# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_setup_mounts"
+do_build[depends] = "buildchroot:do_build"

# Add dependency between Isar recipes
DEPENDS ?= ""
@@ -20,11 +20,22 @@ dpkg_runbuild() {
die "This should never be called, overwrite it in your derived class"
}

+mount_if_needed() {
+
+ # The test may race with parallel builds. So probe, mount if needed, and
+ # then only fail if both the mount and yet another probe failed as well.
+ grep -q $2 /proc/mounts || sudo mount $1 $2 || grep -q $2 /proc/mounts
+}
+
# Wrap the function dpkg_runbuild with the bind mount for buildroot
do_build() {
mkdir -p ${BUILDROOT}
sudo mount --bind ${WORKDIR} ${BUILDROOT}

+ mount_if_needed "--bind ${DEPLOY_DIR_APT}/${DISTRO}" ${BUILDCHROOT_DIR}/isar-apt
+ mount_if_needed "-t devtmpfs -o mode=0755,nosuid devtmpfs" ${BUILDCHROOT_DIR}/dev
+ mount_if_needed "-t proc none" ${BUILDCHROOT_DIR}/proc
+
dpkg_runbuild

sudo umount ${BUILDROOT} 2>/dev/null || true
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 570c0ad..cd91228 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -60,7 +60,9 @@ do_build() {
-e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
"${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"

- do_setup_mounts
+ sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
+ sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
+ sudo mount -t proc none ${BUILDCHROOT_DIR}/proc

# Create root filesystem
sudo -E multistrap -a ${DISTRO_ARCH} -d "${BUILDCHROOT_DIR}" -f "${WORKDIR}/multistrap.conf"
@@ -70,30 +72,4 @@ do_build() {

# Configure root filesystem
sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
-
- do_cleanup_mounts
-}
-
-# Invalidate stamp for do_setup_mounts before each build start.
-# This will guarantee that this function will be executed once
-# per build.
-python __anonymous() {
- stamp = d.getVar("STAMP") + ".do_setup_mounts." + d.getVarFlag("do_setup_mounts", 'stamp-extra-info')
- os.remove(stamp) if os.path.exists(stamp) else None
-}
-
-do_setup_mounts[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
-
-do_setup_mounts() {
- sudo mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt
- sudo mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev
- sudo mount -t proc none ${BUILDCHROOT_DIR}/proc
-}
-
-addtask setup_mounts after do_build
-
-do_cleanup_mounts() {
- sudo umount ${BUILDCHROOT_DIR}/isar-apt 2>/dev/null || true
- sudo umount ${BUILDCHROOT_DIR}/dev 2>/dev/null || true
- sudo umount ${BUILDCHROOT_DIR}/proc 2>/dev/null || true
}
--
2.13.6

[Alexander Smirnov]

This could lead to have lots of remounts on the same folder during
parallel execution. This should be done in a separate task and protected
by [lockfiles].

Alex

[Jan Kiszka]

No, it won't - please re-read the code. It's a classic retry pattern,
and it's even detecting errors.

[Alexander Smirnov]

Jan Kiszka <jan.k...@siemens.com> 13 февраля 2018 г. 23:48:02 написал:

'mount --bind' never fails. So if several do_build() have passed first grep
in parallel, then each one will mount 'isar-apt'. Or I don't understand this.

Alex

> Jan
>
> --
> Siemens AG, Corporate Technology, CT RDA IOT SES-DE
> Corporate Competence Center Embedded Linux

[Jan Kiszka]

That is a valid remark - I wasn't aware of that behaviour (and
explaining this earlier would have helped a lot).

I will come up with a safe variant using flock soon.

And before you pull out your task idea again, I'd like to cite from the
bitbake manual:

"Any task that depends (possibly indirectly) on a [nostamp] task will
always be executed as well. This can cause unnecessary rebuilding if you
are not careful."

That's why its broken by design.

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This avoids the unfortunate setup_mounts task which is either not
executed on rebuilds or causes excessive package rebuilds. We just need
ensure prior to switching into the buildchroot that all mount points are

populated. The mounts may race with parallel build, so lock-protect the
check if they are needed and their execution.

Note that the more beautiful syntax using ( flock -n 9; ... ) 9>lockfile
as described in the flock man page does not work in bitbake shell
functions.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Changes in v3:
- use flock instead of unreliable check pattern

meta/classes/dpkg-base.bbclass | 11 ++++++++-
meta/recipes-devtools/buildchroot/buildchroot.bb | 30 +++---------------------
2 files changed, 13 insertions(+), 28 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a45fbce..895ff26 100644

--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
# Copyright (C) 2017 Siemens AG

# Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_setup_mounts"
+do_build[depends] = "buildchroot:do_build"

# Add dependency between Isar recipes
DEPENDS ?= ""

@@ -20,11 +20,20 @@ dpkg_runbuild() {

die "This should never be called, overwrite it in your derived class"
}

+MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount.lock"

+
# Wrap the function dpkg_runbuild with the bind mount for buildroot
do_build() {
mkdir -p ${BUILDROOT}
sudo mount --bind ${WORKDIR} ${BUILDROOT}

+ sudo flock ${MOUNT_LOCKFILE} -c ' \
+ if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
+ mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
+ mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
+ mount -t proc none ${BUILDCHROOT_DIR}/proc; \
+ fi'

[Alexander Smirnov]

With this name lockfile will be shared for all multiconfigs. I realize
that it's small, but unnecessary serialization of parallel parts.

Would it be ok?

MOUNT_LOCKFILE = "${TMPDIR}/buildchroot-mount-${DISTRO}-${DISTRO_ARCH}.lock"

or

MOUNT_LOCKFILE =
"${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/buildchroot/mount.lock"

or

MOUNT_LOCKFILE = "${BUILDCHROOT_DIR}/mount.lock"

Alex

[Jan Kiszka]

Yes, good point! Feel free to pick what ever is preferred when merging,
unless there is a reason I need to resend the whole thing anyway.

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:

> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
>
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
>
> This massively increases the fun factor when developing Isar projects.

Wow, just noticed that with this series bitbake has started to run tasks
in parallel.. How it's working? :-)


Currently 4 running tasks (54 of 217) 24%
|########################################

|
0: mc:rpi-jessie:buildchroot-1.0-r0 do_build - 101s (pid 3271)
1: mc:qemui386-stretch:buildchroot-1.0-r0 do_build - 101s (pid 3336)
2: mc:qemui386-jessie:buildchroot-1.0-r0 do_build - 100s (pid 3457)
3: mc:qemuarm-wheezy:buildchroot-1.0-r0 do_build - 100s (pid 3524)

Alex

[Jan Kiszka]

Didn't notice that this wasn't the case before (most of my workload had
linear ordering), but I could imagine that the lacking signatures forced
bitbake to serialize. Just guessing.

[Alexander Smirnov]

There is still the same build problem as reported yesterday, log is
attached. Dropping patch #3 fixes it.

Line I used to build:

$ time bitbake multiconfig:qemuarm-wheezy:isar-image-base
multiconfig:qemuarm-jessie:isar-image-base
multiconfig:qemuarm-stretch:isar-image-base
multiconfig:qemui386-jessie:isar-image-base
multiconfig:qemui386-stretch:isar-image-base
multiconfig:qemuamd64-jessie:isar-image-base
multiconfig:qemuamd64-stretch:isar-image-base
multiconfig:rpi-jessie:isar-image-base

Just for sure stated build in CI server:
http://isar-build.org:8080/job/isar_asmirnov_devel/8/console

Also failed. Could I dropped this patch from the series?

Alex

[Jan Kiszka]

Not before you can explain why things fail and why that patch causes
this. We need to understand the problem. Let me reproduce...

[Jan Kiszka]

Yeah, I get some bug as well, but a different & well-known one:

dpkg: error: dpkg status database is locked by another process

So, I'm afraid we will find more of these sporadic breakage, now that
parallelization works better....

[Jan Kiszka]

BTW, I'm testing with
https://github.com/siemens/isar/commits/jan/staging, i.e. the sum of all
my misdoing.

Good news is that I just reproduced your issue, now digging into what
corrupts the isar-apt repo /wrt libhello-dev. Bad news: I had to restart
5 times due to the contention bug above.

[Jan Kiszka]

OK, understood this one: repo_clean. The arm task kills the results of
the i386 or vice versa. Need to work on the filtering rules here. Or we
need to split the isar-apt repos per multiconf target...

[Alexander Smirnov]

I think that the issue is in repo_clean function. "isar_apt" splits
databases by ${DISTRO}:

builder@zbook:~/isar/build$ ls tmp/deploy/apt/ -1
debian-jessie
debian-stretch
debian-wheezy
raspbian-jessie

And each distro could contains packages of whatever arch.

I suppose that clean_repo() as it's called, remove all the architectures
for specified package name. So building for ARM erases content for AMD
and i386. Looking into the manual, -A option should be passed to reprepro.

Alex

[Alexander Smirnov]

This how ftp.debian.org is organized, the pool contains all the
supported architectures for specific package:

http://ftp.de.debian.org/debian/pool/main/liba/libapache-mod-auth-radius/

Alex

[Jan Kiszka]

Yes, exactly. I will try the -A next.

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:

> From: Jan Kiszka <jan.k...@siemens.com>
>
> This ensures we are always re-building a clean image.
>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---
> meta/recipes-devtools/buildchroot/buildchroot.bb | 10 ++++++----
> 1 file changed, 6 insertions(+), 4 deletions(-)
>
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
> index cd91228..62979f5 100644
> --- a/meta/recipes-devtools/buildchroot/buildchroot.bb
> +++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
> @@ -31,15 +31,17 @@ BUILDCHROOT_PREINSTALL ?= "gcc \
> WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
>
> do_build[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> -do_build[dirs] = "${BUILDCHROOT_DIR}/isar-apt \
> - ${BUILDCHROOT_DIR}/dev \
> - ${BUILDCHROOT_DIR}/proc \
> - ${BUILDCHROOT_DIR}/sys"
> do_build[depends] = "isar-apt:do_cache_config"
>
> do_build() {
> E="${@ bb.utils.export_proxies(d)}"
>

Do you have any concerns if I duplicate commit comment here? Otherwise I
expect in near future patch which moves this to [dirs]. Already had
several cycles of doing this in `git log` :-)

> + sudo rm -rf ${BUILDCHROOT_DIR}
> + mkdir -p ${BUILDCHROOT_DIR}/isar-apt
> + mkdir -p ${BUILDCHROOT_DIR}/dev
> + mkdir -p ${BUILDCHROOT_DIR}/proc
> + mkdir -p ${BUILDCHROOT_DIR}/sys
> +
> chmod +x "${WORKDIR}/setup.sh"
> chmod +x "${WORKDIR}/configscript.sh"
>
>

Alex

[Jan Kiszka]

Yes, absolutely!

[Alexander Smirnov]

On 02/13/2018 11:05 PM, Jan Kiszka wrote:

> Yeah, finally Yocto/OE-like usability: This ensures for many cases that
> changes to recipes lead to rebuilds of dependent recipes, including the
> final image. Some extra measures are needed so that those rebuilds work
> with clean dirs.
>
> And if the change detection should not work, e.g. changes to file://
> resources are not detected, then a clean or cleanall task is now
> available and ensures a proper manual rebuild.
>
> This massively increases the fun factor when developing Isar projects.
>
> Changes in v2:
> - include patch 1 (dpkg-raw: Install hooks automatically) to clarify
> that this is a dependency
> - add patch 5 (Move buildchroot mounts into dpkg-base class) to fix
> broken mounts when doing rebuilds
> - add patch 9 (buildchroot: Clean up rootfs prior to populating it),
> analogously to patch 7 and 8

Applied to next, thanks!

Alex

[Jan Kiszka]

Oh... Does it pass your CI?

I just started a build that tests flock around the dependency
installations so that it can complete all targets in one run here.
Should be simple (if it works...), will send the patch on top ASAP.

Jan

[Alexander Smirnov]

Local build passed :-)

Started CI one: http://isar-build.org:8080/job/isar_next/103/console

Will try local one again, possibly I missed something.

Alex

[Jan Kiszka]

BTW, is there a way for me to extract also the bitbake locks from those
builds? Generally, you are lost without them.

>
> Will try local one again, possibly I missed something.

How many cores do you have? I've also lifted BB_NUMBER_THREADS to all
available ones, but already 4 gave enough trouble.

[Alexander Smirnov]

Jenkins provides possibility to browse build folders, but for now they
are not enabled in UI. This task is in my todo-list but with low
priority. So if anybody else except me is going to use this server, I
could handle this soon.

>
>>
>> Will try local one again, possibly I missed something.
>
> How many cores do you have? I've also lifted BB_NUMBER_THREADS to all
> available ones, but already 4 gave enough trouble.

I use only the default settings, so 4 threads. The second build is also
passed.

Alex

[Claudius Heine]

Hi Jan,

On 02/13/2018 09:05 PM, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>

> When we run unprivileged, writing files in that directory will not work
> after the first run because everything is root-owned at the end.

>
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
> ---

> meta/classes/dpkg-raw.bbclass | 5 ++---
> 1 file changed, 2 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
> index 44aa078..e3ae9e8 100644
> --- a/meta/classes/dpkg-raw.bbclass
> +++ b/meta/classes/dpkg-raw.bbclass
> @@ -16,10 +16,9 @@ do_install() {
> do_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> addtask install after do_unpack before do_deb_package_prepare
>
> -# so we can put hooks etc. in there already
> -do_install[dirs] = "${D}/DEBIAN"
> -
> do_deb_package_prepare() {
> + sudo rm -rf ${D}/DEBIAN
> + mkdir ${D}/DEBIAN
> cat<<-__EOF__ > ${D}/DEBIAN/control
> Package: ${PN}
> Architecture: ${DISTRO_ARCH}
>

This:

do_install() {
sudo install -v -m 755 ${WORKDIR}/postinst ${D}/DEBIAN/postinst
}

causes now:

install: cannot create regular file ‘/[...]/DEBIAN/postinst’: No such
file or directory

How should that be fixed?

Cheers,
Claudius

--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: c...@denx.de

[Jan Kiszka]

Simply by removing the install above. "dpkg-raw: Install hooks
automatically" makes it obsolete.

[Claudius Heine]

Thx! But now it shows:

DEBUG: Executing shell function do_deb_package_prepare
mkdir: cannot create directory ‘/[...]/DEBIAN’: No such file or directory

I suppose it needs "mkdir -p ${D}/DEBIAN".

[Alexander Smirnov]

BTW: this series is already in next and it has passed all CI tests.
Which git head you use?

Alex

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

If do_install did not do this for us, we failed.

Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

meta/classes/dpkg-raw.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-raw.bbclass b/meta/classes/dpkg-raw.bbclass
index e0ea811..28233ac 100644
--- a/meta/classes/dpkg-raw.bbclass
+++ b/meta/classes/dpkg-raw.bbclass
@@ -18,7 +18,7 @@ addtask install after do_unpack before do_deb_package_prepare

do_deb_package_prepare() {

sudo rm -rf ${D}/DEBIAN

- mkdir ${D}/DEBIAN
+ mkdir -p ${D}/DEBIAN

cat<<-__EOF__ > ${D}/DEBIAN/control
Package: ${PN}
Architecture: ${DISTRO_ARCH}

--
2.13.6

[Claudius Heine]

Hi,

I rebased my own branch to next (e7e40a68adbfa60), and my stuff broke.
AFAIK next is there to test stuff with our own setup and provide
feedback, so that commits can be fixed there with sometimes even some
history rewriting.

I wouldn't be against squashing Jans patch into this one on next.

regards,

[Alexander Smirnov]

On 02/15/2018 01:53 PM, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> If do_install did not do this for us, we failed.
>
> Fixes: b04530d19c79 (dpkg-raw: Clean DEBIAN dir prior to filling it)
> Signed-off-by: Jan Kiszka <jan.k...@siemens.com>

[Andreas Reichel]

On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>

I just noticed, for me it tries to create a DEBIAN folder inside
an "image" folder, that has root-only write permissions and thus
fails since the function is executed as user "builder".

I reproduced this with a clean build and a dpkg-raw recipe.

No time to debug this atm.

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To post to this group, send email to isar-...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/3ae97982-962e-be4d-22a6-477db2fad432%40siemens.com.
> For more options, visit https://groups.google.com/d/optout.

--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas...@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

[Jan Kiszka]

On 2018-02-26 12:39, Andreas Reichel wrote:
> On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
>> From: Jan Kiszka <jan.k...@siemens.com>
>
> I just noticed, for me it tries to create a DEBIAN folder inside
> an "image" folder, that has root-only write permissions and thus
> fails since the function is executed as user "builder".
>
> I reproduced this with a clean build and a dpkg-raw recipe.
>
> No time to debug this atm.

I suppose that was a rebuild, not a clean build (otherwise the ownership
would not be on root already). Anyway, yet another example for our mess
with ownerships vs. cleanups.

Maybe there is some way to overload the functions that bitbake uses when
it tries to do mytask[cleandirs] and switch back to a consistent bitbake
model. But I didn't figure this out yet.

Jan

[Andreas Reichel]

On Mon, Feb 26, 2018 at 12:39:34PM +0100, Andreas Reichel wrote:
> On Thu, Feb 15, 2018 at 11:53:39AM +0100, [ext] Jan Kiszka wrote:
> > From: Jan Kiszka <jan.k...@siemens.com>
>

Please ignore this report :) my do_install was wrong