[PATCH] dpkg-base: Remove newly deployed debs from buildchroots

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This ensures clean reinstallation after partial rebuilds.

A typical error pattern so far was that firmware packages pulled by the
buildchroot-target were not updated there on rebuilds, causing the old
firmware being deployed into the image.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---
meta/classes/dpkg-base.bbclass | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 9aa2d546..80da713d 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -154,20 +154,26 @@ python do_dpkg_build() {

addtask dpkg_build before do_build

-CLEANFUNCS += "repo_clean"
+CLEANFUNCS += "deb_clean"

-repo_clean() {
+deb_clean() {
DEBS=$( find ${S}/.. -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
if [ -n "${DEBS}" ]; then
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
+ package=$(basename "${d}")
+ package_remove="/usr/bin/apt-get remove -y -U ${package%%_*}"
+ sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
+ if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
+ sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
+ fi
done
fi
}

do_deploy_deb() {
- repo_clean
+ deb_clean
repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
}
--
2.26.2

[Harald Seiler]

Hi Jan,

after wasting way too much time hunting another bug that turned out to be
caused by the bootloader not getting updated, I gave this patch another
try. It works as intended, I just had to fix one small issue. See below.

I had to remove -U here. Not sure what it was for, I can't find any
documentation about this switch.

> + sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
> + if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
> + sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
> + fi
> done
> fi
> }
>
> do_deploy_deb() {
> - repo_clean
> + deb_clean
> repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
> "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
> }
> --
> 2.26.2
>

--
Harald

DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: +49-8142-66989-62 Fax: +49-8142-66989-80 Email: h...@denx.de

[Jan Kiszka]

No idea anymore where it came from - and why it worked. I'll send v2.

Jan

>> + sudo -E chroot ${BUILDCHROOT_DIR} ${package_remove} || true
>> + if [ "${BUILDCHROOT_DIR}" != "${BUILDCHROOT_TARGET_DIR}" ]; then
>> + sudo -E chroot ${BUILDCHROOT_TARGET_DIR} ${package_remove} || true
>> + fi
>> done
>> fi
>> }
>>
>> do_deploy_deb() {
>> - repo_clean
>> + deb_clean
>> repo_add_packages "${REPO_ISAR_DIR}"/"${DISTRO}" \
>> "${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" ${S}/../*.deb
>> }
>> --
>> 2.26.2
>>

--

Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[Jan Kiszka]

From: Jan Kiszka <jan.k...@siemens.com>

This ensures clean reinstallation after partial rebuilds.

A typical error pattern so far was that firmware packages pulled by the
buildchroot-target were not updated there on rebuilds, causing the old
firmware being deployed into the image.

Signed-off-by: Jan Kiszka <jan.k...@siemens.com>
---

Changes in v2:
- removed bogus -U options which sneaked in from reading the wrong man
page (zypper's apt-get wrapper: -U, --no-clean-deps)

meta/classes/dpkg-base.bbclass | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass

index 9aa2d546..08880d7d 100644

--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -154,20 +154,26 @@ python do_dpkg_build() {

addtask dpkg_build before do_build

-CLEANFUNCS += "repo_clean"
+CLEANFUNCS += "deb_clean"

-repo_clean() {
+deb_clean() {
DEBS=$( find ${S}/.. -maxdepth 1 -name "*.deb" || [ ! -d ${S} ] )
if [ -n "${DEBS}" ]; then
for d in ${DEBS}; do
repo_del_package "${REPO_ISAR_DIR}"/"${DISTRO}" \
"${REPO_ISAR_DB_DIR}"/"${DISTRO}" "${DEBDISTRONAME}" "${d}"
+ package=$(basename "${d}")

+ package_remove="/usr/bin/apt-get remove -y ${package%%_*}"

[Harald Seiler]

On Tue, 2020-08-11 at 09:05 +0200, Jan Kiszka wrote:
> From: Jan Kiszka <jan.k...@siemens.com>
>
> This ensures clean reinstallation after partial rebuilds.
>
> A typical error pattern so far was that firmware packages pulled by the
> buildchroot-target were not updated there on rebuilds, causing the old
> firmware being deployed into the image.

Tested-by: Harald Seiler <h...@denx.de>
Reviewed-by: Harald Seiler <h...@denx.de>

[Henning Schild]

This will cause serious headache when rebuilding essential packages.
You risk removing dpkg/apt and others.

I think we rather need an "apt-get upgrade --allow-downgrades" together
with a "apt-get autoremove" somewhere.

Henning

[Jan Kiszka]

Not sure if that would already work, at least when the result actually
targets the buildchroot and not just the device.

>
> I think we rather need an "apt-get upgrade --allow-downgrades" together
> with a "apt-get autoremove" somewhere.

I think those only work if the version number changed which is not
guaranteed. We would need some switch that enforces re-installation, and
that possibly only when some trigger tells us that the package has
changed. apt-get reinstall?

If we do not find a quick solution, I think such corner cases could also
be addressed when needed. The current situation is too problematic to
leave it like it us much longer.

Jan

[Henning Schild]

I am also not sure, especially when looking at the buildchroots.

> >
> > I think we rather need an "apt-get upgrade --allow-downgrades"
> > together with a "apt-get autoremove" somewhere.
>
> I think those only work if the version number changed which is not
> guaranteed. We would need some switch that enforces re-installation,
> and that possibly only when some trigger tells us that the package has
> changed. apt-get reinstall?

Not changing the version number is probably bad style. I think one
should always change it when working on a recipe. While working on it
people hopefully know what to clean, and others merging changes will
simply get a proper rebuild with a new version.

> If we do not find a quick solution, I think such corner cases could
> also be addressed when needed. The current situation is too
> problematic to leave it like it us much longer.

Agreed, it could just be a made-up corner-case. Just wanted to point
out the potential danger and stress the fact that changed content
without changing the version is a bad idea.

Henning

> Jan
>

[Baurzhan Ismagulov]

On Tue, Aug 11, 2020 at 09:05:59AM +0200, Jan Kiszka wrote:
> This ensures clean reinstallation after partial rebuilds.
>
> A typical error pattern so far was that firmware packages pulled by the
> buildchroot-target were not updated there on rebuilds, causing the old
> firmware being deployed into the image.

Applied to next, thanks for the patch and the review.

Yes, Debian Policy section 3.2 requires that different packages have different
version numbers. Maybe we should introduce a script that would do that for the
developer.

With kind regards,
Baurzhan.

[Jan Kiszka]

We've discussed this too often: This is not how people work, and it also
makes zero sense for a local development cycle to continuously bump
version numbers.

Jan

>
>> If we do not find a quick solution, I think such corner cases could
>> also be addressed when needed. The current situation is too
>> problematic to leave it like it us much longer.
>
> Agreed, it could just be a made-up corner-case. Just wanted to point
> out the potential danger and stress the fact that changed content
> without changing the version is a bad idea.
>
> Henning
>
>> Jan
>>
>

[Jan Kiszka]

This won't help in the cases you play with a recipe, change small
details, rebuild, test. It's reasonable to update the version when
publishing. However, it's a needless burden to do that for a local
development cycle. Here, Isar shall support that the just built instance
of a package is also used locally, without an explicit "clean".

[Baurzhan Ismagulov]

Hello Jan,

On Tue, Aug 11, 2020 at 09:05:59AM +0200, Jan Kiszka wrote:

> This ensures clean reinstallation after partial rebuilds.
>
> A typical error pattern so far was that firmware packages pulled by the
> buildchroot-target were not updated there on rebuilds, causing the old
> firmware being deployed into the image.

With this applied, we encounter random build failures downstream, e.g.:

1. ERROR: mc:pixi-cdl100-buster:desktop-shortcuts-1.0-r0 do_install_builddeps: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.do_install_builddeps.19582' failed with exit code 1:
mk-build-deps: You must have equivs installed to use this program.
WARNING: exit code 1 from a shell command.

2. Exception: bb.process.ExecutionError: Execution of 'build/tmp/work/debian-buster-armhf/desktop-shortcuts/1.0-r0/temp/run.dpkg_runbuild.16998' failed with exit code 2:
dpkg-buildpackage: info: source package desktop-shortcuts
dpkg-buildpackage: info: source version 1.0
dpkg-buildpackage: info: source distribution UNRELEASED
dpkg-buildpackage: info: source changed by isar-users <isar-...@googlegroups.com>
dpkg-architecture: warning: specified GNU system type arm-linux-gnueabihf does not match CC system type x86_64-linux-gnu, try setting a correct CC environment variable
dpkg-source -I --before-build .
dpkg-buildpackage: info: host architecture armhf
fakeroot debian/rules clean
dh clean
make: dh: Command not found

Seems we have a race, possibly related to removing packages. What I don't
understand is how it could affect equivs or debhelper -- we don't do anything
with them. Ideas?

With kind regards,
Baurzhan.

[Baurzhan Ismagulov]

On Tue, Aug 18, 2020 at 07:57:26AM +0200, Jan Kiszka wrote:
> > Yes, Debian Policy section 3.2 requires that different packages have different
> > version numbers. Maybe we should introduce a script that would do that for the
> > developer.
>
> This won't help in the cases you play with a recipe, change small
> details, rebuild, test. It's reasonable to update the version when
> publishing. However, it's a needless burden to do that for a local
> development cycle. Here, Isar shall support that the just built instance
> of a package is also used locally, without an explicit "clean".

If this is how the upstream tooling works (which is a requirement for them
since there is no other way), things will get complicated and keep breaking. An
alternative could be to automate version bumping by requiring a local commit.
In the long term, I don't see a way around that.

With kind regards,
Baurzhan.

[Jan Kiszka]

Which packages are you (re-)building? Debian packages? Or just custom
dependencies of that desktop-shortcuts package?

In theory, do_deploy_deb should be locked against do_install_builddeps.
Anything that the latter needs should be available after the former.
Maybe you are missing a build dependency in desktop-shortcuts? Try build
that only over a clean buildchroot, that will tell you.

[Jan Kiszka]

On 23.08.20 18:30, Baurzhan Ismagulov wrote:
> On Tue, Aug 18, 2020 at 07:57:26AM +0200, Jan Kiszka wrote:
>>> Yes, Debian Policy section 3.2 requires that different packages have different
>>> version numbers. Maybe we should introduce a script that would do that for the
>>> developer.
>>
>> This won't help in the cases you play with a recipe, change small
>> details, rebuild, test. It's reasonable to update the version when
>> publishing. However, it's a needless burden to do that for a local
>> development cycle. Here, Isar shall support that the just built instance
>> of a package is also used locally, without an explicit "clean".
>
> If this is how the upstream tooling works (which is a requirement for them
> since there is no other way), things will get complicated and keep breaking. An

Can you name concrete other issues?

> alternative could be to automate version bumping by requiring a local commit.
> In the long term, I don't see a way around that.

Commits for rebuilds are user-unfriendly. We will need to solve that in
a better way versioning should actually require a bump.

[Jan Kiszka]

Meanwhile, I understood the issue is it happened on out CI system as
well during nightly runs (that shouldn't have happen, but that a local
issue). The logs you cited were incomplete:

[...]
NOTE: recipe cowsay-git-r0: task do_dpkg_build: Succeeded
NOTE: Running task 1617 of 1683 (mc:qemumipsel-buster:/builds/ebsy/debian/isar/meta-isar/recipes-app/cowsay/cowsay_git.bb:do_deploy_deb)
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Started
[...]
NOTE: Running noexec task 1637 of 1683 (mc:rpi-stretch:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_build)
[...]
ERROR: mc:qemuarm-buster:isar-image-base-1.0-r0 do_rootfs_install: Error executing a python function in exec_python_func() autogenerated:
[...]
E: Failed to fetch file:/isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb File not found - /isar-apt/pool/main/c/cowsay/cowsay_3.03+dfsg2-4_all.deb (2: No such file or directory)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
WARNING: exit code 100 from a shell command.


ERROR: Logfile of failure stored in: /builds/ebsy/debian/isar/build/tmp/work/debian-buster-armhf/isar-image-base-qemuarm-ext4-img/1.0-r0/temp/log.do_rootfs_install.32804
NOTE: recipe isar-image-base-1.0-r0: task do_rootfs_install: Failed
ERROR: Task (mc:qemuarm-buster:/builds/ebsy/debian/isar/meta-isar/recipes-core/images/isar-image-base.bb:do_rootfs_install) failed with exit code '1'
NOTE: recipe cowsay-git-r0: task do_deploy_deb: Succeeded

The problem is that we have a false sharing between all archs building
cowsay for buster. That is because this demo package is arch "all", but
the tasks assume it is arch-specific, thus are building and
(re-)deploying it happily in parallel. This is similar to the issue I
had with [1]. It is not caused by this patch, it is revealed by it,
finally.

What can we do?
- select a different rebuild-example, one that does not generate "all"
packages (i.e. stick our head into the sand)
- break up isar-apt on a per-arch basis (as considered for [1])
- find a magic spell that allows to express the commonality between
all mc:<arch>-<distro>:<package> recipes that actually produce the
same output

I suspect the latter will not happen any time soon, so I'm inclined to
fix this via the second option.

Jan

[1] https://groups.google.com/d/msgid/isar-users/e4ca6d44-8b99-24ba-4fcd-c36cedac1ff7%40siemens.com

[Kanagarajan, Vijaikumar]

Hello Jan,

I have a similar condition now. We build openssl with a patch downstream. Looks like with this patch now, in our setup openssl is removed from buildchrootdir, which inturn removes
devscripts(contains mk-build-deps). This causes other recipe builds to fail. Please see the log below.


ERROR: expand-on-first-boot-1.1-r0 do_install_builddeps: Execution of '/work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/run.do_install_builddeps.10653' failed with exit code 1:
/isar/deps.sh: line 39: mk-build-deps: command not found

WARNING: exit code 1 from a shell command.

ERROR: Logfile of failure stored in: /work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/log.do_install_builddeps.10653
Log data follows:
| DEBUG: Executing shell function do_install_builddeps
| /isar/deps.sh: line 39: mk-build-deps: command not found

| WARNING: exit code 1 from a shell command.

| ERROR: Execution of '/work/build/tmp/work/ebsy-buster-amd64/expand-on-first-boot/1.1-r0/temp/run.do_install_builddeps.10653' failed with exit code 1:
| /isar/deps.sh: line 39: mk-build-deps: command not found

| WARNING: exit code 1 from a shell command.
|

ERROR: Task (/work/isar/meta/recipes-support/expand-on-first-boot/expand-on-first-boot_1.1.bb:do_install_builddeps) failed with exit code '1'
ERROR: customizations-debug-0.1-ebsy-r0 do_install_builddeps: Execution of '/work/build/tmp/work/ebsy-buster-amd64/customizations-debug/0.1-ebsy-r0/temp/run.do_install_builddeps.10655' failed with exit code 1:
/isar/deps.sh: line 39: mk-build-deps: command not found

WARNING: exit code 1 from a shell command.

Thanks,
Vijai Kumar K

--
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/c3998f33-521e-49ed-41c5-95f1f935d5df%40siemens.com.

[Jan Kiszka]

I've proposed
https://groups.google.com/d/msgid/isar-users/76603767-1d25-c347-f7b1-84d4c106e771%40siemens.com
to address that scenario. We are using that in one internal layer already.

Jan

--
Siemens AG, T RDA IOT

[Kanagarajan, Vijaikumar]

Great. I will try that out.

Thanks,
Vijai Kumar K