[PATCH] Fix manifest generation

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

finalize-rootfs task removes qemu-*-static. It is needed for
rootfs_generate_manifest to work when HOST_ARCH != DISTRO_ARCH.

Make sure a proper chroot environment is available when
generating manifest.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---
meta/classes/rootfs.bbclass | 20 ++++++++++----------
1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/meta/classes/rootfs.bbclass b/meta/classes/rootfs.bbclass
index da66b3f..1da27ff 100644
--- a/meta/classes/rootfs.bbclass
+++ b/meta/classes/rootfs.bbclass
@@ -9,8 +9,8 @@ ROOTFS_PACKAGES ?= ""
# available features are:
# 'deploy-package-cache' - copy the package cache ${WORKDIR}/apt_cache
# 'clean-package-cache' - delete package cache from rootfs
-# 'finalize-rootfs' - delete files needed to chroot into the rootfs
# 'generate-manifest' - generate a package manifest of the rootfs into ${ROOTFS_MANIFEST_DEPLOY_DIR}
+# 'finalize-rootfs' - delete files needed to chroot into the rootfs
ROOTFS_FEATURES ?= ""

ROOTFS_APT_ARGS="install --yes -o Debug::pkgProblemResolver=yes"
@@ -185,6 +185,15 @@ rootfs_postprocess_clean_package_cache() {
sudo rm -rf "${ROOTFSDIR}/var/lib/apt/lists/"*
}

+ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
+rootfs_generate_manifest () {
+ mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
+ sudo -E chroot --userspec=$(id -u):$(id -g) '${ROOTFSDIR}' \
+ dpkg-query -W -f \
+ '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' > \
+ ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
+}
+
ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'finalize-rootfs', 'rootfs_postprocess_finalize', '', d)}"
rootfs_postprocess_finalize() {
sudo -s <<'EOSUDO'
@@ -223,15 +232,6 @@ rootfs_postprocess_finalize() {
EOSUDO
}

-ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
-rootfs_generate_manifest () {
- mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
- sudo -E chroot --userspec=$(id -u):$(id -g) '${ROOTFSDIR}' \
- dpkg-query -W -f \
- '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' > \
- ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
-}
-
do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
python do_rootfs_postprocess() {
# Take care that its correctly mounted:
--
2.17.1

[vijaikumar....@gmail.com]

From: Vijai Kumar K <Vijaikumar_...@mentor.com>

finalize-rootfs task removes qemu-*-static. It is needed for
rootfs_generate_manifest to work when HOST_ARCH != DISTRO_ARCH.

Make sure a proper chroot environment is available when
generating manifest.

Signed-off-by: Vijai Kumar K <Vijaikumar_...@mentor.com>
---

Changes in v2:
- Updated commit message.

[Gylstorff Quirin]

Is it possible to add a guard to finalize to check that it is the last
command in ROOTFS_POSTPROCESS_COMMAND.

> -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
> -rootfs_generate_manifest () {
> - mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
> - sudo -E chroot --userspec=$(id -u):$(id -g) '${ROOTFSDIR}' \
> - dpkg-query -W -f \
> - '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' > \
> - ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> -}
> -
> do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
> python do_rootfs_postprocess() {
> # Take care that its correctly mounted:
>

Thanks.

This should have been found in the CI build.
Is the CrossBuild not part of the CI configuration?

Quirin

[Henning Schild]

Am Tue, 22 Oct 2019 18:28:57 +0200
schrieb "[ext] Gylstorff Quirin" <quirin.g...@siemens.com>:

Not sure whether i can see which args ci_build.sh was called with, but
i quickly found a failing build of next in the CI.

For those who have access:
http://isar-build.org:8080/job/isar_next/488/console

>> chroot: failed to run command 'dpkg-query': No such file or directory
...
that is the missing qemu
...
>> ERROR: mc:qemuarm64-stretch:isar-image-base-1.0-r0
>> do_rootfs_postprocess: Function failed: rootfs_generate_manifest
>> (log file is located
>> at /workspace/build/isar_next/488/build/tmp/work/debian-stretch-arm64/isar-image-base-qemuarm64-ext4-img/1.0-r0/temp/log.do_rootfs_postprocess.23077)

Henning

> Quirin
>

[Vijai Kumar K]

We could use _append instead of += for finalize-rootfs. May be add a comment as well to warn developers
that any postprocess function that needs chroot should go before finalize-rootfs.

Thanks,
Vijai Kumar K

>
> > -ROOTFS_POSTPROCESS_COMMAND += "${@bb.utils.contains('ROOTFS_FEATURES', 'generate-manifest', 'rootfs_generate_manifest', '', d)}"
> > -rootfs_generate_manifest () {
> > - mkdir -p ${ROOTFS_MANIFEST_DEPLOY_DIR}
> > - sudo -E chroot --userspec=$(id -u):$(id -g) '${ROOTFSDIR}' \
> > - dpkg-query -W -f \
> > - '${source:Package}|${source:Version}|${binary:Package}|${Version}\n' > \
> > - ${ROOTFS_MANIFEST_DEPLOY_DIR}/"${PF}".manifest
> > -}
> > -
> > do_rootfs_postprocess[vardeps] = "${ROOTFS_POSTPROCESS_COMMAND}"
> > python do_rootfs_postprocess() {
> > # Take care that its correctly mounted:
> >
>
> Thanks.
>
> This should have been found in the CI build.
> Is the CrossBuild not part of the CI configuration?
>
> Quirin
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/28666b08-fc78-8ad3-083b-f94a345b98ce%40siemens.com.

[Baurzhan Ismagulov]

On Wed, Oct 23, 2019 at 10:46:00AM +0530, Vijai Kumar K wrote:
> > Is it possible to add a guard to finalize to check that it is the last
> > command in ROOTFS_POSTPROCESS_COMMAND.
>
> We could use _append instead of += for finalize-rootfs. May be add a comment as well to warn developers
> that any postprocess function that needs chroot should go before finalize-rootfs.

Thanks for heads-up. I've indeed let this slip in after it worked on a
different machine. If the patch works, I'm going to apply it.

That said, we've traced the skeleton problem to a similar issue with qemu in
meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py. I think we should:

1. Move finalize-rootfs out of ROOTFS_FEATURES and hard-code the call to e.g.
do_rootfs_postprocess() as the last step, and

2. Centralize qemu cp / rm in global functions using bb.persist_data.persist()
to copy it at the first request and remove at the last one.

Comments and other suggestions welcome.

With kind regards,
Baurzhan.

[Vijai Kumar K]

On Wed, Oct 23, 2019 at 06:25:35PM +0200, Baurzhan Ismagulov wrote:
> On Wed, Oct 23, 2019 at 10:46:00AM +0530, Vijai Kumar K wrote:
> > > Is it possible to add a guard to finalize to check that it is the last
> > > command in ROOTFS_POSTPROCESS_COMMAND.
> >
> > We could use _append instead of += for finalize-rootfs. May be add a comment as well to warn developers
> > that any postprocess function that needs chroot should go before finalize-rootfs.
>
> Thanks for heads-up. I've indeed let this slip in after it worked on a
> different machine. If the patch works, I'm going to apply it.

Yes, it works. Tested with qemuarm-stretch and rpi-stretch.

>
> That said, we've traced the skeleton problem to a similar issue with qemu in
> meta/scripts/lib/wic/plugins/source/rootfs-u-boot.py. I think we should:
>
> 1. Move finalize-rootfs out of ROOTFS_FEATURES and hard-code the call to e.g.
> do_rootfs_postprocess() as the last step, and
>
> 2. Centralize qemu cp / rm in global functions using bb.persist_data.persist()
> to copy it at the first request and remove at the last one.
>
> Comments and other suggestions welcome.
>
> With kind regards,
> Baurzhan.
>

> --
> You received this message because you are subscribed to the Google Groups "isar-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/isar-users/20191023162534.lo5mwc3nqexmqme7%40yssyq.m.ilbers.de.

[Baurzhan Ismagulov]

On Tue, Oct 22, 2019 at 08:23:36PM +0530, vijaikumar....@gmail.com wrote:
> finalize-rootfs task removes qemu-*-static. It is needed for
> rootfs_generate_manifest to work when HOST_ARCH != DISTRO_ARCH.
>
> Make sure a proper chroot environment is available when
> generating manifest.

Applied to next, thanks.

With kind regards,
Baurzhan.

[Henning Schild]

Am Thu, 24 Oct 2019 11:07:30 +0530
schrieb Vijai Kumar K <vijaikumar....@gmail.com>:

> On Wed, Oct 23, 2019 at 06:25:35PM +0200, Baurzhan Ismagulov wrote:
> > On Wed, Oct 23, 2019 at 10:46:00AM +0530, Vijai Kumar K wrote:
> > > > Is it possible to add a guard to finalize to check that it is
> > > > the last command in ROOTFS_POSTPROCESS_COMMAND.
> > >
> > > We could use _append instead of += for finalize-rootfs. May be
> > > add a comment as well to warn developers that any postprocess
> > > function that needs chroot should go before finalize-rootfs.
> >
> > Thanks for heads-up. I've indeed let this slip in after it worked
> > on a different machine. If the patch works, I'm going to apply it.
>
> Yes, it works. Tested with qemuarm-stretch and rpi-stretch.

I had to apply it on my ilbers-ci branch as well, which fixed the build
of the series i sent yesterday.

Henning