QA errors found whilst validating image: xenclient-installer-image-openxt-installer-20220203092754

[Vaishali Dhakate]

Dear openxt Team,
I am trying to build openxt dunfell branch, which I had already built, which I was working on about 2 weeks ago, had been working on synchronizer and openxt device : an upgrade to the openxt device from the Synchronizer.
Till I realized a repo sync was needed to get all the changes from the updated code.
Now I am facing an error related to image_qa_module_sigs failed.
I believe I have already cleaned the xenclient-image-installer and I am rebuilding this.
Like so, MACHINE=openxt-installer bitbake -c cleansstate xenclient-installer-image.
Before attempting MACHINE=openxt-installer bitbake -c cleanall xenclient-installer-image and rebuilding as this would take time. I do want to see if anyone has any idea why these QA errors are coming.
The error log is as follows:
NOTE: recipe xenclient-installer-image-1.0-r16: task do_image_qa: Started
ERROR: xenclient-installer-image-1.0-r16 do_image_qa: /tmp/vmlinux-5.10.49.5KTKt1r1 at /home/build/openxt/build-a/tmp-glibc/work-shared/openxt-installer/kernel-source/scripts/extract-sys-certs.pl line 28.
ERROR: xenclient-installer-image-1.0-r16 do_image_qa: QA errors found whilst validating image: xenclient-installer-image-openxt-installer-20220203091611
Image QA function do_image_qa_module_sigs failed

ERROR: Logfile of failure stored in: /home/build/openxt/build-a/tmp-glibc/work/openxt_installer-oe-linux/xenclient-installer-image/1.0-r16/temp/log.do_image_qa.14407
NOTE: recipe xenclient-installer-image-1.0-r16: task do_image_qa: Failed
ERROR: Task (/home/build/openxt/build-a/layers/xenclient-oe/recipes-core/images/xenclient-installer-image.bb:do_image_qa) failed with exit code '1'
NOTE: Running task 4205 of 4679 (/home/build/openxt/build-a/layers/xenclient-oe/recipes-core/images/xenclient-installer-image.bb:do_vm_common)
NOTE: recipe xenclient-installer-image-1.0-r16: task do_vm_common: Started
NOTE: recipe xenclient-installer-image-1.0-r16: task do_vm_common: Succeeded
NOTE: Tasks Summary: Attempted 4205 tasks of which 4201 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
 /home/build/openxt/build-a/layers/xenclient-oe/recipes-core/images/xenclient-installer-image.bb:do_image_qa
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
build-manifest command "MACHINE=openxt-installer bitbake xenclient-installer-image" failed.
I'll continue to investigate before I take action but thought it was right to seek help.
Thanks and Regards
Appreciate and am Grateful for the help.
Vaishali.

[Jason Andryuk]

On Thu, Feb 3, 2022 at 4:51 AM Vaishali Dhakate
<vaishali...@networkmarvels.com> wrote:
>
> Dear openxt Team,
> I am trying to build openxt dunfell branch, which I had already built, which I was working on about 2 weeks ago, had been working on synchronizer and openxt device : an upgrade to the openxt device from the Synchronizer.
> Till I realized a repo sync was needed to get all the changes from the updated code.
> Now I am facing an error related to image_qa_module_sigs failed.
> I believe I have already cleaned the xenclient-image-installer and I am rebuilding this.
> Like so, MACHINE=openxt-installer bitbake -c cleansstate xenclient-installer-image.
> Before attempting MACHINE=openxt-installer bitbake -c cleanall xenclient-installer-image and rebuilding as this would take time. I do want to see if anyone has any idea why these QA errors are coming.
> The error log is as follows:
> NOTE: recipe xenclient-installer-image-1.0-r16: task do_image_qa: Started
> ERROR: xenclient-installer-image-1.0-r16 do_image_qa: /tmp/vmlinux-5.10.49.5KTKt1r1 at /home/build/openxt/build-a/tmp-glibc/work-shared/openxt-installer/kernel-source/scripts/extract-sys-certs.pl line 28.
> ERROR: xenclient-installer-image-1.0-r16 do_image_qa: QA errors found whilst validating image: xenclient-installer-image-openxt-installer-20220203091611
> Image QA function do_image_qa_module_sigs failed

Hi, Vaishali,

tl;dr: Try `MACHINE=openxt-installer bitbake -c cleansstate
linux-openxt` to clean the kernel and then re-run
`MACHINE=openxt-installer bitbake xenclient-installer-image`.

xenclient-oe/classes/image-qa-module-sigs.bbclass is trying to verify
that the kernel module (/lib/modules/$ver/**.ko) have valid signatures
according to the public key embedded in /boot/vmlinuz.

/tmp/vmlinux-5.10.49.5KTKt1r1 was uncompressed from bzImage. Most
images have /boot/vmlinuz, but the installer is special and doesn't
put the kernel in /boot. So image-qa-modules-sigs looks for
installer's kernel at ${DEPLOY_DIR_IMAGE}/bzImage. Then
extract-sys-certs.pl failed for some reason on line 28.

26: open FD, "objdump -h $vmlinux |" || die $vmlinux;
27: my @lines = <FD>;
28: close(FD) || die $vmlinux;

It's unexpected that it failed in close().

The file should still be there, so you could try `object -h
/tmp/vmlinux-5.10.49.5KTKt1r1` manually to see if you get output.

But just cleaning and rebuilding linux-openxt should help rebuild the
installer. Let me know if it doesn't.

Regards,
Jason

[Vaishali Dhakate]

Hi, Jason and Dear openxt Team,
I had to bitbake -c cleanall for the xenclient-installer-image. As the image didn't pick the linux signatures correctly on doing a bitbake -c cleansstate xenclient-installer-image.
My linux-openxt was already "bitbake -c cleanall linux-openxt" and build successfully before this error.
So After the "bitbake -c cleanall xenclient-installer-image", this  build was successful.
Except now the openxt is not coming to the UI and not even shows "Sealing Config", It is going into a loop at the stage wherein during the firstboot init script should be run and we see the "Sealing Config" dialog. So the sealing config never shows, instead it keeps rebooting from the logs of finding/searching for the correct SINIT bin file while the DELL logo is still there etc.
But I had a backup of the previous dunfell build installer that I built before this repo sync; which works completely all right.

So I tried Openxt with console that shows "Sealing Config" and boots to openxt UI correctly. Also I believe i saw kernel panic and not finding the correct root and mapper/xencliet-boot not there kind of logs.

So openxt without a console never boots fully right now.

I need help with this.

Thanks and Regards.

Vaishali.

[Jason Andryuk]

On Wed, Feb 9, 2022 at 12:08 AM Vaishali Dhakate
<vaishali...@networkmarvels.com> wrote:
>
> Hi, Jason and Dear openxt Team,
> I had to bitbake -c cleanall for the xenclient-installer-image. As the image didn't pick the linux signatures correctly on doing a bitbake -c cleansstate xenclient-installer-image.
> My linux-openxt was already "bitbake -c cleanall linux-openxt" and build successfully before this error.
> So After the "bitbake -c cleanall xenclient-installer-image", this build was successful.
> Except now the openxt is not coming to the UI and not even shows "Sealing Config", It is going into a loop at the stage wherein during the firstboot init script should be run and we see the "Sealing Config" dialog. So the sealing config never shows, instead it keeps rebooting from the logs of finding/searching for the correct SINIT bin file while the DELL logo is still there etc.
> But I had a backup of the previous dunfell build installer that I built before this repo sync; which works completely all right.
> So I tried Openxt with console that shows "Sealing Config" and boots to openxt UI correctly. Also I believe i saw kernel panic and not finding the correct root and mapper/xencliet-boot not there kind of logs.
> So openxt without a console never boots fully right now.

It's hard to say what is going on. Going in via console mode and
seeing what you can in /var/log/messages is the best you can do. If a
regular boot doesn't make it that far, then it's hard to say what is
going wrong.

My hunch is that your build was corrupted by your earlier out-of-space
error. While you have cleaned enough to complete the build, something
in the build may still have been corrupted. Without more info, you
may want to just delete tmp-glibc and sstate and restart the build.
It'll take a while, but assuming you have enough free space, it should
produce a working image.

Having said that, I have seen rebooting after listing the SINIT
modules. On some machines, it was usually seen when trying to boot
into console mode. When I was doing some working launching under
OVMF, I made this change:
https://github.com/jandryuk/tboot/commit/0531ebe7d82ad2321a28e496a3e385c73be59081

It might be related, and should be ported over to the xenclient-oe
tboot patch queue.

Regards,
Jason