Entering emergency mode: ova VM SLES15SP4

[Jan Robinson]

Hi All

Booting to emergency mode: ova VM SLES15SP4

Deploy Environment:
OVA image installed to VMWare.
Image kernel 5.14.21-150400.11-default

Build Environment:
Kiwi build server SLES15SP3
python3-kiwi-9.24.29-1.1.x86_64
kiwi-pxeboot-9.24.29-1.1.x86_64
kiwi-systemdeps-core-9.24.29-1.1.x86_64
kiwi-tools-9.24.29-1.1.x86_64

repo for build: (used in ova image)
dracut-kiwi-lib-9.24.29-1.1.x86_64.rpm
dracut-kiwi-live-9.24.29-1.1.x86_64.rpm
dracut-kiwi-oem-dump-9.24.29-1.1.x86_64.rpm
dracut-kiwi-oem-repart-9.24.29-1.1.x86_64.rpm
dracut-kiwi-overlay-9.24.29-1.1.x86_64.rpm

Notes:
Two UUIDs are assigned in /dev/disk/by-uuid for LABEL: BOOT /dev/sda1
- one is the real on listed by 'blkid –o udev'
- one from grub.cfg

This causes a 'device start' timeout.
That causes the emergency mode.
Fixing the /etc/fstab to remove the offending UUID for /boot with /dev/sda1 and reboot works.
Except that after the image to boot is selected from the initial boot selection screen(where you can press 'e')

This appears:
error: ../../grub—core/commands/search.c : 296 : no such device:
d63b27d9-7c54-443f-bdff-80726544fbe5              <--- uuid from grub.cfg
Loading Linux 5. 14.21-150400.11-default  …
Loading initial ramdisk
Press any key to continue...


=======================================
Config.xml
    </preferences>
    <preferences profiles="OVA">
        <type image="oem" kernelcmdline="splash biosdevname=0" bootpartition="true" bootpartsize="1024" bootfilesystem="ext4" filesystem="xfs" format="ova" initrd_system="dracut">
            <bootloader  name="grub2" console="console"/>
            <size unit="G" additive="true">5</size>
            <oemconfig>
                <oem-systemsize>8192</oem-systemsize>
                <oem-resize>false</oem-resize>
            </oemconfig>
            <machine memory="1024" guestOS="suse" HWversion="15" ovftype="vmware">
                <vmdisk id="0" controller="pvscsi"/>
                <vmdvd id="0" controller="ide"/>
                <vmnic driver="vmxnet3" interface="0" mode="bridged"/>
            </machine>
        </type>
    </preferences>
=======================================
Deployed VM:

/boot/grub2/grub.cfg
submenu 'Advanced options for ALD-SLES15SP4' --hotkey=1 $menuentry_id_option 'gnulinux-advanced-69d1c25e-4e30-4868-9c62-537c15923f10' {
        menuentry 'ALD-SLES15SP4, with Linux 5.14.21-150400.11-default' --hotkey=2 --class ALD_sles15sp4 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-5.14.21-150400.11-default-advanced-69d1c25e-4e30-4868-9c62-537c15923f10' {
                load_video
                set gfxpayload=keep
                insmod gzio
                insmod part_msdos
                insmod ext2
                search --no-floppy --fs-uuid --set=root d63b27d9-7c54-443f-bdff-80726544fbe5  <-- problem UUID
                echo        'Loading Linux 5.14.21-150400.11-default ...'
                linux        /vmlinuz-5.14.21-150400.11-default root=UUID=69d1c25e-4e30-4868-9c62-537c15923f10  splash biosdevname=0
                echo        'Loading initial ramdisk ...'
                initrd        /initrd-5.14.21-150400.11-default
        }

Problem: Device start timeout on searching for a nonexistent UUID
The searched UUID is found in grub.cfg (in the search line)

[   11.153068] localhost systemd[1]: Finished Wait for udev To Complete Device Initialization.
[  100.142683] localhost systemd[1]: dev-disk-by\x2duuid-d63b27d9\x2d7c54\x2d443f\x2dbdff\x2d80726544fbe5.device: Job dev-disk-by\x2duuid-d63b27d9\x2d7c54\x2d443f\x2dbdff\x2d80726544fbe5.device/start timed out.
[  100.143790] localhost systemd[1]: Timed out waiting for device /dev/disk/by-uuid/d63b27d9-7c54-443f-bdff-80726544fbe5.
[  100.143892] localhost systemd[1]: Dependency failed for /boot.
[  100.143997] localhost systemd[1]: Dependency failed for Local File Systems.
[  100.144087] localhost systemd[1]: Dependency failed for Early Kernel Boot Messages.
[  100.144163] localhost systemd[1]: klog.service: Job klog.service/start failed with result 'dependency'.
[  100.144243] localhost systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
[  100.144313] localhost systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
[  100.144544] localhost systemd[1]: Dependency failed for Apply Kernel Variables for 5.14.21-150400.11-default from /boot.
[  100.144711] localhost systemd[1]: boot-sysctl.service: Job boot-sysctl.service/start failed with result 'dependency'.
[  100.144797] localhost systemd[1]: boot.mount: Job boot.mount/start failed with result 'dependency'.
[  100.144868] localhost systemd[1]: dev-disk-by\x2duuid-d63b27d9\x2d7c54\x2d443f\x2dbdff\x2d80726544fbe5.device: Job dev-disk-by\x2duuid-d63b27d9\x2d7c54\x2d443f\x2dbdff\x2d80726544fbe5.device/start failed with result 'timeout'.

blkid –o udev:

ID_FS_LABEL=BOOT
ID_FS_LABEL_ENC=BOOT
ID_FS_UUID=632ec612-c504-44fd-b8d5-0b2a53372fbc
ID_FS_UUID_ENC=632ec612-c504-44fd-b8d5-0b2a53372fbc
ID_FS_BLOCK_SIZE=4096
ID_FS_TYPE=ext4
ID_FS_PARTUUID=808313ae-01

/etc/fstab will contain the UUID not found by 'lbkid –o uuid' to mount /boot.
Two UUIDs for /dev/sda1

/dev/disk/by-uuid:
total 0
lrwxrwxrwx 1 root root  9 Mar 22 10:16 2022-03-22-09-16-07-0 -> ../../sr0
lrwxrwxrwx 1 root root 10 Mar 22 10:18 577d3913-f6a2-4322-b6d8-5d560d0c5dd1 -> ../../sdb1
lrwxrwxrwx 1 root root 10 Mar 22 10:16 632ec612-c504-44fd-b8d5-0b2a53372fbc -> ../../sda1
lrwxrwxrwx 1 root root 10 Mar 22 10:16 69d1c25e-4e30-4868-9c62-537c15923f10 -> ../../sda2
lrwxrwxrwx 1 root root 10 Mar 22 10:16 d63b27d9-7c54-443f-bdff-80726544fbe5 -> ../../sda1 <- uuid used in fstab for /boot as listed in grub.cfg in the search line.

/etc/fstab
UUID=69d1c25e-4e30-4868-9c62-537c15923f10 / xfs defaults 0 1
UUID=d63b27d9-7c54-443f-bdff-80726544fbe5 /boot ext4 defaults 0 0

Thank you for the help.
Jan

[Marcus Schäfer]

Hi,

> Booting to emergency mode: ova VM SLES15SP4
> Deploy Environment:
> OVA image installed to VMWare.
> Image kernel 5.14.21-150400.11-default
> Build Environment:
> Kiwi build server SLES15SP3
> python3-kiwi-9.24.29-1.1.x86_64
> kiwi-pxeboot-9.24.29-1.1.x86_64
> kiwi-systemdeps-core-9.24.29-1.1.x86_64
> kiwi-tools-9.24.29-1.1.x86_64
> repo for build: (used in ova image)
> dracut-kiwi-lib-9.24.29-1.1.x86_64.rpm
> dracut-kiwi-live-9.24.29-1.1.x86_64.rpm
> dracut-kiwi-oem-dump-9.24.29-1.1.x86_64.rpm
> dracut-kiwi-oem-repart-9.24.29-1.1.x86_64.rpm
> dracut-kiwi-overlay-9.24.29-1.1.x86_64.rpm
> Notes:
> Two UUIDs are assigned in /dev/disk/by-uuid for LABEL: BOOT /dev/sda1
> - one is the real on listed by 'blkid –o udev'
> - one from grub.cfg
> This causes a 'device start' timeout.
> That causes the emergency mode.

Thanks for the details on the environment

> Fixing the /etc/fstab to remove the offending UUID for /boot with
> /dev/sda1 and reboot works.

The question is why is the fstab UUID wrong. From the build of your image
you should have among others one disk file ending with .raw and one with .ova
The latter is created by kiwi calling ovftool from the formerly created .raw
disk file.

Can you check the partition IDs of the .raw disk file as follows:

kpartx -a file.raw
blkid /dev/mapper/loop0pX

where X is the number of the partition, 1, 2, 3 ...

blkid lists among others the UUID. Does this UUID match with the settings
in the /etc/fstab file ?

If not can you provide the build log file from your kiwi-ng call ?

If yes I assume the conversion into the ovf via ovftool might have
changed the IDs ? just guessing

Thanks

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Jan Robinson]

Hello Marcus.

Thank you for the reply.

Two builds were done today to compare. SP3 and SP4

The requested kiwi-ng log is attached as well ad the 'blkid' of the loop device from the .raw files

=============

SLES15SP3

 # cat /etc/os-release

NAME="SLES"
VERSION="15-SP3"
VERSION_ID="15.3"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP3"

 # cat /etc/fstab

UUID=2e57fbdc-e372-4a68-8605-ce00d4b8d7ea / xfs defaults 0 1
UUID=357398d4-e496-4b14-828e-4bc033f669c0 /boot ext4 defaults 0 0
/dev/sdb1       /app    auto    defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig    02

 # blkid|grep OO

/dev/sda1: LABEL="BOOT" UUID="357398d4-e496-4b14-828e-4bc033f669c0" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6ce1e6fa-01"
/dev/sda2: LABEL="ROOT" UUID="2e57fbdc-e372-4a68-8605-ce00d4b8d7ea" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="6ce1e6fa-02"

ltvmwcmdbtest0176:~ # grep -i uuid /boot/grub2/grub.cfg

          search --no-floppy --fs-uuid --set=root  357398d4-e496-4b14-828e-4bc033f669c0
          search --no-floppy --fs-uuid --set=root 357398d4-e496-4b14-828e-4bc033f669c0
        linux   /vmlinuz-5.3.18-59.27-default root=UUID=2e57fbdc-e372-4a68-8605-ce00d4b8d7ea  splash biosdevname=0
                  search --no-floppy --fs-uuid --set=root  357398d4-e496-4b14-828e-4bc033f669c0
                  search --no-floppy --fs-uuid --set=root 357398d4-e496-4b14-828e-4bc033f669c0
                linux   /vmlinuz-5.3.18-59.27-default root=UUID=2e57fbdc-e372-4a68-8605-ce00d4b8d7ea  splash biosdevname=0
                  search --no-floppy --fs-uuid --set=root  357398d4-e496-4b14-828e-4bc033f669c0
                  search --no-floppy --fs-uuid --set=root 357398d4-e496-4b14-828e-4bc033f669c0
                linux   /vmlinuz-5.3.18-59.27-default root=UUID=2e57fbdc-e372-4a68-8605-ce00d4b8d7ea


kiwisrv# kpartx -a /global/local/ova-result/ALD-SLES15SP3.x86_64-2.4.0.raw

kiwiserv# blkid /dev/mapper/loop1p1
/dev/mapper/loop1p1: LABEL="BOOT" UUID="357398d4-e496-4b14-828e-4bc033f669c0" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="6ce1e6fa-01"

kiwiserv# blkid /dev/mapper/loop1p2
/dev/mapper/loop1p2: LABEL="ROOT" UUID="2e57fbdc-e372-4a68-8605-ce00d4b8d7ea" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="6ce1e6fa-02"


--------------------------------

SLES15SP4

 # cat /etc/os-release
NAME="SLES"
VERSION="15-SP4"
VERSION_ID="15.4"

 # cat /etc/fstab

UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7 / xfs defaults 0 1
UUID=8c4bc876-1dd7-4b29-9a70-3313bb8f802e /boot ext4 defaults 0 0
/dev/sdb1       /app    auto    defaults,nofail,x-systemd.requires=cloud-init.service,comment=cloudconfig    02

 # blkid|grep OO

/dev/sda2: LABEL="ROOT" UUID="b43a7ca6-6658-47d3-be16-25a7f9e026f7" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="bda695a1-02"
/dev/sda1: LABEL="BOOT" UUID="42e68baa-143d-4cb9-b9c6-1063711ec8d6" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="bda695a1-01"

 # grep -i uuid /boot/grub2/grub.cfg

        search --no-floppy --fs-uuid --set=root 8c4bc876-1dd7-4b29-9a70-3313bb8f802e
        linux   /vmlinuz-5.14.21-150400.11-default root=UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7  splash biosdevname=0
                search --no-floppy --fs-uuid --set=root 8c4bc876-1dd7-4b29-9a70-3313bb8f802e
                linux   /vmlinuz-5.14.21-150400.11-default root=UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7  splash biosdevname=0
                search --no-floppy --fs-uuid --set=root 8c4bc876-1dd7-4b29-9a70-3313bb8f802e
                linux   /vmlinuz-5.14.21-150400.11-default root=UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7


kiwisrv # blkid /dev/mapper/loop0p1
/dev/mapper/loop0p1: LABEL="BOOT" UUID="8c4bc876-1dd7-4b29-9a70-3313bb8f802e" BLOCK_SIZE="4096" TYPE="ext4" PARTUUID="dfc7930c-01"

kiwisrv # blkid /dev/mapper/loop0p2
/dev/mapper/loop0p2: LABEL="ROOT" UUID="b43a7ca6-6658-47d3-be16-25a7f9e026f7" BLOCK_SIZE="512" TYPE="xfs" PARTUUID="dfc7930c-02"

Thanks so much,

Jan

[Marcus Schäfer]

Hi Jan,

> The requested kiwi-ng log is attached as well ad the 'blkid' of the
> loop device from the .raw files

Thanks much for the details. I have checked the information you provided.
So the grub.cfg says:

root=UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7

and the blkid value for the .raw disk file matches this value:

ID_FS_LABEL=ROOT
ID_FS_UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7

the fstab also contains:

UUID=b43a7ca6-6658-47d3-be16-25a7f9e026f7 / xfs defaults 0 1

This part looks clean to me. Now the part that is broken. Your
attached grub.cfg lists the following:

search --no-floppy --fs-uuid --set=root 8c4bc876-1dd7-4b29-9a70-3313bb8f802e

The ID here has to match the UUID of the boot partition. As you use
an extra boot partition (not sure why) this ID should be:

ID_FS_LABEL=BOOT
ID_FS_UUID=db085b61-291f-4aee-a527-6217897e515e

So here is a mismatch. In fact the above ID 8c4b... does not appear
at all. The same non-existing ID is also present in the fstab

I wonder where this comes from and I'm not able to reproduce this.
I build a simple image with the following type:

<type image="vmx" filesystem="xfs" bootloader="grub2" bootpartition="true" bootfilesystem="ext4"/>

And all IDs used for root and boot were correctly added. Please note
when kiwi adds the UUIDs for e.g fstab it does that by a direct blkid
call to the device present at that time. This means the ID must
have been a valid at build time

What makes me really wonder is that the broken ID
"8c4bc876-1dd7-4b29-9a70-3313bb8f802e" does not appear at all
in the produced .raw disk file

So I'm sorry I don't have a solution for you at the moment because
I cannot reproduce the issue. Maybe you can check the following:

- Is there any script code that messes with the block IDs ?
- You don't need an extra boot partition for XFS so you could also
give bootpartition="false" a try
- Can you reproduce the issue when building with the latest
kiwi version from here:
https://download.opensuse.org/repositories/Virtualization:/Appliances:/Builder/SLE_15_SP3/

Hope this helps with some next steps

Cheers,

[Jan Robinson]

Hello Markus

Thank you for the suggestions.

I have used the version you suggested. e.g. dracut-kiwi-lib|(none)|9.24.29|1.1|x86_64

The error is the same.

The kiwi-ng environment  between 15.3 and 15.4 are 100% the same apart from the repos.

The error only appears for 15.4

    <preferences profiles="OVA">
        <type image="oem" kernelcmdline="splash biosdevname=0" bootpartition="true" bootpartsize="1024" bootfilesystem="ext4" filesystem="xfs" format="ova" initrd_system="dracut">
            <bootloader  name="grub2" console="console"/>
            <size unit="G" additive="true">5</size>
            <oemconfig>
                <oem-systemsize>8192</oem-systemsize>
                <oem-resize>false</oem-resize>
            </oemconfig>
            <machine memory="1024" guestOS="suse" HWversion="15" ovftype="vmware">
                <vmdisk id="0" controller="pvscsi"/>
                <vmdvd id="0" controller="ide"/>
                <vmnic driver="vmxnet3" interface="0" mode="bridged"/>
            </machine>
        </type>
    </preferences>

I am not aware of any script that edits the grub.cfg.

Any guidance you can provide in finding such a script would help a lot.

The setting 'bootpartition=fase' make 15.3 and 15.4 hang forever during the boot process. No emergency shell.

Your help is appreciated.
Jan

[Marcus Schäfer]

Hi Jan,

> I have used the version you suggested. e.g.
> dracut-kiwi-lib|(none)|9.24.29|1.1|x86_64
>
> The error is the same.

ok thanks for taking the time to rebuild and check

> The kiwi-ng environment between 15.3 and 15.4 are 100% the same apart
> from the repos.
>
> The error only appears for 15.4

ok, as I still cannot reproduce the error you see, can we try
the following suggestion:

* I have created a suse-leap-15.4 example description here:
https://github.com/OSInside/kiwi-descriptions/tree/master/suse/x86_64/suse-leap-15.4

* I used this description as the base to reproduce your issue
but so far it builds me a good image. Can you checkout the
kiwi-descriptions git from above and rebuild the image in the
same way like I did but in your environment. I called the
following command to build the image:

sudo kiwi-ng --profile Virtual system build \
--description kiwi-descriptions/suse/x86_64/suse-leap-15.4/ \
--target-dir /tmp/mytest-15.4

I booted the image via:

qemu-kvm -m 4096 -serial stdio \
-hda /tmp/mytest-15.4/Leap-15.4_appliance.x86_64-1.15.4.raw

This just boots up to a login, user root, pwd: linux


a) If you get to that point and see the same behavior I see, we
somehow know that your description might include something that
is causing harm when it shouldn't.

b) If you can reproduce the error by just building the image
from kiwi-description without any modifications, there seems
to be a difference on the building host which is causing this.

I don't have good answers to either a) or b) but I think going
forward we need to eliminate one or the other path

I did not add the ova creation as I think the issue with the IDs
applies before

Thoughts ?

Regards,

[Jan Robinson]

Hi Markus.

A few builds a deploys have been done today. So new UUIDS

Then it dawned on me to look for both the BOOT and ROOT UUID in the build/image-root/

There it is, the wrong UUID in fstab

/global/local/ova-result/build/image-root # cat etc/fstab
UUID=05b50e20-06d6-4dae-b4e4-4834154be307 / xfs defaults 0 1
UUID=0d2fe01d-a180-4ca6-ad1a-995b25bd58fc /boot ext4 defaults 0 0

Neither UUIDs is mentioned in the kiwi-ng debug log.

Are there maybe a 'deeper' debug level during build to see why the UUID of the xx.raw boot is not used to populate fstab?

I will try out QEMU. 

Thank you again.

Jan

[Jan Robinson]

Hi Markus

Here the qemu.

Please see the last piece, the wrong UUID as in the build fstab.

qemu-kvm -nographic -m 4096  -hda /global/local/ova-result/ALD-SLES15SP4.x86_64-15.4.0.raw

         Specify the 'raw' format explicitly to remove the restrictions.
SeaBIOS (version rel-1.14.0-0-g155821a-rebuilt.opensuse.org)


iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+BFF904B0+BFEF04B0 CA00

Booting from Hard Disk...
GRUB loading.......
Welcome to GRUB!
                             GNU GRUB  version 2.06
 ┌────────────────────────────────────────────────────────────────────────────┐
 │*BMW-SLES15SP4                                                              │
 │ Advanced options for BMW-SLES15SP4                                         │
 │                                                                            │
 │                                                                            │
 │                                                                            │
 └────────────────────────────────────────────────────────────────────────────┘

      Use the ↑ and ↓ keys to select which entry is highlighted.
      Press enter to boot the selected OS, `e' to edit the commands
      before booting or `c' for a command-line.


error: ../../grub-core/commands/search.c:296:no such device:
0d2fe01d-a180-4ca6-ad1a-995b25bd58fc.
Loading Linux 5.14.21-150400.11-default ...
Loading initial ramdisk ...

Press any key to continue...

Here it hangs...

Regards,

Jan

[Marcus Schäfer]

Hi,

> Here the qemu.
> Please see the last piece, the wrong UUID as in the build fstab.
>
> qemu-kvm -nographic -m 4096 -hda
> /global/local/ova-result/ALD-SLES15SP4.x86_64-15.4.0.raw
> Specify the 'raw' format explicitly to remove the
> restrictions.

yeah man, so how can this happen ?? :)

I added ... bootpartition="true" bootpartfs="ext4" to the example
image description and rebuild it, all IDs are correct

Is there any chance that you can provide me your complete
image description as a tarball. Not only the XML file but
the complete description ?

> Use the ↑ and ↓ keys to select which entry is highlighted.
> Press enter to boot the selected OS, `e' to edit the commands
> before booting or `c' for a command-line.
> error: ../../grub-core/commands/search.c:296:no such device:
> 0d2fe01d-a180-4ca6-ad1a-995b25bd58fc.
> Loading Linux 5.14.21-150400.11-default ...
> Loading initial ramdisk ...
> Press any key to continue...
>
> Here it hangs...

Are you sure ? do you have configured the serial console in
your image and maybe your VM call does not expose it ?
'-serial stdio' in QEMU

This one is weird ;)

[Jan Robinson]

Hi Marcus

Hand in face time for me.....

I mislead myself and you to believe there cannot be anything that changed the UUIDs 

Well   root/usr/lib/dracut/modules.d/99ald-uuid-update.....

I am so sorry to have done this to you but you did steer me... a lot.

I have inherited this build from a normal VMware and Xen build to take to OVA that worked for SP3.

As soon as SP4 came, the update uuid in dracut modules.d made a stand - to show me who is boss.

So all over now, thanks you so much.

Now to find out why SP4 is actually doing this to the boot UUID and not SP3.

But the system is running and cloud-init is doing it's thing.

Happiness.

Great appreciation for your inputs/guidance!!!

Regards,

Jan

[Marcus Schäfer]

Hi Jan,

> Hand in face time for me.....

:-))

> I mislead myself and you to believe there cannot be anything that
> changed the UUIDs
>
> Well root/usr/lib/dracut/modules.d/99ald-uuid-update.....

aaaaargh :) hey but at least a good name was choosen, you know it
could have been much worse ;)

> I am so sorry to have done this to you but you did steer me... a lot.

No problem, I'm happy you found it

> But the system is running and cloud-init is doing it's thing.
> Happiness.

yay :)

Cheers,