Leap 15.5 not booting

24 views
Skip to first unread message

Gavin Lambert

unread,
Jul 30, 2025, 3:23:07 AMJul 30
to kiwi
Presumably following this change, any new images that I build now are failing to boot.

As in my previous message, I'm still targeting a Leap 15.5 system due to other constraints but using the Leap 15.6 download repository for kiwi-ng (since the 15.5 repo has been removed).

AFAIK I'm using the latest available version of the packages from that repository:
[ DEBUG   ]: 05:44:38 | system: Retrieving: dracut-kiwi-lib-10.2.30-lp156.1.1.x86_64 (kiwi) (114/117), 975.6 KiB
[ DEBUG   ]: 05:44:41 | system: Retrieving: dracut-kiwi-lib-10.2.30-lp156.1.1.x86_64.rpm [..................done (734 B/s)]
[ DEBUG   ]: 05:44:41 | system: Retrieving: dracut-kiwi-oem-repart-10.2.30-lp156.1.1.x86_64 (kiwi) (115/117), 965.8 KiB
[ DEBUG   ]: 05:44:45 | system: Retrieving: dracut-kiwi-oem-repart-10.2.30-lp156.1.1.x86_64.rpm [...................done (690 B/s)]

Screenshot 2025-07-30 185956.png
Failing to boot on this error does seem a bit of an overreaction?

Edit: after a bit of juggling to get better logs, it looks like it might be this change which actually broke it:
Screenshot 2025-07-30 191658.png
Apparently my udevadm doesn't support the lock command.

Is this a bug, or do I need to pin an older version of the dracut packages somehow to avoid future incompatibilities?  Is there a recommended way to do that if so?

Marcus Schäfer

unread,
Jul 30, 2025, 5:12:07 AMJul 30
to kiwi-...@googlegroups.com
Hi,

> As in my previous message, I'm still targeting a Leap 15.5 system due
> to other constraints but using the Leap 15.6 download repository for
> kiwi-ng (since the 15.5 repo has been removed).

ok

> Screenshot 2025-07-30 185956.png
> Apparently my udevadm doesn't support the lock command.

Wow, this is quite unexpected. When we fixed the inconsistency
caused due to missing device locking, I tested the images down
the stream of the support matrix (https://osinside.github.io/kiwi/overview.html#system-requirements) and also tested Leap15 as I myself still
use it. The version of Leap I use is however 15.6 and not 15.5.
On 15.6 there is:

Repository : openSUSE Leap 15.6 - Updates from SUSE Linux Enterprise
Name : udev
Version : 254.25-150600.4.40.1

And you are saying the udev version on 15.5 has no lock command ?

> Failing to boot on this error does seem a bit of an overreaction?

Well not really, if you haven't experienced issues during resize,
dump or fsck when using a kiwi installation ISO it was mostly luck
as we really missed to add proper device locking to avoid inconsistencies
between udev actions and kiwi initrd code actions to happen at the same
time.

> Is this a bug,

I don't think it's a bug, but I can see the pain you are having here

> or do I need to pin an older version of the dracut
> packages somehow to avoid future incompatibilities? Is there a
> recommended way to do that if so?

* The most simple way to overcome this would be to move to 15.6

* The suggested pin to older version of the kiwi dracut module package(s)
is also a solution but that would disconnect you from future changes

* You can consider to branch off from https://build.opensuse.org/package/show/system%3Asystemd/systemd-suse and build that for Leap 15.5. Next add the
repo to your build and this should lead to a system with an updated
systemd including udev

* There is also a "solution" in code which you could consider as
follows:

add a config.sh script (or use it if you already have one) with
the following content:

---
# mock udevadm lock...
cp /usr/bin/udevadm /usr/bin/udevadm.orig
cat >/usr/bin/udevadm <<- EOF
#!/bin/bash
test "$1" = "lock" || exec /usr/bin/udevadm.orig "$@"
EOF
chmod 755 /usr/bin/udevadm

cat >/etc/dracut.conf.d/99-mock-lock.conf <<- EOF
install_items+=" /usr/bin/udevadm.orig "
EOF
---

add a pre_disk_sync.sh script with the following content:

---
mv /usr/bin/udevadm.orig /usr/bin/udevadm
rm /etc/dracut.conf.d/99-mock-lock.conf
---

Hope this helps ?

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------
signature.asc

Gavin Lambert

unread,
Jul 31, 2025, 12:32:59 AMJul 31
to kiwi
On Wednesday, July 30, 2025 at 9:12:07 PM UTC+12 Marcus wrote:
On 15.6 there is:

Repository : openSUSE Leap 15.6 - Updates from SUSE Linux Enterprise
Name : udev
Version : 254.25-150600.4.40.1

And you are saying the udev version on 15.5 has no lock command ?

Correct; on 15.5, udev is version 249.17-150400.8.46.1 and it doesn't appear to be a thing there.  AFAICT it looks like systemd 251 is when that was introduced.
 
* The most simple way to overcome this would be to move to 15.6

I agree, but another library I'm using gets very grumpy at that idea and it's currently in the "too hard no time" basket, though issues like this do add to the other side of the scales.

* You can consider to branch off from https://build.opensuse.org/package/show/system%3Asystemd/systemd-suse and build that for Leap 15.5. Next add the
repo to your build and this should lead to a system with an updated
systemd including udev

That also sounds a bit scary. :)

add a config.sh script (or use it if you already have one) with
the following content:

Thanks for this, it makes sense to me.  Though as written that would ignore the command entirely (so not doing any resizing).  I rewrote it to do this instead, which in theory should be the equivalent:

if [ "$1" = "lock" ]; then
    # assuming 'udevadm lock --device "$device" REAL_COMMAND...'
    shift 2
    exec flock -x "$@"
else
    exec /usr/bin/udevadm.orig "$@"
fi

One additional caveat is that I also had to fix up the udevd link, because that's the same binary for some weird reason:
    sudo ln -sfr /usr/bin/udevadm.orig /usr/lib/systemd/systemd-udevd

This seemed to work fine (not sure if the lock actually does anything absent additional support in systemd).  In hindsight it might have been easier to just s/udevadm lock --device/flock -x/ on the kiwi dracut scripts instead.

Possibly of note:
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@237(source): initialize
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@14(initialize): declare root=block:/dev/disk/by-uuid/ef620772-e957-40c9-bc06-1e7d1629d447
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@15(initialize): local profile=/.profile
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@16(initialize): local partition_ids=/config.partids
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@18(initialize): test -f /.profile
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@20(initialize): warn 'Settings from the kiwi description will be ignored'
Aug 01 02:05:41 localhost dracut-pre-mount[697]: //lib/dracut-lib.sh@85(warn): echo 'Warning: Settings from the kiwi description will be ignored'
Aug 01 02:05:41 localhost dracut-pre-mount[697]: Warning: Settings from the kiwi description will be ignored
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@22(initialize): test -f /.profile
Aug 01 02:05:41 localhost dracut-pre-mount[697]: ///lib/dracut/hooks/pre-mount/20-kiwi-repart-disk.sh@22(initialize): import_file /.profile
(It did then load values from .profile, despite printing that warning...)

add a pre_disk_sync.sh script with the following content:

This part I don't think makes sense for me since I'm also installing a dracut conf file to keep kiwi-repart loaded in subsequent initrd builds (as mentioned in a prior thread), but I can see how it would be useful otherwise.

Marcus Schäfer

unread,
Jul 31, 2025, 3:59:04 AMJul 31
to kiwi-...@googlegroups.com
Hi,

> Thanks for this, it makes sense to me. Though as written that would
> ignore the command entirely (so not doing any resizing).

Yeah I was short thinking this, you are right

> I rewrote it
> to do this instead, which in theory should be the equivalent:
> if [ "$1" = "lock" ]; then
> # assuming 'udevadm lock --device "$device" REAL_COMMAND...'
> shift 2
> exec flock -x "$@"
> else
> exec /usr/bin/udevadm.orig "$@"
> fi

Yes that makes more sense

> One additional caveat is that I also had to fix up the udevd link,
> because that's the same binary for some weird reason:

Interesting on my 15.6 Leap there is no udevd link, probably that
has changed there as well

> This seemed to work fine (not sure if the lock actually does anything
> absent additional support in systemd).

There is one difference and also the reason why I did not use flock.
Since udev is event based, there is an event in the case of a conflicting
action on a device. If you let udevadm do the lock it also postpones
the associated event for a later execution. I believe if you only set
a device lock the actual action for udev might be lost. Again I'm not
sure and did not dig deeper into the topic.

> (It did then load values from .profile, despite printing that
> warning...)

Yes because I made a stupid shell coding error. It's almost a bit
embarrassing but something like this is not correct

test -f ${profile} || \
warn "No profile setup found"
warn "Settings from the kiwi description will be ignored"

The second warn is always executed. This has to change to

if ! test -f ${profile}; then
warn "No profile setup found"
warn "Settings from the kiwi description will be ignored"
fi

I will fix it. Thanks for catching it

> This part I don't think makes sense for me since I'm also installing a
> dracut conf file to keep kiwi-repart loaded in subsequent initrd builds
> (as mentioned in a prior thread), but I can see how it would be useful
> otherwise.

Yes if you keep the kiwi resizer in the system, you are
right. Unfortunately this will then keep you with the problem
of the .orig symlinks.

If you install updates I believe that will drop your shell
script version of udevadm and you would have to restore it.
I have no solution for this problem atm

Hmm...

maybe I should add a patch that allows for udevadm lock and
a fallback with flock in case that doesn't work. Let me check
that part
signature.asc

Marcus Schäfer

unread,
Jul 31, 2025, 4:27:20 AMJul 31
to kiwi-...@googlegroups.com
Hi,

> In hindsight it might have been
> easier to just s/udevadm lock --device/flock -x/ on the kiwi dracut
> scripts instead.

Please find the following PRs:

https://github.com/OSInside/kiwi/pull/2868
https://github.com/OSInside/kiwi/pull/2869

I believe this should fix the issue and allows you to continue
without messing with the system :)

Would be great if you can take a look

Thanks much
signature.asc

Gavin Lambert

unread,
Jul 31, 2025, 8:47:51 PMJul 31
to kiwi
On Thursday, July 31, 2025 at 8:27:20 PM UTC+12 Marcus wrote:
Please find the following PRs:

https://github.com/OSInside/kiwi/pull/2868
https://github.com/OSInside/kiwi/pull/2869

Thanks for that, LGTM.  I think I'm going to bite the bullet and upgrade to Leap 15.6 in the end though. 

Marcus Schäfer

unread,
Aug 1, 2025, 5:57:50 AMAug 1
to kiwi-...@googlegroups.com
Hi,

> Please find the following PRs:
> [1]https://github.com/OSInside/kiwi/pull/2868
> [2]https://github.com/OSInside/kiwi/pull/2869
>
> Thanks for that, LGTM.

Thanks, they are merged now. v10.2.31 should fix your issue.
I submitted to Virt:App:Builder project if you like to test
the packages. Submission to TW will follow soon

> I think I'm going to bite the bullet and
> upgrade to Leap 15.6 in the end though.

That might solve also other issues... or create some new one ;)

If there is more we can help with, just let us know
signature.asc

Gavin Lambert

unread,
Aug 4, 2025, 2:52:29 AMAug 4
to kiwi
On Friday, August 1, 2025 at 12:47:51 PM UTC+12 Gavin Lambert wrote:
Thanks for that, LGTM.  I think I'm going to bite the bullet and upgrade to Leap 15.6 in the end though. 

I'm trying out Leap 15.6 now and while it's an improvement (the oem disk image boots fine without my evil hacks), unfortunately the oem install image still fails to boot:
Screenshot 2025-08-04 184242.png
I didn't catch the top of the log but it looks like the main culprit is a stray install prefix from the root parameter:
Screenshot 2025-08-04 183749.png

FWIW the installer cmdline has cdinst=1 splash=silent quiet rd.kiwi.oem.force_resize loglevel=0 root=install:CDLABEL=KATO_INSTALL (where KATO_INSTALL is my preferences/type/volid, and is the label on the install stick).  I tried removing the force_resize but it didn't help.

Gavin Lambert

unread,
Aug 4, 2025, 3:27:09 AMAug 4
to kiwi
On Monday, August 4, 2025 at 6:52:29 PM UTC+12 I wrote:
I didn't catch the top of the log but it looks like the main culprit is a stray install prefix from the root parameter:

My best guess for the culprit is that previous to this the repart module wasn't installed in the install iso image because the partids file wasn't present?

And this leaves the install: prefix present, which the rest of the function doesn't cope with.

Marcus Schäfer

unread,
Aug 4, 2025, 5:51:41 AMAug 4
to kiwi-...@googlegroups.com
Hi,

> I'm trying out Leap 15.6 now and while it's an improvement (the oem
> disk image boots fine without my evil hacks), unfortunately the oem
> install image still fails to boot:
> Screenshot 2025-08-04 184242.png

Something is broken with the root= assignment

"Storage device install:/dev/disk/by-label/KATO_INSTALL did not appear"

That is very weird and wrong and can imho only happen if you have
a static root= assignment in your kiwi image description ?

> FWIW the installer cmdline has cdinst=1 splash=silent quiet
> rd.kiwi.oem.force_resize loglevel=0 root=install:CDLABEL=KATO_INSTALL
> (where KATO_INSTALL is my preferences/type/volid, and is the label on
> the install stick). I tried removing the force_resize but it didn't
> help.

Does this mean your image description has something like:

<type ... kernelcmdline=" ... root=install:CDLABEL=KATO_INSTALL ..."/>

This would explain it, because if you explicitly set root= in the kernel
cmdline this information is taken into account for the bootloader of all
images that are being built. Meaning the same root= information is passed
to the kernel for the install ISO as well as for the image that got installed.
While the information is correct for the install ISO it will be
wrong for the boot of the actually installed image.

Setting root= should not be needed.

I guess you want an alternative volume label. You can achieve this
by setting:

<type ... volid="KATO_INSTALL"/>

I adapted our leap based integration test to achieve this. You can
steal from my image description here:

https://github.com/OSInside/kiwi/blob/main/build-tests/x86/leap/test-image-disk/appliance.kiwi#L19

and the actual image build you can find here:

https://download.opensuse.org/repositories/Virtualization:/Appliances:/Images:/Testing_x86:/leap/images/iso/kiwi-test-image-disk.x86_64.iso

isoinfo -d -i kiwi-test-image-disk.x86_64.iso
...
Volume id: KATO_INSTALL
...

I'm still guessing, but I hope I'm guessing right. It would be
really an issue if that root device setup would be broken. I was
not able to reproduce it though.

Thanks
signature.asc

Gavin Lambert

unread,
Aug 4, 2025, 6:51:41 PMAug 4
to kiwi
On Monday, August 4, 2025 at 9:51:41 PM UTC+12 Marcus wrote:
That is very weird and wrong and can imho only happen if you have
a static root= assignment in your kiwi image description ?

> FWIW the installer cmdline has cdinst=1 splash=silent quiet
> rd.kiwi.oem.force_resize loglevel=0 root=install:CDLABEL=KATO_INSTALL
> (where KATO_INSTALL is my preferences/type/volid, and is the label on
> the install stick). I tried removing the force_resize but it didn't
> help.

Does this mean your image description has something like:

<type ... kernelcmdline=" ... root=install:CDLABEL=KATO_INSTALL ..."/>

No, this root parameter is what kiwi generates itself for the installiso image.

You can see this generating it and this parsing it.

There is no error in my script, this is totally a bug in kiwi.  Try it yourself, you just need any spec that uses image="oem" installiso="true" and installs both dracut-kiwi-oem-repart and dracut-kiwi-oem-dump, then try to boot the install iso.  This is a pretty standard configuration, I think.

As noted, I'm pretty sure this is because dracut-kiwi-oem-repart doesn't expect this root parameter format because it used to never be installed in the installiso due to the missing partid file, but with that linked change, that changed.

So the fix would either be to make dracut-kiwi-oem-repart recognise that root format and ignore it (probably best) or make it otherwise smart enough to not install the repart module in the installiso in the first place (possibly fragile if someone wants to use a custom module instead).

FWIW, this is my complete <type> entry but I'm pretty sure it's just the things above that are needed to break it:

        <type image="oem" primary="true" filesystem="ext4" firmware="uefi"
              volid="KATO_INSTALL" installiso="true" installboot="install"
              kernelcmdline="splash=silent quiet rd.kiwi.oem.force_resize">
            <bootloader name="grub2" timeout="3"/>
            <oemconfig>
                <oem-reboot-interactive>true</oem-reboot-interactive>
                <oem-device-filter>/dev/ram</oem-device-filter>
                <oem-multipath-scan>false</oem-multipath-scan>
            </oemconfig>
        </type>

Gavin Lambert

unread,
Aug 5, 2025, 1:30:26 AMAug 5
to kiwi
On Tuesday, August 5, 2025 at 10:51:41 AM UTC+12 Gavin Lambert wrote:
As noted, I'm pretty sure this is because dracut-kiwi-oem-repart doesn't expect this root parameter format because it used to never be installed in the installiso due to the missing partid file, but with that linked change, that changed.

Actually that might not be the right reason; after closer inspection it does appear the partid file gets installed into the installiso too.  So I guess there was some other reason this previously worked?  It's definitely this point that's failing, though; specifically getting trapped forever inside wait_for_storage_device (after printing some lsblk errors as shown above).
 
So the fix would either be to make dracut-kiwi-oem-repart recognise that root format and ignore it (probably best) or make it otherwise smart enough to not install the repart module in the installiso in the first place (possibly fragile if someone wants to use a custom module instead).

I made a PR which at least seems to fix the failure to boot, but there are still some lsblk errors displayed briefly just before it shows the normal dump confirmation menu.  I managed to trace them back to another oddity in the code:

Screenshot 2025-08-05 165528.png
kiwi-dump-image calls the initialize function, which isn't defined in that module so it calls across to kiwi-repart-disk instead.  This seems dubious, what if both weren't installed?  What if another module had been sourced in between with its own initialize function?  (Actually the conditional library includes at the top don't seem entirely safe in that regard either.)

Screenshot 2025-08-05 165755.png
Which then prints some of the lsblk errors that I was previously seeing from kiwi-repart-disk (unsurprisingly).  Fortunately it doesn't then get stuck forever in wait_for_storage_device like kiwi-repart-disk did, but moves onto the code that expects the install: prefix and parses it properly.

Marcus Schäfer

unread,
Aug 5, 2025, 5:27:37 AMAug 5
to kiwi-...@googlegroups.com
Hi,

> No, this root parameter is what kiwi generates itself for the
> installiso image.

ok, good I had hoped you would say that

> As noted, I'm pretty sure this is because dracut-kiwi-oem-repart
> doesn't expect this root parameter format because it used to never be
> installed in the installiso due to the missing partid file, but with

I was wrapping my head around why I cannot reproduce this in my
integration test image but I think I know what happpened. You told
me that the repart code is called in the install ISO and I thought
how can this happen because when kiwi creates the initrd for
the purpose of the install ISO it does not include it.

Then I realized you told me that you maintain an etc/dracut.conf.d/...conf
file to ensure that the kiwi-repart module stays in the initrd
of the system. Is that still correct ?

If so this .conf file of course also has an impact at build time
when kiwi calls dracut to produce the initrd for the install ISO.
That's how the kiwi-repart module find its way to the install ISO
where it does not make any sense and causes issues.

The PR you have made partially circumvents this issue but I
think we should fix this in the kiwi python code and make sure
that the dracut call never includes the kiwi-repart module when
building the initrd for the install ISO.

Can you confirm this assumption ?

Thanks and sorry for the trouble
signature.asc

Gavin Lambert

unread,
Aug 5, 2025, 11:18:53 PMAug 5
to kiwi
On Tuesday, August 5, 2025 at 9:27:37 PM UTC+12 Marcus wrote:
Then I realized you told me that you maintain an etc/dracut.conf.d/...conf
file to ensure that the kiwi-repart module stays in the initrd
of the system. Is that still correct ?

Ah!  My apologies, yes, you're right.  I mentioned this in passing in this thread and in detail in the previous thread, but I currently do have a root/etc/dracut.conf.d/99-kiwi.conf file with the following content:
add_dracutmodules+=" kiwi-repart "
install_optional_items+=" /.profile /config.partids "

I can confirm that if I remove this file, the repart module (and partids file) is not installed into the installiso and the problem goes away.  I can probably live without this now that the repartition is happening more reliably on first boot thanks to the other fixes (and using 15.6), though it still seems like a nice idea to be able to retain that functionality if the disk is expanded further later (either for a VM or if cloned to a larger physical disk) after a subsequent dracut run.

The PR you have made partially circumvents this issue but I
think we should fix this in the kiwi python code and make sure
that the dracut call never includes the kiwi-repart module when
building the initrd for the install ISO.

I think it would be better to do this in the repart code (either the hook itself as I did or perhaps in the module-setup.sh if possible to detect it's building for an iso), rather than directly in the python code, as that feels less flexible if someone wanted to create an alternative version of the module, or a similar module that they also only want in the installer.  Or perhaps some kind of mode attribute on <preferences><type><initrd>?  Though I prefer auto-detection where feasible.

Doing it in module-setup might resolve the clobbering initialize functions as well -- it looks like dump is expecting to run the one in kiwi-lib but it gets the one from repart instead when both are installed.  It does seem to work as expected (other than the lsblk errors) despite this, however.

Marcus Schäfer

unread,
Aug 8, 2025, 7:01:59 AMAug 8
to kiwi-...@googlegroups.com
Hi Gavin,

thank you so much for debugging all that kiwi dracut issues.
I believe we have now a set of PRs that touches the remaining
issues. So there are the two open PRs

https://github.com/OSInside/kiwi/pull/2871
https://github.com/OSInside/kiwi/pull/2870

They fix the following defects:

1. in kiwi-repart make sure it does not operate on unexpected root= setting
This was your PR, thanks much

2. in kiwi builder/install make sure to omit kiwi-repart for building
an ISO. Including kiwi-repart there happened because a custom
dracut.conf.d/xxx.conf file existed. The specific omit makes sure
we never add it for install ISOs as it doesn't make any sense

> kiwi-dump-image calls the initialize function, which isn't defined in
> that module so it calls across to kiwi-repart-disk instead. This seems
> dubious, what if both weren't installed? What if another module had
> been sourced in between with its own initialize function? (Actually
> the conditional library includes at the top don't seem entirely safe in
> that regard either.)

3. Fix that name conflict which I introduced with the initialize
method implemented in different modules under the same name
but with different functionality. shame on me :/

That together I think would improve the situation

If you can have a short look on the open PRs that would
be great.

Thanks again for your time and effort
signature.asc

Gavin Lambert

unread,
Aug 10, 2025, 7:48:34 PMAug 10
to kiwi
On Friday, August 8, 2025 at 11:01:59 PM UTC+12 Marcus wrote:
thank you so much for debugging all that kiwi dracut issues.
I believe we have now a set of PRs that touches the remaining
issues. So there are the two open PRs

https://github.com/OSInside/kiwi/pull/2871
https://github.com/OSInside/kiwi/pull/2870

I haven't tested the new one, but they both look good to me.  Thanks for looking into this.
Reply all
Reply to author
Forward
0 new messages