Question on deterministic OS disk selection with multiple SAN multipath disks
Saish Patil
Hi Marcus / Neil,
We're using KIWI NG to build an Oracle Linux OEM image for automated bare-metal provisioning.
The installation target is a server booted from an ISO, where the storage is presented from a Pure Storage SAN through multipath. We're trying to understand how KIWI determines the installation disk when multiple multipath devices are present.
Our KIWI configurationOur image is built with the following configuration (simplified):
<type image="oem" filesystem="xfs" initrd_system="dracut" installboot="install"
firmware="uefi" installiso="true" bootpartition="true" bootpartsize="1024"
efipartsize="600"
kernelcmdline="console=tty1 biosdevname=0 ipv6.disable=1 net.ifnames=0 rd.kiwi.oem.installdevice=/dev/mapper/mpatha rd.auto=1 rd.lvm=1 root=/dev/mapper/vg.sys.1-LVRoot rd.debug rd.kiwi.debug">
<bootloader name="grub2" console="console" timeout="10" />
<oemconfig>
<oem-unattended>true</oem-unattended>
<oem-swap>false</oem-swap>
<oem-multipath-scan>true</oem-multipath-scan>
</oemconfig>
<systemdisk name="vg.sys.1" preferlvm="true">
<volume name="@root" size="8G" mountpoint="/" />
<volume name="lv.usr.0" size="12G" mountpoint="/usr" />
<volume name="lv.var.0" size="400G" mountpoint="/var" />
<volume name="lv.tmp.0" size="10G" mountpoint="/tmp" />
<volume name="lv.opt.0" size="32G" mountpoint="/opt" />
<volume name="lv.var.log.0" size="30G" mountpoint="/var/log" />
<volume name="lv.var.log.audit.0" size="4G" mountpoint="/var/log/audit" />
</systemdisk>
</type>
</preferences>
Our intended OS disk is always 512 GB.
Environment
The bare-metal servers receive SAN LUNs through multipath.
Typical setup:
Both disks are presented through multipath.
Example:
However, this ordering is not consistent across servers.
Sometimes we observe:
Likewise, the underlying sdX device names also change between boots.
Cases we observed Case 1
If
and on a particular server
KIWI attempts to install onto the 128 GB LUN, which eventually fails because our partition layout requires much more space.
This is expected because the kernel parameter explicitly points to mpatha.
Case 2
Using
is also not an option because those names are not stable across boots or servers.
Case 3 (the confusing one)
This is the behavior we're trying to understand.
On one server we had:
The runtime kernel command line confirms this:
However, after installation we observed that the installed system was actually using mpathb, which was the 128 GB disk.
The resulting layout looked like this (identifiers masked):
The multipath output showed:
So despite rd.kiwi.oem.installdevice being set to mpatha, the installed partitions ended up on mpathb.
This is the part we don't understand.
Our questions
We're trying to understand the internal selection logic.
- At what stage is rd.kiwi.oem.installdevice evaluated?
- Is there any scenario where KIWI ignores this parameter and chooses another disk?
- If multiple multipath devices exist, how does KIWI decide which disk becomes the system disk?
- Does dracut or udev rename/reorder multipath devices during installation?
- Is there a recommended method to specify the installation target using something more stable
- Is there an officially recommended approach for unattended OEM installations on SAN-attached systems with multiple LUNs?
Our goal
We need deterministic OS disk selection across hundreds of bare-metal servers.
The only thing that is guaranteed is:
- one SAN LUN is the OS disk (512 GB)
- one or more additional SAN LUNs may exist
- device names (sdX, mpatha, mpathb) are not guaranteed to remain consistent
We'd like to know what the recommended KIWI approach is to ensure the correct LUN is always selected for installation.
Relevant logs (identifiers masked)
KIWI debug log:
lookup_disk_device_from_root():
root:block=/dev/mapper/vg.sys.1-LVRoot
kiwi_oemmultipath_scan=true
root_device=/dev/dm-6
lsblk -p -n -r -s -o NAME,TYPE /dev/dm-6
...
/dev/sdf
get_mapped_multipath_disk /dev/sdf
multipath -l -v1 /dev/sdf
/dev/mapper/mpathb
disk_device=/dev/mapper/mpathb
From the KIWI debug logs, it appears that lookup_disk_device_from_root() starts from /dev/mapper/vg.sys.1-LVRoot, traces it back to /dev/sdf, and then resolves the corresponding multipath device as /dev/mapper/mpathb. This matches the final installation layout where all boot partitions and LVM volumes were created on mpathb. However, the kernel command line explicitly contained:
console=tty1 biosdevname=0 ipv6.disable=1 net.ifnames=0
rd.kiwi.oem.installdevice=/dev/mapper/mpatha
rd.auto=1
rd.lvm=1
root=/dev/mapper/vg.sys.1-LVRoot
rd.debug
rd.kiwi.debug
We're trying to understand why the installation ultimately resolved to mpathb instead of respecting the configured rd.kiwi.oem.installdevice.
lsblk output on this server:
mpathb 253:1 0 128G 0 mpath
├── mpathb1 253:2 0 2M 0 part
├── mpathb2 253:3 0 600M 0 part /boot/efi
├── mpathb3 253:4 0 1G 0 part /boot
└── mpathb4 253:5 0 126.4G 0 part
├── vg.sys.1-LVRoot 253:6 0 8G 0 lvm /
├── vg.sys.1-lv.opt.0 253:7 0 32G 0 lvm /opt
├── vg.sys.1-lv.tmp.0 253:8 0 10G 0 lvm /tmp
├── vg.sys.1-lv.usr.0 253:9 0 12G 0 lvm /usr
├── vg.sys.1-lv.var.0 253:10 0 576M 0 lvm /var
├── vg.sys.1-lv.var.log.0 253:11 0 30G 0 lvm /var/log
└── vg.sys.1-lv.var.log.audit.0
253:12 0 4G 0 lvm /var/log/audit
Thank you for any guidance on how KIWI determines the installation target in this scenario and what the recommended approach is for deterministic SAN LUN selection.
Marcus Schäfer
> We're using KIWI NG to build an Oracle Linux OEM image for automated
> bare-metal provisioning.
> The installation target is a server booted from an ISO, where the
> storage is presented from a Pure Storage SAN through multipath. We're
> trying to understand how KIWI determines the installation disk when
> multiple multipath devices are present.
> Our KIWI configuration
>
> <oem-unattended>true</oem-unattended>
> <oem-swap>false</oem-swap>
> <oem-multipath-scan>true</oem-multipath-scan>
> </oemconfig>
> mpatha -> 512 GB mpathb -> 128 GB
> However, this ordering is not consistent across servers.
>
> Sometimes we observe:
> mpatha -> 128 GB mpathb -> 512 GB
>
> Likewise, the underlying sdX device names also change between boots.
are mapped against the WWID.
I usually recommend to provide a storage specific etc/multipath.conf
file which maps the WWID to a custom alias. e.g.
multipaths {
multipath {
wwid "some"
alias "some_alias"
}
}
This file must also be made availabe in the initrd that dumps
the image. To achieve this also add e.g /etc/dracut.conf.d/media.conf
install_items+=" /etc/multipath.conf "
After that a "/dev/mapper/some_alias" always correctly bound
to the WWID of your storage cluster should appear
> We're trying to understand the internal selection logic.
This is done in get_disk_list and searches for the given device
name in the list of devices found by lsblk.
https://github.com/OSInside/kiwi/blob/main/dracut/modules.d/55kiwi-dump/kiwi-dump-image.sh#L202
> 2. Is there any scenario where KIWI ignores this parameter and chooses
> another disk?
only if you also specify rd.kiwi.oem.smallest_disk
> 3. If multiple multipath devices exist, how does KIWI decide which
into account and if it does not exist an error message appears.
If you don't specify any of the available rd.xxx selectors and
you have specified the unattended mode, kiwi takes the first
entry in the list (which is a dangerous mode). If you don't
specify <oem-unattended> kiwi offers a menu from which you
can manually select
> 4. Does dracut or udev rename/reorder multipath devices during
> installation?
not to my knowledge. For multipath devices the multipathd creates
the node in /dev/mapper. udev does not interfere with them
> 5. Is there a recommended method to specify the installation target
> using something more stable
see my recommendation above
> 6. Is there an officially recommended approach for unattended OEM
Hope this helps a bit
Best regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------