SLES15.4 iPXE boot fails: missing ram drive

[Jan Robinson]

ello All

Would you please assist?

We are booting a PXE build from the same config.xml file where we change repos from 15.3 to 15.4.

SLES15.3 boots fine (except for resolv.conf.)

SELS15.4 goes to the emergency shell.
In this shell there are no /dev/ram* devices.

From the rdsosreport ( please see attached log):

localhost systemd[150]: /usr/lib/systemd/system-generators/dracut-kiwi-ramdisk-generator failed with exit status 1.1"

From the emergency shell:
The IP is fully configured
the /etc/resolve.conf  is configured correctly to what is on the kernel-cmdline.


A note.
From that shell if you do:

modprobe brd then /dev/ram* are created.


Environment:
VMWARE VM
The PXE installed server a VMWARE VM SLES15.4
Linux ldocker03 5.14.21-150400.22-default
dracut-kiwi-oem-dump-9.24.36-1.1.x86_64
dracut-kiwi-lib-9.24.36-1.1.x86_64
dracut-kiwi-oem-repart-9.24.36-1.1.x86_64
dracut-kiwi-live-9.24.36-1.1.x86_64

Kiwi server:
kiwi101 5.3.18-59.27-default
kiwi-systemdeps-core-9.24.36-1.1.x86_64
kiwi-pxeboot-9.24.36-1.1.x86_64
kiwi-tools-9.24.36-1.1.x86_64
kiwi-systemdeps-9.24.36-1.1.x86_64
python3-kiwi-9.24.36-1.1.x86_64
kiwi-boot-descriptions-1.2.0-2.51.x86_64
kiwi-man-pages-9.24.36-1.1.x86_64

<type image="oem" filesystem="ext4" installpxe="true" initrd_system="dracut" installboot="install" kernelcmdline="splash rd.kiwi.ramdisk ramdisk_size=4096000">
<oemconfig>
    <oem-skip-verify>true</oem-skip-verify>
    <oem-unattended>true</oem-unattended>
    <oem-unattended-id>/dev/ram0</oem-unattended-id>
    <oem-device-filter>/dev/sd</oem-device-filter>
    <oem-swap>false</oem-swap>
    <oem-multipath-scan>false</oem-multipath-scan>
</oemconfig>

boot.ipxe
kernel http://kiwi101.abcgroup.net/ldocker03.abcgroup.net.kernel console=ttyS0,115200 console=tty1 debug initrd=vmware initrd=boot/initrd rd.neednet=1 ip=${ip}::${gateway}:${netmask}:${hostname}:eth0:none:${dns}:15.30.252.2 rd.debug rd.kiwi.debug rd.kiwi.install.pxe rd.kiwi.ramdisk ramdisk_size=4096000 rd.kiwi.install.image=http://kiwi101.bmwgroup.net/pxe-result/SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.3.xz module_blacklist=vmwgfx root="/dev/mapper/ram0p1"


Thank you,
Jan

[Marcus Schäfer]

Hi,

> SLES15.3 boots fine (except for resolv.conf.)
> SELS15.4 goes to the emergency shell.

I guess no autoloading of the brd driver anymore

> From that shell if you do:
> modprobe brd then /dev/ram* are created.

Did you boot with rd.kiwi.ramdisk ?

If this option is present on the cmdline there is code in the
kiwi dracut module that dumps your image on target to actively load
the brd driver beforehand. Also see:

https://github.com/OSInside/kiwi/blob/master/dracut/modules.d/90kiwi-dump/kiwi-dump-image.sh#L44

Hope this helps ?

Regards,
Marcus
--
Public Key available via: https://keybase.io/marcus_schaefer/key.asc
keybase search marcus_schaefer
-------------------------------------------------------
Marcus Schäfer Brunnenweg 18
Tel: +49 7562 905437 D-88260 Argenbühl
Germany
-------------------------------------------------------

[Jan Robinson]

Hello Marcus.

Thank you for the suggestion.

My reply does not read easily so it is numbered.

1.
rd.kiwi.ramdisk are being used on the cmdline
console=ttyS0,115200 console=tty1 debug initrd=vmware initrd=boot/initrd rd.neednet=1 ip=10.145.102.88::10.145.102.1:255.255.255.0:ltitavmwdocker03-restore:eth0:none:160.50.250.2:160.50.252.2 nameserver=160.50.250.2 rd.debug rd.kiwi.debug rd.kiwi.install.pxe rd.kiwi.ramdisk ramdisk_size=4096000 rd.kiwi.install.image=http://itadell101.bmwgroup.net/pxe-result/SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.3.xz module_blacklist=vmwgfx root="/dev/mapper/ram0p1"

2.
At the emergency shell
if doing - modprobe brd
Then yes /dev/ram* are there.

3.
After this thread was logged I added 'brd' here to have it loaded.
It was loaded by the time the emergency shell was created but no ramdisk.

root/etc # cat dracut.conf.d/01-add-drivers.conf
add_drivers+=" xfs "
add_drivers+=" ext4 "
add_drivers+=" xen-blkfront "
add_drivers+=" vmw_pvscsi "
add_drivers+=" sd_mod "
add_drivers+=" libata "
add_drivers+=" brd "

and
root/etc # cat dracut.conf.d/01-add-fs.conf
install_items+=" /etc/modules-load.d/01-fs.conf "

root/etc # cat modules-load.d/01-fs.conf
ext4
sd_mod
libata
brd


4.
Your suggestion implemented and is part of the attached log.

root/etc/modprobe.d/:
-rw-r--r-- 1 root root  22 Jun  7 10:52 99-brd.conf

cat 99-brd.conf
install brd


5.
Unless it is not logged(possible), the dev-mapper-ram0p1 failed before 'brd' are loaded.


[    1.278134] localhost systemd[1]: dev-mapper.mount: Failed to load configuration: No such file or directory
[    1.278251] localhost systemd[1]: dev-mapper-ram0p1.mount: Failed to load configuration: No such file or directory
[    1.278491] localhost systemd[1]: bloc...@dev-mapper-ram0p1.target: Failed to load configuration: No such file or directory
[    1.282402] localhost systemd[1]: var.mount: Failed to load configuration: No such file or directory
[    1.282517] localhost systemd[1]: var-tmp.mount: Failed to load configuration: No such file or directory
[    1.282633] localhost systemd[1]: tmp.mount: Failed to load configuration: No such file or directory
[    1.441028] localhost kernel: SCSI subsystem initialized
[    1.477103] localhost kernel: libata version 3.00 loaded.
[    1.480405] localhost kernel: brd: module loaded
[    1.474629] localhost systemd-journald[204]: Journal started

6.
/usr/lib/dracut/modules.d/90kiwi-dump/kiwi-dump-image.sh is not called
and is not present in the emergency shell.

This is present but cannot see in the logs it being called.

/lib/dracut/hooks/pre-mount/30-kiwi-dump-image.sh

/lib/dracut/hooks/mount/30-kiwi-mount-ramdisk.sh

7.
if /lib/dracut/hooks/pre-mount/30-kiwi-dump-image.sh is executed from the emergency shell
a prompt comes up warning that there is not enough space for this image.

Probably because no parameters for a size was provided.

Good news  - if it is called with values, this could work.

log file attached: rdsosreport-15.4-7jun.txt.gz

Thank you so much.
Jan

[Marcus Schäfer]

Hi,

sorry I'm a little confused so bear with me if my questions
are stupid:

> rd.kiwi.ramdisk are being used on the cmdline
> console=ttyS0,115200 console=tty1 debug initrd=vmware
> initrd=boot/initrd rd.neednet=1
> ip=10.145.102.88::10.145.102.1:255.255.255.0:ltitavmwdocker03-restore:e
> th0:none:160.50.250.2:160.50.252.2 nameserver=160.50.250.2 rd.debug
> rd.kiwi.debug rd.kiwi.install.pxe rd.kiwi.ramdisk ramdisk_size=4096000
> rd.kiwi.install.image=http://itadell101.bmwgroup.net/pxe-result/SLES15-
> MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.3.xz module_blacklist=vmwgfx
> root="/dev/mapper/ram0p1"

Why do you pass a root= parameter at the kernel cmdline for an
install image ? The PXE based install loads the kernel and the
initrd and runs the "dump image to target" process as part of
the initrd. Once the initrd has completed this task it activates
the system dumped on target. In case of a ramdisk dump we directly
boot into the system from the loaded initrd because you can't
kexec from a non persistent ramdisk device. For all this you
don't need to specify root= in the PXE configuration because
the dump target is configured as part of your image description
and provided as xxx.config.bootoptions when kiwi built the
install image. I think the information we provide here:

https://osinside.github.io/kiwi/building_images/build_expandable_disk.html#network-deployment

could be helpful to clarify on the process

So from my perspective the first change should be:

AI: delete root="/dev/mapper/ram0p1" from your PXE kernel config
This causes the initrd to lookup a device /dev/mapper/ram0p1 which
will never appear of course and you end up in the rescue shell


The rest of the parameters looks good to me.

Now let's check on your image description. The target for the
deployment is configured inside of the image description. The
data there leads among others to this "xxx.config.bootoptions"
file which you put on your PXE server. The kernel option
"rd.kiwi.install.pxe" as you pass it tells the kiwi dump dracut
module to load "xxx.config.bootoptions" from the PXE server.
Now it knows where to put the image. According to that you
should have something like this in your kiwi description:

<oemconfig>
<oem-unattended-id>/dev/ram1</oem-unattended-id>
</oemconfig>

This tells the dump code to dump the image to "/dev/ram1"
If you have passed "rd.kiwi.ramdisk" the dump code will load
the brd driver before looking up /dev/ram1

I have checked the device names and loading brd still leads
to devices created in /dev/ramX. There never is a /dev/mapper/ramX
device on the system I use.

So next step imho is

AI: double check on your image description to use the correct device
AI: double check on your PXE server if PXE_SERVER_IP:/srv/tftpboot/image/*.config.bootoptions has the correct values according to your image build

Thanks

[Jan Robinson]

Hello Marcus

Thank you for the time with this.

This is now removed and the log file seems much better.
root="/dev/mapper/ram0p1"

The install now stops at:
[   20.947171][  T762] dracut: FATAL: Not enough space available for this image


It installed on the 15.3 image with wrongly root= , the server has 128GB.
Memory space should not be the real issue.

Added this option to the kernel cmdline to get a shell and not just the halt:
rd.shell=1


Update:

<preferences profiles="RESQ">

        <type image="oem" filesystem="ext4" installpxe="true" initrd_system="dracut" installboot="install" kernelcmdline="splash rd.kiwi.ramdisk ramdisk_size=4096000">
        <oemconfig>
            <oem-skip-verify>true</oem-skip-verify>
            <oem-unattended>true</oem-unattended>

            <oem-unattended-id>/dev/ram1</oem-unattended-id>

            <oem-device-filter>/dev/sd</oem-device-filter>
            <oem-swap>false</oem-swap>
            <oem-multipath-scan>false</oem-multipath-scan>
        </oemconfig>

        </type>
</preferences>

config.bootoptions
splash rd.kiwi.ramdisk ramdisk_size=4096000 root=UUID=4fec6ee1-bc4d-45b8-82a6-853f0a29a188


iPXE:
kernel http://kiwi101.abcgroup.net/docker03.abcgroup.net.kernel console=ttyS0,115200 console=tty1 debug initrd=vmware ip=${ip}::${gateway}:${netmask}:${hostname}:eth0:none:8.4.250.2:8.4.252.2 rd.debug rd.kiwi.debug rd.kiwi.install.pxe initrd=boot/initrd splash rd.kiwi.ramdisk ramdisk_size=4096000 rd.kiwi.install.image=http://kiwi101.abcgroup.net/pxe-result/SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.4.xz rd.shell=1


Please see attached log file rdsosreport-noroot-8may.txt

Thank you once more
Jan

[Marcus Schäfer]

Hi Jan,

> The install now stops at:
> [ 20.947171][ T762] dracut: FATAL: Not enough space available for
> this image

ok, ramdisk_size=4096000 == 3G, your uncompressed image inside of
http://kiwi101.abcgroup.net/pxe-result/SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.4.xz is smaller than 3G right ?

Other than that I saw you install with rd.kiwi.debug, this means
there is a log file written by the kiwi dracut code in:

/run/initramfs/log/boot.kiwi

Could you fetch that file from the debug shell ? It's a bit easier
to analyze

[Jan Robinson]

Hello Marcus

rd.kiwi.debug and debug was on the kernel command line.

These two together caused boot.kiwi not to be created.

Had  play a bit, removing debug and the file is created.

The image is smaller than that yes.

-rw-r--r-- 1 root root 1.4G Jun  8 13:15 SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.4.raw
-rw-r--r-- 1 root root 242M Jun  8 13:15 SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.4.xz

SLES15-MiniOS-VMW-OEM-PXE.DEFAULT.x86_64-15.4.config.bootoptions
splash rd.kiwi.ramdisk ramdisk_size=4096000 root=UUID=4fec6ee1-bc4d-45b8-82a6-853f0a29a188

The boot.kiwi is attached and reveals a lot.

Size of /dev/ram1 is 128 MB

The VM allocated memory is 4GB.

Thank you,

Jan

[Jan Robinson]

Hi Marcus

Just an update:

Adding this to the kernel cmdline made it boot to /dev/ram0 as rd_nr=1.

brd.rd_size=3072000 brd.max_part=2 brd.rd_nr=1

This is probably not a final solution but we can continue.

# blockdev --getsize64 /dev/ram0
3145728000

# df -hP
Filesystem          Size  Used Avail Use% Mounted on
devtmpfs            4.0M     0  4.0M   0% /dev
tmpfs               2.0G     0  2.0G   0% /dev/shm
tmpfs               786M   64M  723M   9% /run
tmpfs               4.0M     0  4.0M   0% /sys/fs/cgroup
/dev/mapper/ram0p1  1.3G 1011M  212M  83% /
tmpfs               393M     0  393M   0% /run/user/0

It is great that 15.4 is now booting but this all started by the empty /etc/resolv.conf.

We need DNS to work.

From all the emergency shells I have checked /tmp contained the following files.

This made me hope for the /etc/resolf.conf to be populated, in stead it is cleared now with a good boot.

Attached are the two logfiles.

rdsosreport.ramcmdline.txt

boot.kiwi.ramcmdline.txt

---------

15.4 emergency shell
/tmp
-rw-r--r-- 1 root root      55 Jun  7 11:19 net.eth0.gw
-rw-r--r-- 1 root root      60 Jun  7 11:19 net.eth0.hostname
-rw-r--r-- 1 root root     122 Jun  7 11:19 net.eth0.override
-rw-r--r-- 1 root root      72 Jun  7 11:19 net.eth0.resolv.conf
-rw-r--r-- 1 root root       0 Jun  7 11:19 net.eth0.up
-rw-r--r-- 1 root root       6 Jun  7 11:18 net.ifaces

sh-4.4# cat net.eth0.gw
ip route replace default via '10.145.102.1' dev 'eth0'

sh-4.4# cat net.eth0.hostname
echo 'docker03' > /proc/sys/kernel/hostname

sh-4.4# cat net.eth0.override
ip=10.145.102.88
gw=10.145.102.1
mask=255.255.255.0

hostname=docker03
dns1=8.4.250.2
dns2=8.4.252.2

sh-4.4# cat net.eth0.resolv.conf
nameserver 8.4.250.2
nameserver 8.4.252.2

sh-4.4# cat net.ifaces
 eth0

-----------------

Thanks a mil so far.

Jan

[Jan Robinson]

Hello Marcus.

The new kiwi 9.24.41 is creating the ram drive all fine and the images boots.

Thanks so much for the help.

However the server boot up with no DNS.

/etc/resolv.conf does not exit.

IP configuration is obtained from the Dracut IP= option.

ip=${ip}::${gateway}:${netmask}:${hostname}:eth0:none:${dns}:${dns2}

This all works fine save the DNS.

Could you please assist?

Thanks, Jan.