diskless system with KIWI and a RW filesystem on the client

[Nathan Burnett]

Hello,
I'm not sure if this is the correct place to post this, but I am seeking help with booting a diskless system with KIWI and having a read/writeable filesystem on the client.

I referenced this guide:
  http://osinside.github.io/kiwi/working_with_images/legacy_netboot_root_filesystem.html

I built the image with:
  git clone https://github.com/OSInside/kiwi-descriptions
  kiwi-ng --profile PXE --type pxe system build --description kiwi-descriptions/suse/x86_64/suse-leap-15.1 --target-dir /tmp/mypxe-result2


I used these configurations:
cat /srv/tftpboot/KIWI/config.default
  NFSROOT="192.168.128.54;/storage/pxeboot/"
  UNIONFS_CONFIG=tmpfs,nfs,overlay

cat /srv/tftpboot/pxelinux.cfg/default
  default menu.c32
  DEFAULT KIWI-Boot
  prompt   1
  timeout  30
  display pxelinux.msg/default

  LABEL KIWI-Boot
      kernel boot/linux
      append rd.shell rd.debug rd.kiwi.debug initrd=boot/initrd
      IPAPPEND 2

      
On the client the filesystem is read-only. Checking /proc/mounts shows:
  ...
  overlay / overlay ro,realtime,lowerdir=/read-only,uperdir=/read-write/rw,workdir=/read-write/work 0 0
  ...
 
I think this should be rw.  What do I need to change to get the FS on the diskless client to be RW using memory?  Am I missing a resource that documents these steps?

[David Cassany]

Hi,

Sorry for the late reply. Yes this is the right place to ask for support
regarding image building using KIWI. Note you are using legacy features
that are on maintenance mode only and that they are targeted to
disappear at some point in time.

We encourage you to use one of the methods exposed in `Booting a Root
Filesystem from Network` section in docs [1].

[1]
http://osinside.github.io/kiwi/working_with_images/network_overlay_boot.html

Anyway from what you have share I have few little comments:

> I used these configurations:
> cat /srv/tftpboot/KIWI/config.default
>   NFSROOT="192.168.128.54;/storage/pxeboot/"
>   UNIONFS_CONFIG=tmpfs,nfs,overlay

This part looks good to me.

> cat /srv/tftpboot/pxelinux.cfg/default
>   default menu.c32
>   DEFAULT KIWI-Boot
>   prompt   1
>   timeout  30
>   display pxelinux.msg/default
>
>   LABEL KIWI-Boot
>       kernel boot/linux
>       append rd.shell rd.debug rd.kiwi.debug initrd=boot/initrd
>       IPAPPEND 2

Note that this is a legacy build that does not make use of a dracut
based initrd. Meaning that all `rd.*` parameters are merely ignored
here. In order to get debug traces you should use the `kiwidebug=1` and
then logs will be at `/var/log/boot.kiwi`. There we should be capable to
see the mount commands used for the overlayfs.

Could you share the `/var/log/boot.kiwi` file? Also which kiwi version
are you using? and which version of the kiwi-boot-descriptions package.

This read only mount is rare, as in [2] you can see how we explicitly
set it to `rw`.

[2]
https://github.com/OSInside/kiwi-descriptions/blob/40575ac6787d15f6d5a4d28f1cc4b8b2c8198ea4/custom_boot/functions.sh#L5101

Regards,
David

[Nathan Burnett]

Hello David,

Thank you for the response! I really appreciate the help.

We encourage you to use one of the methods exposed in `Booting a Root
Filesystem from Network` section in docs [1].

[1]
http://osinside.github.io/kiwi/working_with_images/network_overlay_boot.html

Does 'Booting a Root Filesystem from Network' support NFS?  From the documentation it looks to only support NBD and AoE.  We chose legacy netboot due to NFS support. There is a strong inclination to only use NFS here due to other factors.

 

Could you share the `/var/log/boot.kiwi` file? Also which kiwi version
are you using? and which version of the kiwi-boot-descriptions package.

[root@server ~]# kiwi-ng -version
KIWI (next generation) version 9.23.22

Zypper reports:

  kiwi-boot-descriptions  is 1.2.0-2.51

  python3-kiwi is 9.23.22-1.1

I have attached my /var/log/boot.kiwi file.  Please note, I changed /srv/tftpboot/pxelinux.cfg/default to /srv/tftpboot/pxelinux.cfg/01-98-03-9b-54-b0-52 but the contents of the file are the same.

 

This read only mount is rare, as in [2] you can see how we explicitly
set it to `rw`.

[2]
https://github.com/OSInside/kiwi-descriptions/blob/40575ac6787d15f6d5a4d28f1cc4b8b2c8198ea4/custom_boot/functions.sh#L5101

Previously, in my debugging efforts, after I generated the Leap-15.1_appliance.x86_64-1.15.1.initrd.xz and unpacked it, I searched for "mount -t overlay".  I had found a file ./include that contains similar code to this link. Maybe it is a previous version?  Include appears to be used by ./init, ./preinit, and ./linuxrc. It seems from this include script the mount should be RW, but it still is RO in the final system.  I could not find other references to "mount -t overlay".

 

Thanks again for your help.

--

Nathan Burnett

[David Cassany]

Hi Nathan,

> We encourage you to use one of the methods exposed in `Booting a Root
> Filesystem from Network` section in docs [1].
>
> [1]
> http://osinside.github.io/kiwi/working_with_images/network_overlay_boot.html
> <http://osinside.github.io/kiwi/working_with_images/network_overlay_boot.html>
>
>
> Does 'Booting a Root Filesystem from Network' support NFS?  From the
> documentation it looks to only support NBD and AoE.  We chose legacy
> netboot due to NFS support. There is a strong inclination to only use
> NFS here due to other factors.

I believe it should be doable to run a diskless setup using nfs and a
kis image type. However I have never tried this setup, so I am not fully
aware of the difficulties you could find in the way.

Certainly not out of the box, since the kiwi-overlay module does not
support it. However I believe it should not be too difficult to set a
custom module to do what you need.

KIWI kis image type produces a kernel, a dracut based initrd and a
rootfs in a filesystem. Booting remote nfs root is seams to be supported
by dracut, see dracut docs here:

https://man7.org/linux/man-pages/man7/dracut.cmdline.7.html

To my understanding something like root=nfs:<server-ip>:<root-dir>
should boot the nfs root. Then to overlay on RAM on top of it I guess
you need a pre-mount hook on dracut to create the tmpfs mountpoint and
mount an overlayfs on top of it. This how I would start digging this option.

https://man7.org/linux/man-pages/man7/dracut.modules.7.html

> Could you share the `/var/log/boot.kiwi` file? Also which kiwi version
> are you using? and which version of the kiwi-boot-descriptions package.

Thanks for sharing I had a look at it and I could not find anything
wrong with it. In fact, there is the mount I expected to see, with the
rw operation. Could it be that your systemd is remounting it after the
switch root? Do you have some fstab in your root /etc?

See line 3749 of your boot.kiwi log:

mount -t overlay -o
rw,lowerdir=/read-only,upperdir=/read-write/rw,workdir=/read-write/work
overlay /mnt

This is mounted as readwrite and I'd say the /var/log/boot.kiwi is
actually stored in the overlay rw area. So it looks like something after
the init script that KIWI provides is remounting the overlayfs. I'd try
to check if journactl provides some further information regarding the
mounts of the filesystem. Systemd afters switching root form initrd to
real root might remount the filesystems according to the `root=`
paramter in the kernel commandline and according to whatever is in
/etc/fstab.

Hopefully this helps to dig it a little further.

Regards,
David