Some questions related to installation en ZFS storage pools.

[Jan Hugo Prins]

Hello everyone,

I'm currently investigating Flatcar Container Linux to see if we can use it as one of the main components of a new to build in-house Kubernetes cluster. I'm in the very early stages of everything but I run into a few issues:

I have found some documentation on how to install a node on a local disk using a systemd service that does the installation after first boot (https://thenewstack.io/tutorial-install-flatcar-container-linux-on-remote-bare-metal-servers/) and after some modifications to the ignition files I was able to get this working. I now have 2 ignition files, one that is used immediately after PXE boot and starts the installation process on the local disk, and a second one that is used during every subsequent boot to do the complete configuration of the node during normal operation. 

I have some questions though. I have tried adding the section in my ignition file to set a static IP address on a node (https://www.flatcar.org/docs/latest/setup/customization/network-config-with-networkd/), and the file is being created in /etc/systemd/network/ but the static IP is never set. Is there something I'm missing here? The logs tell me the following, which suggests to me that this documentation might be outdated, or I have misunderstood something:

Jul 06 16:40:08 node-1 systemd-networkd[1268]: /etc/systemd/network/10-static.network:1: Unknown section 'match'. Ignoring.
Jul 06 16:40:08 node-1 systemd-networkd[1268]: /etc/systemd/network/10-static.network: No valid settings found in the [Match] section, ignoring file. To match all interfaces, add Name=* in the [Match] section.

[match]
Name=enp4s0f0

[Network]
Address=192.168.8.31/24
Gateway=192.168.8.1

DNS=192.168.8.1

I could of course also use a static DHCP binding in my DHCP server, but for now I want to see if this would work because it makes the normal day to day operations independent of the DHCP server.

A second question I have is related to storage. One of the requirements I  got from the engineers that are going to use this cluster in the future is that we create nodes with ZFS storage pools on them. I have not found any documentation on how to do this, of if it is even possible with Flatcar Linux. Could someone give me some pointers?

Regards,

Jan Hugo Prins

[Jan Hugo Prins]

After posting this, I noticed the difference between [match] and [Match]. Did not know this was case sensitive.

So at the moment only the ZFS storage pool is an open question.

Jan Hugo Prins

[k...@kinvolk.io]

Hello,

for ZFS there is no official docs section but two approaches exist, and I think both will have problems to solve first:
https://github.com/jepio/flatcar-zfs-overlay
https://github.com/varasys/corezfs

In general, the first approach to package it in a systemd-sysext image makes sense nowadays but I guess it will require some workaround at the moment to force the module loading and create /etc/zfs through a systemd oneshot service instead of modules-load.d and tmpfiles.d entries. Jeremi can maybe add some more details on the last state of this attempt.

Kai

[Jeremi Piotrowski]

Hi Jan,

Like Kai says: nowadays we recommend the sysext approach. You will need to dive fairly deep into Flatcar to build a zfs one and will need to setup your own pipeline to keep it up to date (and to fetch a new version after an OS update). There are also some OS integrations that currently don't work so well with sysexts, most of which a zfs sysext appears to hit ( :) ):

- systemd presets don't work so you'll need to enable zfs related services/targets yourself

- udev rules don't work so the default approach of loading zfs mods when a zfs formatted volume is found doesn't work

- kernel module loading through /etc/modules-load.d happens before sysext activation so you'll need to mess with systemd service dependencies through drop-ins until you succeed

- kernel modules dependencies may be overwritten, which can break modprobe (though for a single sysext things appear to work)

Now that you've been warned about the dragons, this is more or less the incantation that you need to build one:

git clone https://github.com/flatcar/scripts -b <release>

./run_sdk_container -t -a amd64

./build_packages --usepkgonly

./build_image

./update_ebuilds sys-fs/zfs

./update_ebuilds sys-fs/zfs-kmod

# repeat update_ebuilds for any additional dependency or required eclasses

sudo USE="-rootfs minimal" KBUILD_OUTPUT=$(echo /build/amd64-usr/lib/modules/*/build) KERNEL_DIR=$(echo /build/amd64-usr/lib/modules/*/source) ./build_sysext --metapkgs=sys-fs/zfs,sys-fs/zfs-kmod --build_dir=../build/images/amd64-usr/latest/ zfs

Hope this helps,

Jeremi