Using logical volumes with CoreOS

[abdulazee...@gmail.com]

Hi,

Are logical volumes supported on CoreOS?

So here's my setup:

I have 4 baremetal servers with one being the manager (iPXE, DNS, DHCP, TFTP and HTTP services) and the other 3 grabbing CoreOS from the manager via iPXE. Then iIplan to have the 3 severs on a 3-node docker swarm  and each server has six 2TB HDD drives on them. I have configured RAID 5 on 3 of the HDDs and I want to have the remaining 3 HDDs (non-RAID) as one big volume group then create logical volumes as needed (probably 4 LVMs for a start in the ignition config).

Is there a way to specify this lvm config using the container linux config and then converting it to an ignition file?

Any input or recommendation is welcomed.

Azeem

[Andrew Jeddeloh]

Unfortunately Ignition (and thus CLCs) don't support LVM directly
right now. Ideally Ignition would be able to set up LVM then write out
units to mount LVs. However, it sounds like in your case you might be
able to use systemd units to create the volume if it doesn't exist and
then use it. Container Linux in general doesn't support root on LVM
though, so if you were hoping to do that you're out of luck.

Are you looking to create the LVM setup or just mount an existing one?

- Andrew

> --
> You received this message because you are subscribed to the Google Groups "CoreOS User" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[abdulazee...@gmail.com]

Hi Andrew,

I'm looking to use the ignition config to create the volume group and the logical volumes, then mount it on specified directories. which would in turn be used as a bind mount for my database containers.

I noticed that the "lvm" commands work on CoreOS. Can I pass in a script that runs after the servers are up that does the lvm configs? If yes, how do I go about that.

Also I have CoreOS installed on memory and I have no plan of installing to disk. I'm working on a very closed system so if I need to update CoreOS, I'll just download the latest kernel and intdr from the repo and host on my local webserver. So the second question is "Once I reboot my servers (for update purposes only), does it just grabs the new kernel and initrd files via iPXE and skips the ignition part? I don't want it to touch my configs".

Azeem

[Andrew Jeddeloh]

Ignition is designed to only run at first boot, but running PXE like
you described there is no "first boot" since there is no boot
partition and the rootfs isn't persisted. Every boot looks like a
"first boot", so Ignition runs every time. For those reasons we added
support for reusing existing filesystems[1] and partitions[2] to
Ignition. Unfortunately we haven't implemented LVM support, so you'll
have to roll your own. Luckily, you don't need root on LVM, so that's
not an issue.

Since you're not installing to disk, having Ignition run every boot is
actually what you want. Every time you boot you'll get a blank system,
aside from whatever is not on the rootfs. You don't need to worry
about configuration the previous boot wrote since you'll be starting
fresh every time. This only applies to things on the root filesystem,
data stored on a different filesystem will persist and you'll need to
be able to handle that.

This is how I'd generally go about it:
1) Write a script that sets up the LVM but check if it's already set
up and not perform any actions. Use a CLC/Ignition to install that
script.
2) Write a systemd service unit that calls that script. Use your CLC
to add the unit.
3) Write a systemd mount unit that depends on that script (you'll want
both Requires= and After=) and anything else you need to expose or
start the LVM LV.

I can't go into too much more detail about the LVM bits since I
haven't dug into LVM in detail yet.

- Andrew

[1] https://coreos.com/ignition/docs/latest/operator-notes.html#filesystem-reuse-semantics
[2] https://coreos.com/ignition/docs/latest/operator-notes.html#partition-reuse-semantics

[abdulazee...@gmail.com]

Thanks for the info Andrew. I'll follow your recommendation and revert to you if any issue pops up.

Azeem

[abdulazee...@gmail.com]

Hi Andrew,

I thought to give you an update based on your recommendation.

I followed step 1 and 2 and everything works perfectly. Instead of writing a systemd mount unit as you recommended in Step 3, I added the mount bit into my script and had the systemd service call the script.

Is there any advantage to using the systemd mount unit as opposed to adding the mount command in the script?

Azeem

[Andrew Jeddeloh]

Glad to hear it's working!

Breaking things down into more units might make it clearer what
happened in the case of failure and would allow you to depend on each
unit individually or have each unit depend on separate things. It's
not a huge difference tough. In fact systemd actually will generate a
mount unit when you mount something, so you don't even need to write
one to depend on one.
- Andrew

[abdulazee...@gmail.com]

Hey Andrew,

I decided to break things down into more units to make them cleaner. Thanks for the help!

Azeem