Filesystem provisioning in warewolf 3.8.1

[Darkned]

Hello everyone,

I am currently trying to create a new model (virtualized at first) of my HPC clusters based on CentOS-7 with Warewulf 3.8.1 (from Github).

Previously, I used Warewulf-3.3.1 without any problems but, with this latest version, I am facing a disk partitioning one.

If in a first time I tried my usual command line:
[root@warewulf ~]# wwsh object modify -s FILESYSTEMS="dev=sda1:type=swap:size=1024,mountpoint=/:dev=sda2:type=ext3:size=fill" node003 —yes

I found in a discussion from a few months ago that in a 3.8.0+ version we should define partitioning in a dedicated file. So I also tried:
[root@warewulf ~]# cat /etc/warewulf/filesystem/my_partitioning.cmds
# BIOS / MBR Example
select /dev/sda
mklabel msdos
mkpart primary linux-swap 1MiB 1025MiB
mkpart primary ext3 1025MiB 100%
set 1 boot on
mkfs 1 swap
mkfs 2 ext4 -L root
fstab 2 / ext4 defaults 0 0
fstab 1 swap swap defaults 0 0
[root@warewulf ~]# wwsh object modify -s FILESYSTEMS=/etc/warewulf/filesystem/my_partitioning.cmds node003 —yes


But the result is the same for me: no partition created and no specific error in log files.

Below my node and provisioning definitions and, in attachments, the node's filesystem log file where I can find no explanation for the problem.

[root@warewulf ~]# wwsh bootstrap print
BOOTSTRAP NAME SIZE (M) ARCH
3.10.0-1062.12.1.el7.x86_64 30.1 x86_64
3.10.0-1062.9.1.el7.x86_64 30.1 x86_64
[root@warewulf ~]# wwsh node print node003
#### node003.cluster ##########################################################
node003.cluster: ID = 31
node003.cluster: NAME = node003,node003.cluster
node003.cluster: NODENAME = node003
node003.cluster: ARCH = x86_64
node003.cluster: CLUSTER = UNDEF
node003.cluster: DOMAIN = cluster
node003.cluster: GROUPS = cluster-nodes
node003.cluster: ENABLED = TRUE
node003.cluster: eth0.HWADDR = 08:00:27:25:57:46
node003.cluster: eth0.HWPREFIX = UNDEF
node003.cluster: eth0.IPADDR = 10.0.1.3
node003.cluster: eth0.NETMASK = 255.255.254.0
node003.cluster: eth0.NETWORK = 10.0.0.0
node003.cluster: eth0.GATEWAY = 10.0.0.100
node003.cluster: eth0.MTU = UNDEF
node003.cluster: eth0.FQDN = node003.cluster
[root@warewulf ~]# wwsh provision print node003
#### node003.cluster ##########################################################
node003.cluster: BOOTSTRAP = 3.10.0-1062.12.1.el7.x86_64
node003.cluster: VNFS = centos7-base-v2
node003.cluster: FILES = authorized_keys,dynamic_hosts
node003.cluster: PRESHELL = FALSE
node003.cluster: POSTSHELL = FALSE
node003.cluster: CONSOLE = UNDEF
node003.cluster: PXELINUX = UNDEF
node003.cluster: SELINUX = DISABLED
node003.cluster: KARGS = "net.ifnames=0 biosdevname=0 quiet"
node003.cluster: BOOTLOCAL = FALSE
[root@warewulf ~]# wwsh object -p:all print node003
#### node node003.cluster #####################################################
31: NAME = node003,node003.cluster
31: ARCH = x86_64
31: BOOTSTRAPID = 32
31: DISKFORMAT = sda1
31: DISKPARTITION = sda
31: DOMAIN = cluster
31: FILEIDS = 6,7
31: FILESYSTEMS = /etc/warewulf/filesystem/my_partitioning.cmds
31: GROUPS = cluster-nodes
31: KARGS = biosdevname=0,net.ifnames=0,quiet
31: NETDEVS = ObjectSet
NETDEVS.eth0.NAME = eth0
NETDEVS.eth0.FQDN = node003.cluster
NETDEVS.eth0.GATEWAY = 10.0.0.100
NETDEVS.eth0.HWADDR = 08:00:27:25:57:46
NETDEVS.eth0.IPADDR = 10.0.1.3
NETDEVS.eth0.NETMASK = 255.255.254.0
NETDEVS.eth0.NETWORK = 10.0.0.0
31: NODENAME = node003
31: VNFSID = 34

Maybe I missed something?

Any idea? Thanks in advance.

Ned

[Jason Stover]

Run:

wwsh provision set node003 -f /etc/warewulf/filesystem/my_partitioning.cmds

-J

> --
> You received this message because you are subscribed to the Google Groups "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/D76D2B31-F03B-4203-9CE4-2E80D92FD2C8%40gmail.com.
>
>
> --
> You received this message because you are subscribed to the Google Groups "Warewulf" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to warewulf+u...@lbl.gov.
> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/D76D2B31-F03B-4203-9CE4-2E80D92FD2C8%40gmail.com.

[Darkned]

Hi Jason,

Thanks for having reply so quickly.

I tried your propose but, if it does manifestly something, everything falls in fail from now : "running parted" is in error, the same for "extracting" VNFS, etc.

Here my node's updated provision definition :

[root@warewulf ~]# wwsh provision print node003

#### node003.cluster ##########################################################

node003.cluster: BOOTSTRAP        = 3.10.0-1062.12.1.el7.x86_64

node003.cluster: VNFS             = centos7-base-v2

node003.cluster: FILES            = authorized_keys,dynamic_hosts

node003.cluster: PRESHELL         = FALSE

node003.cluster: POSTSHELL        = FALSE

node003.cluster: CONSOLE          = UNDEF

node003.cluster: PXELINUX         = UNDEF

node003.cluster: SELINUX          = DISABLED

node003.cluster: KARGS            = "net.ifnames=0 biosdevname=0 quiet"

node003.cluster: FS               = "select /dev/sda,mklabel msdos,mkpart primary linux-swap 1MiB 1025MiB,mkpart primary ext4 1025MiB 100%,set 1 boot on,mkfs 1 swap,mkfs 2 ext4 -L root,fstab 1 swap swap defaults 0 0"

node003.cluster: BOOTLOCAL        = FALSE

[root@warewulf ~]# cat /etc/warewulf/filesystem/my_partitioning.cmds 

# BIOS / MBR Example

# Parted specific commands

select /dev/sda

mklabel msdos

mkpart primary linux-swap 1MiB 1025MiB

mkpart primary ext4 1025MiB 100%

set 1 boot on

# mkfs NUMBER FS-TYPE [ARGS...]

mkfs 1 swap

mkfs 2 ext4 -L root

# fstab NUMBER fs_file fs_vfstype fs_mntops fs_freq fs_passno

fstab 2 / ext4 defaults 0 0

fstab 1 swap swap defaults 0 0

I didn’t do anything else since my last demand.

Thanks in advance for your help,

Ned.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt8Lo7VHdmUwkA0RyEB3cveJEuzBxrbLgL%3Dvy7GtGShW3g%40mail.gmail.com.

[Jason Stover]

Okay... fun debug time....

Set a preshell on the node:
wwsh provision set node003 --preshell=true

This will start a shell in the bootstrap just prior to the
provisioning steps. Next manually run the script:

sh -x /warewulf/provision/20-filesystems

Hopefully that will give you a bit of output. Also see the file:
/var/log/warewulf/provision/filesystems.log on a normal boot.

When done run: wwsh provision set node003 --preshell=UNDEF

to unset the PRESHELL setting.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/99AC497F-5B11-4CA7-8DF1-337E403659DC%40gmail.com.

[Darkned]

Jason, thank you again!

I tried your propose and the result is "/usr/sbin/parted: error while loading shared libraries: libparted.so.2: cannot open shared object file: No such file or directory" while I can found it in /usr/lib

Do you have an idea how to fix it?

Ned.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt8erKznjd1yns4rmQJzxDy8vjabDbZYgNYMArB-eC_CQA%40mail.gmail.com.

[Jason Stover]

Hrmm... from the bootstrap can you do:

ldconfig -p | grep 'libparted'

If that comes up empty run: ldconfig -n /usr/lib

Quick guessing... either the library is a different architecture... or
it's not being picked up in the cache for some reason.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/F381ECBB-16A3-45A9-A92F-31FED6BD49AE%40gmail.com.

[Darkned]

Re,

I tried but ldconfig is not present on the bootstrap (not in PATH nor locatable with find command). An additional idea to fix it?

From my readings of the archived discussions of Warewulf, it seems to come from the bootstrap/kernel version used. Some people seem to have solved this problem by downgrading the bootstrap.

Can you confirm it to me?

Thanks.

Ned.

To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt-eeWoXOqAsPjd5bOrVo%3DocqYDOfgRrt%2B-UTXqp4Lcf0Q%40mail.gmail.com.

[Jason Stover]

> ldconfig is not present on the bootstrap

Yeah ... I completely forgot about not having those tools in there...

As far as downgrading... I can't remember off the top of my head with
100% certainty. I know there has been issues where an updated Kernel
was causing problems but the older worked, but I don't know if this is
one of those cases.

If the libparted* libraries are in /usr/lib ... try copying them to
/usr/lib64 (or /lib64) ... But, the error _could_ be because a
required library that libparted needs can't be found or loaded...
Generally that is:

$ readelf -a initramfs/rootfs/usr/lib/libparted.so.2.0.1 | grep 'NEEDED'
0x0000000000000001 (NEEDED) Shared library: [libuuid.so.1]
0x0000000000000001 (NEEDED) Shared library: [libdevmapper.so.1.02]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]

libdevmapper itself generally has a bunch of requirements. And if one
along the chain that libparted is trying to load fails, you'll get an
error like you did.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/23298F58-6B34-4CC1-A11D-12CCE8138CE7%40gmail.com.

[Darkned]

Jason,

Congrats! Typing "ln -s /usr/lib/libparted.so.2 /lib64/libparted.so.2" solves the issue in the preshell.
Now, because this goes beyond my skills, could you explain me how to make a "good" version of my bootstrap?

Thanks in advance,
Ned.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt9M6YiXePXcGTEME-b1gVgofO5pwoEX7CbTG3npHnKjJg%40mail.gmail.com.

[Jason Stover]

Okay... If that fixes it we're going to need to make make some changes
to install under /lib64 instead of /usr/lib. But, you can do something
like:

--- start cut ---
mkdir /tmp/boot
cd /tmp/boot
cpio -i --no-absolute-filenames < /var/warewulf/initramfs/x86_64/base
[ change link structure around ]
find . | cpio -o -H newc -A -F /var/warewulf/initramfs/x86_64/base
--- end cut ---

You can move the original `base` file out of the way as a backup...
just make sure it's out of the directory structure.

After the new one is built, you can then do: wwsh bootstrap rebuild

And that will rebuild all of your bootstrap images.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/41DE4FFE-7B3D-452A-8466-1472224654B6%40gmail.com.

[Darkned]

Congratulations!!! Thank you very much for solving my problem.
From now, everything is fine for me.

Just a quick additional question on another subject: when I want to add a new file, I notice that I have to specify the full path. Exampe: if I copy /etc/resolv.conf to my current directory and try to add it:
# wwsh file import resolv.conf --name=resolv.conf --path=/etc/resolv.conf
ERROR: the name of the import file contains illegal characters.

But if I specify the full path, everything works fine.

I notice in sources that seem to be as-designed, but I don't figure out why.

Thanks again!!
Ned

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt8fnkF_RbvjaivdoVbfZkfgqObDFojyF8BQJ6si5EFGag%40mail.gmail.com.

[Jason Stover]

Yeah, it has to start with a / character. I can't remember exactly why
it was put in place this way. I think it's because for syncing, we
need to know the exact file to reread in for any possible updates. So
we want the full path to the file we're importing.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/D246CB9D-6A65-47F6-8091-57EA47C2196D%40gmail.com.

[Darkned]

I was just surprised in my first steps with this new version.
So, if I understand correctly, after importing your file, you cannot delete it from your file system (even if it was already ingested in the store)?

Thanks again for spending time helping me.
See you,
Ned.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt9CLBR3C2y%2ByOE1ccNFO%3DCgb09-7m0riAW9_SMLD_miLw%40mail.gmail.com.

[Jason Stover]

It depends on your need. If it's a file that gets sync'd... like what
becomes /etc/passwd or some configuration file, etc... Then no, you
cannot delete it. But if the file never needs to be synced again after
the initial import you can remove it from the filesystem.

If you do: wwsh object dump [file] ... there's a ORIGIN key. This is
the location on disk we're going to be looking for when syncing the
file. I believe this requirement only exists when importing a file.
We're basically saying: Hey take this file on disk and make a file
object from it. If we sync, re-read from disk and update the object.

If you don't need to re-sync a file, you can just unset the ORIGIN
key. IIRC you can do something like:

wwsh file set resolv.conf --origin=UNDEF

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/178023C9-20D5-4ACA-83E1-8336239CB596%40gmail.com.

[Darkned]

HI Jason,

OK, thanks for this additional information.

One more question about filesystem definition: is the "wwsh object modify -s FILESYSTEMS=..." can still be used to define filesystem layout? Or do I have to forgot this way?

See you later, and thanks again!
Ned.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt-BJkUBtM5d%2Bma4a-dg1ewTe0gUm-LQMozv-jb0pZj7xQ%40mail.gmail.com.

[Jason Stover]

You don't need to do that anymore. What you need to do is run: wwsh
provision set [node] -f [cmds file] ... any time you modify the file.

What happens is we take the instructions from the file, and build out
the FS key.

-J

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/3F86195B-F29D-43DF-A1D2-7F335ACFF596%40gmail.com.

[Darkned]

One more time, thank you!!

Ned.

> To view this discussion on the web visit https://groups.google.com/a/lbl.gov/d/msgid/warewulf/CAGfAqt_qz2Oug3OuBD7sgMS%3DgXwPz%3DRaeijj-GbWx9Qd1u3r6Q%40mail.gmail.com.