[slurm-users] StateSaveLocation and Slurm HA

[Pierre Abele via slurm-users]

Hi all,

I am looking for a clean way to set up Slurms native high availability
feature. I am managing a Slurm cluster with one control node (hosting
both slurmctld and slurmdbd), one login node and a few dozen compute
nodes. I have a virtual machine that I want to set up as a backup
control node.

The Slurm documentation says the following about the StateSaveLocation
directory:

> The directory used should be on a low-latency local disk to prevent file system delays from affecting Slurm performance. If using a backup host, the StateSaveLocation should reside on a file system shared by the two hosts. We do not recommend using NFS to make the directory accessible to both hosts, but do recommend a shared mount that is accessible to the two controllers and allows low-latency reads and writes to the disk. If a controller comes up without access to the state information, queued and running jobs will be cancelled. [1]

My question: How do I implement the shared file system for the
StateSaveLocation?

I do not want to introduce a single point of failure by having a single
node that hosts the StateSaveLocation, neither do I want to put that
directory on the clusters NFS storage since outages/downtime of the
storage system will happen at some point and I do not want that to cause
an outage of the Slurm controller.

Any help or ideas would be appreciated.

Best,
Pierre


[1] https://slurm.schedmd.com/quickstart_admin.html#Config

--
Pierre Abele, M.Sc.

HPC Administrator
Max-Planck-Institute for Evolutionary Anthropology
Department of Primate Behavior and Evolution

Deutscher Platz 6
04103 Leipzig

Room: U2.80
E-Mail: pierre...@eva.mpg.de
Phone: +49 (0) 341 3550 245

[Fabio Ranalli via slurm-users]

You can try DRBD
https://linbit.com/drbd/

or a shared-disk (clustered) FS like GFS2, OCFS2, etc

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html-single/configuring_gfs2_file_systems/index
https://docs.oracle.com/en/operating-systems/oracle-linux/9/shareadmin/shareadmin-ManagingtheOracleClusterFileSystemVersion2inOracleLinux.html

--

Fabio Ranalli | Principal Systems Administrator

[Davide DelVento via slurm-users]

Are you seeking something simple rather than sophisticated? If so, you can use the controller local disk for StateSaveLocation and place a cron job (on the same node or somewhere else) to take that data out via e.g. rsync and put it where you need it (NFS?) for the backup control node to use if/when needed. That obviously introduces a time delay which might or might not be problematic depending on what kind of failures you are trying to protect from and with what level of guarantee you wish the HA would have: you will not be protected in every possible scenario. On the other hand, given the size of the cluster that might be adequate and it's basically zero effort, so it might be "good enough" for you.

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com