[slurm-users] Slurm - Munge configuration details

[Zainul Abiddin]

Hi All,

I am new to Slurm and trying to setup Slurm20.11.2 on Centos 7

My environment is Master node (smaster) + compute Node (snode)

and i am using https://www.slothparadise.com/how-to-install-slurm-on-centos-7-cluster/ link to setup Slurm on Master and compute nodes.

I have tried installing Munge on both the nodes and it's running fine. However when i try to run the Munge command from Master to Node its asking password.

export MUNGEUSER=1001
groupadd -g $MUNGEUSER munge
useradd  -m -c "MUNGE Uid 'N' Gid Emporium" -d /var/lib/munge -u $MUNGEUSER -g munge  -s /sbin/nologin munge
export SlurmUSER=1002
groupadd -g $SlurmUSER slurm
useradd  -m -c "Slurm workload manager" -d /var/lib/slurm -u $SlurmUSER -g slurm  -s /bin/bash slurm

yum install -y epel-release
yum install munge munge-libs munge-devel -y

yum install rng-tools -y
rngd -r /dev/urandom
/usr/sbin/create-munge-key -r
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key

scp /etc/munge/munge.key root@snode:/etc/munge

chown munge: /etc/munge/munge.key
chmod 400 /etc/munge/munge.key
chown -R munge: /etc/munge/ /var/log/munge/
chmod 0700 /etc/munge/ /var/log/munge/

systemctl enable munge
systemctl start munge
systemctl status munge

[root@smaster ~]# systemctl status munge
? munge.service - MUNGE authentication service
   Loaded: loaded (/usr/lib/systemd/system/munge.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-02-01 12:52:54 IST; 1h 4min ago
     Docs: man:munged(8)
  Process: 2547 ExecStart=/usr/sbin/munged (code=exited, status=0/SUCCESS)
 Main PID: 2550 (munged)
    Tasks: 4
   CGroup: /system.slice/munge.service
           +-2550 /usr/sbin/munged

Feb 01 12:52:54 smaster.calligotech.com systemd[1]: Starting MUNGE authentication service...
Feb 01 12:52:54 smaster.calligotech.com systemd[1]: Started MUNGE authentication service.
[root@smaster ~]# munge -n
MUNGE:AwQDAAAg5PQzQhz/D4h7OGUU4Cx4QAgZ4z/0MMt0SP+uhuP927Xcl2t8EC4izsUj6xpMRslnIb2g4RCz2vayu0wW1o8mNNuy7cVv/PmsuO9XsAJ7aLl1n/M=:
[root@smaster ~]#

Below is the screenshot for reference.

Smaster:

Snode:

Am I configuring properly or Do I need to set up passwordless authentication on Master to Node and vice-versa? 

Please clarify to me, whether Mugne will do passwordless login else do we need to setup passwordless.

Please guide me with a proper setup link/Doc which includes Munge Configuration, Slurm account database Daemon configuration and Slurm installation and configuration with testing simple jobs on Master and Compute Nodes.

--

Regards

Zain

[Zainul Abiddin]

Hi,

[root@smaster ~]# munge -n | unmunge
STATUS:           Success (0)
ENCODE_HOST:      smaster.calligotech.com (192.168.1.195)
ENCODE_TIME:      2021-02-01 13:58:04 +0530 (1612168084)
DECODE_TIME:      2021-02-01 13:58:04 +0530 (1612168084)
TTL:              300
CIPHER:           aes128 (4)
MAC:              sha1 (3)
ZIP:              none (0)
UID:              root (0)
GID:              root (0)
LENGTH:           0

[root@smaster ~]# munge -n | ssh snode unmunge
root@snode's password:
STATUS:           Success (0)
ENCODE_HOST:      smaster.calligotech.com (192.168.1.195)
ENCODE_TIME:      2021-02-01 13:58:16 +0530 (1612168096)
DECODE_TIME:      2021-02-01 13:58:21 +0530 (1612168101)
TTL:              300
CIPHER:           aes128 (4)
MAC:              sha1 (3)
ZIP:              none (0)
UID:              root (0)
GID:              root (0)
LENGTH:           0

[root@smaster ~]# 

--

Thanks and Regards

Zainul Abiddin

Mob: +91 7406649679

[Benson Muite]

> Feb 01 12:52:54 smaster.calligotech.com <http://smaster.calligotech.com>

> systemd[1]: Starting MUNGE authentication service...

> Feb 01 12:52:54 smaster.calligotech.com <http://smaster.calligotech.com>

> systemd[1]: Started MUNGE authentication service.
> [root@smaster ~]# munge -n
> MUNGE:AwQDAAAg5PQzQhz/D4h7OGUU4Cx4QAgZ4z/0MMt0SP+uhuP927Xcl2t8EC4izsUj6xpMRslnIb2g4RCz2vayu0wW1o8mNNuy7cVv/PmsuO9XsAJ7aLl1n/M=:
> [root@smaster ~]#
>
> Below is the screenshot for reference.
> Smaster:

> image.png
>
> Snode:

> image.png
>
> Am I configuring properly or Do I need to set up passwordless
> authentication on Master to Node and vice-versa?
>
> Please clarify to me, whether Mugne will do passwordless login else
> do we need to setup passwordless.
>
> Please guide me with a proper setup link/Doc which includes Munge
> Configuration, Slurm account database Daemon configuration and Slurm
> installation and configuration with testing simple jobs on Master and
> Compute Nodes.
>
> --

> *Regards*
> *Zain*
>
Are you able to do passwordless ssh between the nodes?

May also find the following helpful:
https://github.com/dun/munge/wiki/Installation-Guide
https://southgreenplatform.github.io/trainings/hpc/slurminstallation/

[Benson Muite]

See also
https://wiki.fysik.dtu.dk/niflheim/Slurm_installation

[Zainul Abiddin]

Hi Benson,

I am not able to do passwordless ssh  between master and compute nodes using Munge service.

when i am running below command , here it is asking for a password for the compute node.

Am I configuring properly or not, so I need clarity on this?

[root@smaster ~]# munge -n | ssh snode unmunge
root@snode's password:
STATUS:           Success (0)
ENCODE_HOST:      smaster.calligotech.com (192.168.1.195)
ENCODE_TIME:      2021-02-01 13:58:16 +0530 (1612168096)
DECODE_TIME:      2021-02-01 13:58:21 +0530 (1612168101)
TTL:              300
CIPHER:           aes128 (4)
MAC:              sha1 (3)
ZIP:              none (0)
UID:              root (0)
GID:              root (0)
LENGTH:           0

[root@smaster ~]# 

Regards,

Zain

[Benson Muite]

On 2/2/21 4:00 PM, Zainul Abiddin wrote:
> Hi Benson,
>
> I am not able to do passwordless ssh  between master and compute nodes
> using Munge service.
> when i am running below command , here it is asking for a password for
> the compute node.
>

> /Am I configuring properly or not, so I need clarity on this?/

>
> [root@smaster ~]# munge -n | ssh snode unmunge
> root@snode's password:
> STATUS:           Success (0)
> ENCODE_HOST: smaster.calligotech.com

> <http://smaster.calligotech.com/> (192.168.1.195)

> ENCODE_TIME:      2021-02-01 13:58:16 +0530 (1612168096)
> DECODE_TIME:      2021-02-01 13:58:21 +0530 (1612168101)
> TTL:              300
> CIPHER:           aes128 (4)
> MAC:              sha1 (3)
> ZIP:              none (0)
> UID:              root (0)
> GID:              root (0)
> LENGTH:           0
>
> [root@smaster ~]#
>
> Regards,
> Zain
>

Hi Zain,

Perhaps try using the ipaddress instead of the hostname?

Also, are clocks synchronized? See
https://slurm.schedmd.com/quickstart_admin.html
Benson