[slurm-users] How to fix a node in state=inval?

[Jan Andersen]

I am building a cluster exclusively with dynamic nodes, which all boot
up over the network from the same system image (Debian 12); so far there
is just one physical node, as well as a vm that I have used for the
initial tests:

# sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
all* up infinite 1 inval gpu18c04d858b05
all* up infinite 1 down* node080027aea419

When I compare what the master node thinks of gpu18c04d858b05 with what
the node itself reports, they seem to agree:

On gpu18c04d858b05:

root@gpu18c04d858b05:~# slurmd -C
NodeName=gpu18c04d858b05 CPUs=16 Boards=1 SocketsPerBoard=1
CoresPerSocket=8 ThreadsPerCore=2 RealMemory=64240
UpTime=0-18:04:06

And on the master:

# scontrol show node gpu18c04d858b05
NodeName=gpu18c04d858b05 Arch=x86_64 CoresPerSocket=8
CPUAlloc=0 CPUEfctv=16 CPUTot=16 CPULoad=0.16
AvailableFeatures=(null)
ActiveFeatures=(null)
Gres=gpu:geforce:1
NodeAddr=192.168.50.68 NodeHostName=gpu18c04d858b05 Version=23.02.3
OS=Linux 6.1.0-9-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.27-1
(2023-05-08)
RealMemory=64240 AllocMem=0 FreeMem=63739 Sockets=1 Boards=1
State=DOWN+DRAIN+DYNAMIC_NORM+INVALID_REG ThreadsPerCore=2 TmpDisk=0
Weight=1 Owner=N/A MCS_label=N/A
Partitions=all
BootTime=2023-08-31T15:25:55 SlurmdStartTime=2023-08-31T15:26:20
LastBusyTime=2023-08-31T10:24:01 ResumeAfterTime=None
CfgTRES=cpu=16,mem=64240M,billing=16
AllocTRES=
CapWatts=n/a
CurrentWatts=0 AveWatts=0
ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
Reason=hang [root@2023-08-31T16:38:27]

I tried to fix it with:

# scontrol update nodename=gpu18c04d858b05 state=down reason=hang
# scontrol update nodename=gpu18c04d858b05 state=resume

However, that made no difference; what is the next step in
troubleshooting this issue?

[Markuske, William]

Are you starting the slurmd via 'slurmd -Z' on the dyanmic node?

The next steps would be to check the slurmctld log from the master and slurmd log for the invalid node. Those should provide more insight into why the node is seen as invalid. If you can attach those we might be able to see the issue.

Regards,

--
Willy Markuske

HPC Systems Engineer

MS Data Science and Engineering
SDSC - Research Data Services
(619) 519-4435

wmar...@sdsc.edu