[slurm-users] SLURM configuration help

Skip to first unread message

Alison Peterson via slurm-users

Apr 4, 2024, 11:42:25 AMApr 4
to slurm...@lists.schedmd.com
I am writing to seek assistance with a critical issue on our single-node system managed by Slurm. Our jobs are queued and marked as awaiting resources, but they are not starting despite seeming availability. I'm new with SLURM and my only experience was a class on installing it so I have no experience, running it or using it. 

Issue Summary:

Main Problem: Jobs submitted only one run and the second says NODELIST(REASON) (Resources). I've checked that our single node has enough RAM (2TB) and CPU's (64) available.

NodeName=cusco CPUs=64 Sockets=2 CoresPerSocket=32 ThreadsPerCore=1 RealMemory=2052077 Gres=gpu:1,gpu:1,gpu:1,gpu:1
PartitionName=mainpart Default=YES MinNodes=1 DefaultTime=00:60:00 MaxTime=UNLIMITED AllowAccounts=ALL Nodes=ALL State=UP OverSubscribe=Force

System Details: We have a single-node setup with Slurm as the workload manager. The node appears to have sufficient resources for the queued jobs.

Troubleshooting Performed:
Configuration Checks: I have verified all Slurm configurations and the system's resource availability, which should not be limiting job execution.
Service Status: The Slurm daemon slurmdbd is active and running without any reported issues. System resource monitoring shows no shortages that would prevent job initiation.

Any guidance and help will be deeply appreciated!

Alison Peterson
IT Research Support Analyst
Information Technology
San Diego State University | SDSU.edu
5500 Campanile Drive | San Diego, CA 92182-8080

Renfro, Michael via slurm-users

Apr 4, 2024, 11:59:27 AMApr 4
to Alison Peterson, slurm...@lists.schedmd.com

What does “scontrol show node cusco” and “scontrol show job PENDING_JOB_ID” show?


On one job we currently have that’s pending due to Resources, that job has requested 90 CPUs and 180 GB of memory as seen in its ReqTRES= value, but the node it wants to run on only has 37 CPUs available (seen by comparing its CfgTRES= and AllocTRES= values).


From: Alison Peterson via slurm-users <slurm...@lists.schedmd.com>
Date: Thursday, April 4, 2024 at 10:43
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: [slurm-users] SLURM configuration help

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

Image removed by sender.


Renfro, Michael via slurm-users

Apr 4, 2024, 1:13:58 PMApr 4
to Alison Peterson, Slurm User Community List

Yep, from your scontrol show node output:



The running job (77) has allocated 1 CPU and all the memory on the node. That’s probably due to the partition using the default DefMemPerCPU value [1], which is unlimited.


Since all our nodes are shared, and our workloads vary widely, we set our DefMemPerCPU value to something considerably lower than mem_in_node/cores_in_node . That way, most jobs will leave some memory available by default, and other jobs can use that extra memory as long as CPUs are available.


[1] https://slurm.schedmd.com/slurm.conf.html#OPT_DefMemPerCPU


From: Alison Peterson <apete...@sdsu.edu>
Date: Thursday, April 4, 2024 at 11:58
To: Renfro, Michael <Ren...@tntech.edu>
Subject: Re: [EXT] Re: [slurm-users] SLURM configuration help

External Email Warning

This email originated from outside the university. Please use caution when opening attachments, clicking links, or responding to requests.

Here is the info: 

sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ scontrol show node cusco

NodeName=cusco Arch=x86_64 CoresPerSocket=32
   CPUAlloc=1 CPUTot=64 CPULoad=0.02
   NodeAddr=cusco NodeHostName=cusco Version=19.05.5
   OS=Linux 5.4.0-172-generic #190-Ubuntu SMP Fri Feb 2 23:24:22 UTC 2024
   RealMemory=2052077 AllocMem=2052077 FreeMem=1995947 Sockets=2 Boards=1
   State=MIXED ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   BootTime=2024-03-01T17:06:26 SlurmdStartTime=2024-03-01T17:06:53
   CurrentWatts=0 AveWatts=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s


sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ squeue

                78  mainpart CF1090_w      sma PD       0:00      1 (Resources)
                77  mainpart CF0000_w      sma  R       0:26      1 cusco

sma@cusco:/data/work/sma-scratch/tohoku_wOcean$ scontrol show job 78

JobId=78 JobName=CF1090_wOcean500m.shell
   UserId=sma(1008) GroupId=myfault(1001) MCS_label=N/A
   Priority=4294901720 Nice=0 Account=(null) QOS=(null)
   JobState=PENDING Reason=Resources Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=01:00:00 TimeMin=N/A
   SubmitTime=2024-04-04T09:55:34 EligibleTime=2024-04-04T09:55:34
   StartTime=2024-04-04T10:55:28 EndTime=2024-04-04T11:55:28 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2024-04-04T09:55:58
   Partition=mainpart AllocNode:Sid=newcusco:2450574
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=(null) SchedNodeList=cusco
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=YES Contiguous=0 Licenses=(null) Network=(null)


Error! Filename not specified.


Alison Peterson via slurm-users

Apr 4, 2024, 1:57:31 PMApr 4
to Renfro, Michael, slurm...@lists.schedmd.com
Thank you!!!! That was the issue, I'm so happy :-) sending you many thanks. 

Reply all
Reply to author
0 new messages