[slurm-users] Job stuck in pending state with AssocGrpMemLimit

[Rémy Dernat via slurm-users]

Hi !

I have a very strange behavior that I am unable to explain.

A user have a job stuck in pending state, although there are enough resources to run :

#########################################

[dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools 
[dernatr@io-login-01 ~]$ squeue_account_count_running_rss dedicated-cpu@igroup
Number of CPUs used by dedicated-cpu@igroup: 230
Total of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
       8949801_[0] cpu-dedic access_d  doej PD       0:00      1 (AssocGrpMemLimit)
           8936077 cpu-dedic sim11.13 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930985 cpu-dedic sim33.23 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930981 cpu-dedic sim32.14 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930980 cpu-dedic sim31.61 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930979 cpu-dedic sim23.21 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930976 cpu-dedic sim22.27 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930972 cpu-dedic sim21.23 gamma PD       0:00      2 (AssocGrpCpuLimit)
           8930971 cpu-dedic sim13.24 gamma PD       0:00      2 (AssocGrpCpuLimit)
[dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@inrae --state=running -o "%A %m %u %U %C"
JOBID MIN_MEMORY USER UID CPUS
8946435 64G orbital 1064 32
8929322 480G doej 2746 1
8925315 1420G omega 1294 6
8880518 5000M gamma 1326 191
[dernatr@io-login-01 ~]$ scontrol show job 8949801_0
JobId=8949801 ArrayJobId=8949801 ArrayTaskId=0 JobName=access_d49_9
   UserId=doej(2746) GroupId=igroup-infra(1819) MCS_label=N/A
   Priority=1103 Nice=0 Account=dedicated-cpu@igroup QOS=dedicated
   JobState=PENDING Reason=AssocGrpMemLimit Dependency=(null)
   Requeue=0 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:00 TimeLimit=2-00:00:00 TimeMin=N/A
   SubmitTime=2026-08-03T16:59:53 EligibleTime=2026-08-03T16:59:53
   AccrueTime=2026-08-03T16:59:53
   StartTime=Unknown EndTime=Unknown Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-08-04T09:17:19 Scheduler=Main
   Partition=cpu-dedicated AllocNode:Sid=io-login-01:1163082
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=
   NumNodes=1 NumCPUs=1 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=1,mem=560G,node=1,billing=1
   AllocTRES=(null)
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=1 MinMemoryNode=560G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) Network=(null)
   Command=/scratch/users/doej/run_access_d49_23.sh
   WorkDir=/scratch/users/doej
   AdminComment=159722b1-908e-47e2-82b1-08872ee64906 
   StdErr=/scratch/users/doej/slurm-8949801_0.err
   StdIn=/dev/null
   StdOut=/scratch/users/doej/slurm-8949801_0.out
   TresPerTask=cpu=1
[dernatr@io-login-01 ~]$ show_detail_account_association dedicated-cpu@igroup
                                                Account                  Org                          Descr                        GrpTRES     Share     MaxWall   Def QOS                  QOS 
------------------------------------------------------- -------------------- ------------------------------ ------------------------------ --------- ----------- --------- -------------------- 
                                    dedicated-cpu@igroup                igroup                    not_defined              cpu=364,mem=3360G         1             dedicated            dedicate
[dernatr@io-login-01 ~]$ scontrol show conf |grep -i mem
AccountingStorageTRES   = cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu:nvidia_a100_80gb_pcie,gres/gpu:nvidia_a100_80gb_pcie_1g.10gb,gres/gpu:nvidia_a100_80gb_pcie_4g.40gb,gres/gpu:nvidia_h100_80gb_hbm3,gres/gpu:nvidia_h100_80gb_hbm3_1g.10gb,gres/gpu:nvidia_h200,gres/gpu:nvidia_h200_1g.18gb
DefMemPerCPU            = 2048
MaxMemPerNode           = UNLIMITED
PropagateResourceLimitsExcept = MEMLOCK
SelectTypeParameters    = CR_CORE_MEMORY
MemorySwappiness        = (null)
[dernatr@io-login-01 ~]$ sshare -A dedicated-cpu@igroup --users=lambda,doej,omega,gamma |sort -k7 -rn
 dedicated-cpu@igroup    doej          1    0.005495      782016      0.003362   0.335501 
 dedicated-cpu@igroup     omega          1    0.005495     2239436      0.009627   0.335119 
 dedicated-cpu@igroup        lambda          1    0.005495    48189472      0.207149   0.334353 
 dedicated-cpu@igroup gamma          1    0.005495    85232776      0.366384   0.333971 
dedicated-cpu@igroup                      1    0.030303   232632330      0.129448            
Account                    User  RawShares  NormShares    RawUsage  EffectvUsage  FairShare 
-------------------- ---------- ---------- ----------- ----------- ------------- ---------- 

#########################################

As you can see this account has a memory limit set to 3.36T of RAM and he is requesting 560G. The entire running job are taking 2.11 TB.

I also look at backfilling and fair sharing, but I don't see anything that could block him.

Any idea may be useful,

Thanks,

Best regards,

R

-- 
ISDM Lundi/Mardi/Jeudi/Vendredi
ISEM Mercredi

[Reid, Andrew C.E. (Fed) via slurm-users]

Maybe have a look at the submission script and see if there
are any deal-breakers in there, like requesting a run-time
greater than the partition time-limit?

We have sometimes seen this arise if users have set up job
dependencies, a job might look superficially like it should
run, but if it's dependent on another job that hasn't finished,
it will have to wait.

Not sure if that's helpful, but I have seen corner cases
like that sometimes.

-- A.

On Tue, Aug 04, 2026 at 09:28:44AM +0200, Rémy Dernat via slurm-users wrote:
> Hi !
>
> I have a very strange behavior that I am unable to explain.
>
> A user have a job stuck in pending state, although there are enough
> resources to run :
>
> #########################################
>
> [dernatr@io-login-01 ~]$ ml add io-local slurm-user-tools
> [dernatr@io-login-01 ~]$ squeue_account_count_running_rss
> dedicated-cpu@igroup Number of CPUs used by dedicated-cpu@igroup: 230 Total
> of memory used by dedicated-cpu@igroup: Total: 2.11 TB (1.92 TiB)
> [dernatr@io-login-01 ~]$ squeue -A dedicated-cpu@igroup --state=pending    
>          JOBID PARTITION     NAME     USER ST       TIME  NODES
> NODELIST(REASON)        8949801_[0] cpu-dedic access_d  doej PD       0:00 
>     1 (AssocGrpMemLimit)            8936077 cpu-dedic sim11.13 gammaPD     
>  0:00      2 (AssocGrpCpuLimit)            8930985 cpu-dedic sim33.23

> gammaPD       0:00      2 (AssocGrpCpuLimit)            8930981 cpu-dedic
> sim32.14 gammaPD       0:00      2 (AssocGrpCpuLimit)            8930980
> cpu-dedic sim31.61 gammaPD       0:00      2 (AssocGrpCpuLimit)          
>  8930979 cpu-dedic sim23.21 gammaPD       0:00      2 (AssocGrpCpuLimit)    
>        8930976 cpu-dedic sim22.27 gammaPD       0:00      2
> (AssocGrpCpuLimit)            8930972 cpu-dedic sim21.23 gammaPD       0:00 

>     2 (AssocGrpCpuLimit)            8930971 cpu-dedic sim13.24 gammaPD     
>  0:00      2 (AssocGrpCpuLimit) [dernatr@io-login-01 ~]$ squeue -A
> dedicated-cpu@inrae --state=running -o "%A %m %u %U %C" JOBID MIN_MEMORY
> USER UID CPUS 8946435 64G orbital 1064 32 8929322 480G doej 2746 1 8925315

> 1420G omega 1294 6 8880518 5000M gamma1326 191 [dernatr@io-login-01 ~]$

> --
> slurm-users mailing list -- slurm...@lists.schedmd.com
> To unsubscribe send an email to slurm-us...@lists.schedmd.com


--
Dr. Andrew C. E. Reid
Physical Scientist, Computer Operations Administrator
Center for Theoretical and Computational Materials Science
National Institute of Standards and Technology, Mail Stop 8555
Gaithersburg MD 20899 USA
andre...@nist.gov

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Laura Hild via slurm-users]

I don't suppose dedicated-cpu@igroup has any children? I can only imagine Slurm has a different idea of what the totals are than squeue_account_count_running_rss does. Or maybe you can tell us that over the past week you found some other (weirder?) problem.


________________________________________
Od: Rémy Dernat via slurm-users <slurm...@lists.schedmd.com>
Poslano: torek, 4. avgust 2026 03:28
Za: slurm...@lists.schedmd.com
Zadeva: [slurm-users] Job stuck in pending state with AssocGrpMemLimit

[Rémy Dernat UM via slurm-users]

Hi Laura,

Thanks for your suggestion. Unfortunately, no, this account does not have any child account.

Best regards

Le 10 août 2026 18:37:56 GMT+02:00, Laura Hild <l...@jlab.org> a écrit :

I don't suppose dedicated-cpu@igroup has any children?  I can only imagine Slurm has a different idea of what the totals are than squeue_account_count_running_rss does.  Or maybe you can tell us that over the past week you found some other (weirder?) problem.

[Luis Miguel Silva via slurm-users]

Hi Rémy,

I might be missing something obvious, but I was curious about the difference between what squeue_account_count_running_rss reports and what Slurm is enforcing.

The CPU counts line up correctly (32 + 1 + 6 + 191 = 230), so the script looks to be finding the right jobs. My understanding is that squeue's %m shows the per-CPU value for --mem-per-cpu jobs, so a straight sum of the column would under count things.

So is 8880518 possibly a --mem-per-cpu job? 5000M against 191 CPUs made me wonder, and it's the only one of the four it could be: when requesting per-CPU, 8946435 would charge 2048G (64G x 32) and 8925315 would charge 8520G (1420G x 6), both well over the 3360G limit.

If that's what's happening, usage would be around 2896G against the 3360G GrpTRES, leaving less than the 560G the pending job wants — and AssocGrpMemLimit would just be correct.

Rather than trusting my math, though, scontrol on that job should answer it for us. It'll show either MinMemoryCPU or MinMemoryNode, and AllocTRES will give us the real total:

    scontrol show job 8880518

We can also check the association manager's own view of current usage against the limit:

    scontrol show assoc_mgr account=dedicated-cpu@igroup flags=assoc

By the way, multi-node jobs would have a similar issue: %m is per node, so a two-node job charges twice what the column shows.

Still on the child accounts question, maybe it would be worth looking at doej's own user association under the account? A GrpTRES there would raise the same reason code, and scontrol show job won't say which association actually tripped it. 

The tree view will show both the account and the user rows:
    sacctmgr show assoc where account=dedicated-cpu@igroup format=account,parentname,user,grptres

Entirely possible you've been through all this already, feel free to ignore if so!

Best,
Luis Silva