Hello,
It also appears that random jobs are being identified as using too much memory, despite being well within limits.
For example, a job is running that requested 2048 MB per CPU and all processes are within the limit. But, the job is identified as being over limit when it isn't. Please see below:
> [2022-05-18T10:26:45.020] Job 7724384 exceeded memory limit (13140>2056), cancelling it
> [2022-05-18T10:26:53.736] Job 7724384 exceeded memory limit (13140>2056), cancelling it
> PID PSR RSS VSZ ELAPSED COMMAND
> 16893 1 844172 1337540 15:53 vasp
> 16885 0 842844 1337424 15:53 vasp
> 16892 14 841216 1337652 15:53 vasp
> 16888 6 840848 1337540 15:53 vasp
> 16896 7 840756 1337536 15:53 vasp
> 16891 12 840664 1337544 15:53 vasp
> 16898 11 840556 1337540 15:53 vasp
> 16900 15 840512 1337656 15:53 vasp
> 16890 10 840500 1337540 15:53 vasp
> 16889 8 840468 1337536 15:53 vasp
> 16895 5 840416 1337544 15:53 vasp
> 16886 2 840356 1337540 15:53 vasp
> 16887 4 840348 1337540 15:53 vasp
> 16897 9 840288 1337544 15:53 vasp
> 16894 3 840128 1337536 15:53 vasp
> 16899 13 839952 1337544 15:53 vasp
> ps -U USER -o pid,psr,rss,vsz,etime,args h --sort=-rss|awk '{rss+=$3} END {printf("%.2f\n",rss/1024)}'
> 13140.52
This job isn't hasn't exceeded its memory request on a per CPU basis, yet it has been identified as having done so. Unfortunately, this will cause concern with users because their output logs indicate that the job exceeded its memory limit.
I ran a few test jobs requesting 1K of memory and they were all terminated and identified correctly, so this new behavior is random and frequent.
The commonality between both properly and improperly misidentified jobs is that they are not terminated and their output files log the message from slurmstepd that the memory limit has been exceeded.
Are any other users on the 20.11.9 branch experiencing this issue? Are any users on the 21.08.8 branch experiencing this issue? We use JobAcctGatherParams=OverMemoryKill in our environment to monitor and kill jobs when the physical memory limit has been exceeded.
Thank you,
John DeSantis