[slurm-users] avoid using same GPU by the interactive job

[navin srivastava via slurm-users]

hi,

facing an issue in my environment where the batch job and the interactive job use the same gpu.

Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another job is submitted interactively then it uses the same GPU . Is there a way to avoid this? 

GresTypes=gpu

NodeName=node[01-02] NodeAddr=node[01-02] CPUs=48 Boards=1 SocketsPerBoard=2 CoresPerSocket=24 ThreadsPerCore=1 TmpDisk=6000000 RealMemory=515634 Feature=A100 Gres=gpu:2

PartitionName=onprem Nodes=node[01-10] Default=YES MaxTime=21-00:00:00 DefaultTime=3-00:00:00 State=UP Shared=YES OverSubscribe=NO 

 gres.conf:

Name=gpu File=/dev/nvidia0

Name=gpu File=/dev/nvidia1

Any suggestions on this.

Regards

Navin

[Chintanadilok, Jesse via slurm-users]

Navin,

 

You can isolate GPUs per job if you have cgroups set up properly. What OS are you using? Newer OSes will support cgroupsv2 out of the box, but if necessary you can continue using v1, this workflow should be applicable for both.

 

Add ConstrainDevices=yes to your cgroup.conf

 

This is what the file looks like at my site:

/etc/slurm/cgroup.conf

CgroupMountpoint="/sys/fs/cgroup"

ConstrainCores=yes

ConstrainRAMSpace=yes

ConstrainSwapSpace=no

ConstrainDevices=yes

 

You can find the documentation here:

https://slurm.schedmd.com/cgroup.conf.html

 

If you want to share GPUs you can use CUDA MPS or MIG if your GPU supports it.

 

Regards,

Jesse Chintanadilok

 

From: navin srivastava via slurm-users <slurm...@lists.schedmd.com>
Sent: Wednesday, February 12, 2025 10:30
To: Slurm User Community List <slurm...@lists.schedmd.com>
Subject: [EXTERNAL] [slurm-users] avoid using same GPU by the interactive job

 

hi, facing an issue in my environment where the batch job and the interactive job use the same gpu. Each server has 2 gpu. When 2 batch jobs are running it works fine and use the 2 different gpu's. but if one batch job is running and another

ZjQcmQRYFpfptBannerStart

This message was sent from outside of Texas Instruments. Do not click links or open attachments unless you recognize the source of this email and know the content is safe.     Report Suspicious    ‌

ZjQcmQRYFpfptBannerEnd

[navin srivastava via slurm-users]

Thank you Jesse.

I am  using Enterprise SLES15SP6 as the OS. I have not introduced the cgroup functionality in my environment. I can think about it and will see if this solution works out. but is there any other way to use without Cgroup to achieve the same.  Batch job requests are fine 2 jobs with each one GPU request works fine.  in the case of mix( 1 batch job and other Interactive job) creating the problem. 

Is there a way I can run a job and apply the exclusive way only on GPU resources? 

Regards

Navin.

[Michael Gutteridge via slurm-users]

Well that's kind of the core issue- without cgroups _any_ process in the job will have access to all of the GPUs on the system and there's not much more that Slurm can do about it at that point.

I would have a look at the environment variable CUDA_VISIBLE_DEVICES.  It is set by Slurm and should have an index (0, 1, 2, etc.) directing applications to an appropriate GPU.  I think it's more a case that the batch processes are honoring that variable and the interactive job is not.

 - Michael

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com