Re: [slurm-users] Usage of particular GPU out of 4 GPUs while submitting

[Ravi Konila]

Hi Daniel Letai

Thanks for the quick response and guidance.

I have done the changes as mentioned in gres.conf and slurm.conf and now I
am able to submit the jobs to a particular GPU.

Regarding MIG, it was just a thought came in m mind, in case studentA wants
to submit jobs to both GPU partition (20G and 5G). But anyhow, referred
Nvidia MIG user guide and your suggestion as mentioned above, I am clear
now.

Thanks a lot for the support.


With Warm Regards
Ravi Konila

-----Original Message-----
From: slurm-use...@lists.schedmd.com
Sent: Monday, November 20, 2023 5:30 PM
To: slurm...@lists.schedmd.com
Subject: slurm-users Digest, Vol 73, Issue 31

Send slurm-users mailing list submissions to
slurm...@lists.schedmd.com

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.schedmd.com/cgi-bin/mailman/listinfo/slurm-users
or, via email, send a message with subject or body 'help' to
slurm-use...@lists.schedmd.com

You can reach the person managing the list at
slurm-us...@lists.schedmd.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of slurm-users digest..."


Today's Topics:

1. Re: SLURM new user query, does SLURM has GUI /Web based
management version also (Joseph John)
2. Usage of particular GPU out of 4 GPUs while submitting jobs
to DGX Server (Ravi Konila)
3. Re: Usage of particular GPU out of 4 GPUs while submitting
jobs to DGX Server (Daniel Letai)


----------------------------------------------------------------------

Message: 1
Date: Mon, 20 Nov 2023 03:44:48 +0000
From: Joseph John <jjk_...@yahoo.com>
To: "Ole.H....@fysik.dtu.dk" <Ole.H....@fysik.dtu.dk>, Slurm
User Community List <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] SLURM new user query, does SLURM has GUI
/Web based management version also
Message-ID:
<DU0PR10MB5775509BC2...@DU0PR10MB5775.EURPRD10.PROD.OUTLOOK.COM>

Content-Type: text/plain; charset="us-ascii"

Thanks Ole
I was able to setup the SLURM for 4 nodes and tried out some python code
using srun and trying to understand and practice more of SLURM commands
Thanks for the reply
Joseph John


From: slurm-users <slurm-use...@lists.schedmd.com> on behalf of Ole
Holm Nielsen <Ole.H....@fysik.dtu.dk>
Date: Sunday, 19 November 2023 at 2:35 PM
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: Re: [slurm-users] SLURM new user query, does SLURM has GUI /Web
based management version also
On 19-11-2023 09:11, Joseph John wrote:
> I am new user, trying out SLURM
>
> Like to check if the SLURM has a GUI/web based management tool also

Did you read the Quick Start Administrator Guide at
https://slurm.schedmd.com/quickstart_admin.html ?

I don't believe there are any Slurm management tools as a web GUI, and
that would probably be a security nightmare anyway because privileged
system access is required.

There are a number of monitoring tools for viewing the status of Slurm jobs.

/Ole
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/10e9cbc3/attachment-0001.htm>

------------------------------

Message: 2
Date: Mon, 20 Nov 2023 10:06:42 +0530
From: "Ravi Konila" <ravi...@gmail.com>
To: <slurm...@lists.schedmd.com>
Subject: [slurm-users] Usage of particular GPU out of 4 GPUs while
submitting jobs to DGX Server
Message-ID: <8ED1EDA8185C4F1CAA1D0AB2D216B4B8@RAVIKONILAPC>
Content-Type: text/plain; charset="iso-8859-1"

Hello Everyone

I am just beginner of slurm and started to use the same on our DGX Server
which has 4 numbers of A100, 80GB GPUs.
Everything works fine, jobs goes to random GPUs (free available).
My question is related to submission of jobs to those GPUs. How do a student
submit the job to a particular GPU out of 4 GPUs? For example, studentA
should submit the job to GPU ID 1 instead of GPU ID 0.

Also we are planning for MIG in the server and we would like few students to
submit the jobs to 20G partition and non critical jobs to 5G partition.
How should be the slurm.conf and gres.conf in this case.

Currently our configuration is as below:

gres.conf
Name=gpu type=A100 file=/dev/nvidia[0-2,4]

------------
slurm.conf
.
.
.
GresTypes=gpu
NodeName=rl-dgxs-r21-l2 Gres=gpu:A100:4 CPUs=128 RealMemory=500000
State=UNKNOWN
PartitionName=LocalGPUQ Nodes=ALL Default=YES MaxTime=INFINITE State=UP

-------------

Any suggestions or help in this regard is highly appreciated.

With Warm Regards
Ravi Konila
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/018a699b/attachment-0001.htm>

------------------------------

Message: 3
Date: Mon, 20 Nov 2023 10:09:48 +0200
From: Daniel Letai <da...@letai.org.il>
To: slurm...@lists.schedmd.com
Subject: Re: [slurm-users] Usage of particular GPU out of 4 GPUs while
submitting jobs to DGX Server
Message-ID: <31022502-46de-4a89...@letai.org.il>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL:
<http://lists.schedmd.com/pipermail/slurm-users/attachments/20231120/cbdd1ef0/attachment-0001.htm>

End of slurm-users Digest, Vol 73, Issue 31
*******************************************