[slurm-users] Split node into 2 partitions

[Ratnasamy, Fritz via slurm-users]

Hi,

 Is there a straightforward way to split a gpu node with 8 GPUs into 2 partitions with 4 GPUs on each? How do we edit the slurm.conf and gres.conf in this case? 
Thanks,

Fritz Ratnasamy
Data Scientist
Information Technology

[Xaver Stiensmeier via slurm-users]

Hey,

I haven't used it myself, but I think http://www.schedmd.com/slurmdocs/faq.html#multi_slurmd is what you are looking for.

Best regards,
Xaver

[Cutts, Tim via slurm-users]

I would ask why you want to do this.  There are almost certainly alternatives, depending on what it is you want to achieve.

Splitting into multiple partitions will almost certainly result in lower overall utilisation of the system, and prevents anyone ever using all 8 GPUs.  I’d also be concerned that this sort of approach is quite complex, and easy to make a mess of the configuration.

If it’s a matter of making sure that certain jobs have priority, and a certain amount of the capacity, you can do that with QoS (for example, you could set a QoS so that long running jobs can only ever use half the GPUs, leaving half for shorter running jobs).  Doing that is more flexible, since SLURM can decide at any time which GPUs it allocates to which kind of task, rather than you hard coding it in the partitioning.  You can combine this with overlapping partitions, so that it still looks like partitions.  For example, you could have the node belong to both short and long partitions, give the short partition higher priority, and set partition QoS so that long can only ever use half of the GPUs; this is what we do with most of our nodes, and it generally works pretty well.

Tim

---

AstraZeneca UK Limited is a company incorporated in England and Wales with registered number:03674842 and its registered office at 1 Francis Crick Avenue, Cambridge Biomedical Campus, Cambridge, CB2 0AA.

This e-mail and its attachments are intended for the above named recipient only and may contain confidential and privileged information. If they have come to you in error, you must not copy or show them to anyone; instead, please reply to this e-mail, highlighting the error to the sender and then immediately delete the message. For information about how AstraZeneca UK Limited and its affiliates may process information, personal data and monitor communications, please see our privacy notice at www.astrazeneca.com