[slurm-users] srun using infiniband

[Anne Hammond]

We have a 

  CentOS 8.5 cluster 

  slurm 20.11

  Mellanox ConnectX 6 HDR IB and Mellanox 32 port switch

Our application is not scaling.  I discovered the process communications are going over ethernet, not ib.  I used the ifconfig count for the eno2 (ethernet) and ib0 (infiniband) interfaces at end of a job, and subtracted the count at the beginning.   We are using sbatch and

srun {application}

If I interactively login to a node and use the command

mpiexec -iface ib0 -n 32 -machinefile machinefile {application}

where machinefile contains 32 lines with the ib hostname:

ne08-ib

ne08-ib

...

ne09-ib

ne09-ib

the application runs over ib and scales.  

/etc/slurm/slurm.conf uses the ethernet interface for administrative communications and allocation:

NodeName=ne[01-09] CPUs=32 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN

PartitionName=neon-noSMT Nodes=ne[01-09] Default=NO MaxTime=3-00:00:00 DefaultTime=4:00:00 State=UP OverSubscribe=YES

I've read this is the recommended configuration.

I looked for srun parameters that would instruct srun to run over the ib interface when the job is run through the slurm queue.  

I found the --network parameter:

srun --network=DEVNAME=mlx5_ib,DEVTYPE=IB

but there is not much documentation on this and I haven't been able to run a job yet.

Is this the way we should be directing srun to run the executable over infiniband?

Thanks in advance,

Anne Hammond

[Daniel Letai]

Hello Anne,

On 01/09/2022 02:01:53, Anne Hammond wrote:

We have a 

  CentOS 8.5 cluster 

  slurm 20.11

  Mellanox ConnectX 6 HDR IB and Mellanox 32 port switch

Our application is not scaling.  I discovered the process communications are going over ethernet, not ib.  I used the ifconfig count for the eno2 (ethernet) and ib0 (infiniband) interfaces at end of a job, and subtracted the count at the beginning.   We are using sbatch and

srun {application}

If I interactively login to a node and use the command

mpiexec -iface ib0 -n 32 -machinefile machinefile {application}

Is your application using IPoIB or RDMA?

where machinefile contains 32 lines with the ib hostname:

ne08-ib

ne08-ib

...

ne09-ib

ne09-ib

the application runs over ib and scales.  

/etc/slurm/slurm.conf uses the ethernet interface for administrative communications and allocation:

NodeName=ne[01-09] CPUs=32 Sockets=2 CoresPerSocket=16 ThreadsPerCore=1 State=UNKNOWN

PartitionName=neon-noSMT Nodes=ne[01-09] Default=NO MaxTime=3-00:00:00 DefaultTime=4:00:00 State=UP OverSubscribe=YES

I've read this is the recommended configuration.

I looked for srun parameters that would instruct srun to run over the ib interface when the job is run through the slurm queue.  

I found the --network parameter:

srun --network=DEVNAME=mlx5_ib,DEVTYPE=IB

What is the output of

srun --mpi=list ?

but there is not much documentation on this and I haven't been able to run a job yet.

Is this the way we should be directing srun to run the executable over infiniband?

Thanks in advance,

Anne Hammond

-- 
Regards,
--Dani_L.