[slurm-users] Pending job will be cancelled since transport endpoint is not connected

[Chenyang Yan]

Hi, all

We have one cluster with Slurm version 20.11.8 in CentOS 8.2. Suddenly it produces a wired problem proid for only Pending job will be cancelled since transport endpoint is not connected error(See image https://user-images.githubusercontent.com/19144683/229037078-ca704ba8-23a4-4948-9d1a-bacab82acd1f.png). The all jobs are submitted with srun command.
... ...
srun:job 6367724 queued and waiting for resources
srun:error:Unable to allocate resources: Transport endpoint is not connected
srun:job 6367725 queued and waiting for resources
srun:error: Unable to allocate resources: Transport endpoint is not connected
srun:job 6367726 queued and waiting for resources
srun:job 6367727 queued and waiting for resources
srun:job 6367728 queued and waiting for resources
srun:error: Unable to allocate resources: Transport endpoint is not connected
srun:Force Terminated job 6366908

[root@slurm-master01 bin]# journalctl --since today -p err __COMM=slurmctld
Mar 31 02:50:46 slurm-master01 slurmctld[220654]: error: slurm_receive_msgs: Transport endpoint is not connected
Mar 31 02:50:47 slurm-master01 slurmctld[220654]: error: slurm receive_msgs: Transport endpoint is not connected

According to https://github.com/SchedMD/slurm/blob/slurm-20-11-8-1/src/srun/libsrun/allocate.c#L182-L227 , it seems OS issue? I've google for "transport endpoint is not connected", lots of references report that filesystem IO issue.So:

* How to avoid pending job will be cancelled for slurm

* What caused the slurmctld reported error

Thanks!