[slurm-users] "fatal: can't stat gres.conf"

[Alex Chekholko]

Hi all,

I have a few working GPU compute nodes.  I bought a couple of more identical nodes.  They are all diskless; so they all boot from the same disk image.

For some reason slurmd refuses to start on the new nodes; and I'm not able to find any differences in hardware or software.  Google searches for "error: Waiting for gres.conf file " or "fatal: can't stat gres.conf file" are not helping.

The gres.conf file is there and identical on all nodes. The /dev/nvidia[0-3] files are there and 'nvidia-smi -L' works fine.  What am I missing?

[root@n0038 ~]# slurmd -Dcvvv

slurmd: debug2: hwloc_topology_init

slurmd: debug2: hwloc_topology_load

slurmd: debug:  CPUs:20 Boards:1 Sockets:2 CoresPerSocket:10 ThreadsPerCore:1

slurmd: Node configuration differs from hardware: CPUs=16:20(hw) Boards=1:1(hw) SocketsPerBoard=16:2(hw) CoresPerSocket=1:10(hw) ThreadsPerCore=1:1(hw)

slurmd: Message aggregation disabled

slurmd: debug:  init: Gres GPU plugin loaded

slurmd: error: Waiting for gres.conf file /dev/nvidia[0-1],CPUs="0-9"

slurmd: fatal: can't stat gres.conf file /dev/nvidia[0-1],CPUs="0-9": No such file or directory

SLURM version ohpc-17.02.7-61

[Bill]

Hi Alex,

Try run nvidia-smi before start slurmd, I also found this issue. I have to run nvidia-smi before slurmd when I reboot system.

Regards,

Bill

------------------ Original ------------------

From: Alex Chekholko <al...@calicolabs.com>

Date: Tue,Jul 24,2018 6:10 AM

To: Slurm User Community List <slurm...@lists.schedmd.com>

Subject: Re: [slurm-users] "fatal: can't stat gres.conf"

[Alex Chekholko]

Thanks for the suggestion; if my memory serves me right, I had to do that previously to cause the drivers to load correctly after boot.

However, in this case both 'nvidia-smi' and 'nvidia-smi -L' run just fine and produce expected output.

One thing I see that is different is my older nodes have these two "uvm" devices:

working:

[root@n0035 ~]# ls -alhtr /dev/nvidia*

crw-rw-rw- 1 root root 195, 255 Nov  6  2017 /dev/nvidiactl

crw-rw-rw- 1 root root 195,   0 Nov  6  2017 /dev/nvidia0

crw-rw-rw- 1 root root 195,   1 Nov  6  2017 /dev/nvidia1

crw-rw-rw- 1 root root 195,   2 Nov  6  2017 /dev/nvidia2

crw-rw-rw- 1 root root 195,   3 Nov  6  2017 /dev/nvidia3

crw-rw-rw- 1 root root 241,   1 Nov  7  2017 /dev/nvidia-uvm-tools

crw-rw-rw- 1 root root 241,   0 Nov  7  2017 /dev/nvidia-uvm

not working:

[root@n0039 ~]# ls -alhtr /dev/nvidia*

crw-rw-rw- 1 root root 195, 255 Jul 12 17:09 /dev/nvidiactl

crw-rw-rw- 1 root root 195,   0 Jul 12 17:09 /dev/nvidia0

crw-rw-rw- 1 root root 195,   1 Jul 12 17:09 /dev/nvidia1

crw-rw-rw- 1 root root 195,   2 Jul 12 17:09 /dev/nvidia2

crw-rw-rw- 1 root root 195,   3 Jul 12 17:09 /dev/nvidia3

[Nicholas McCollum]

You may want to check and make sure your GPUs are in persistance mode.  You can enable it through the nvidia-smi utility.

Nicholas McCollum
Alabama Supercomputer Authority

---

From: Alex Chekholko <al...@calicolabs.com>
Sent: Monday, July 23, 2018 6:00 PM

To: Slurm User Community List

[Ian Mortimer]

On Mon, 2018-07-23 at 15:59 -0700, Alex Chekholko wrote:

> However, in this case both 'nvidia-smi' and 'nvidia-smi -L' run just
> fine and produce expected output.

They will because running nvidia-smi triggers loading of the kernel
module and creation of the device files. But are the device files
there when slurmd starts? What happens if you start slurmd after
running nvidia-smi?


--
Ian

[Sean Crosby]

Hi Alex,

What's the actual content of your gres.conf file? Seems to me that you have a trailing comma after the location of the nvidia device

Our gres.conf has

NodeName=gpuhost[001-077] Name=gpu Type=p100 File=/dev/nvidia0 Cores=0,2,4,6,8,10,12,14,16,18,20,22

NodeName=gpuhost[001-077] Name=gpu Type=p100 File=/dev/nvidia1 Cores=0,2,4,6,8,10,12,14,16,18,20,22

NodeName=gpuhost[001-077] Name=gpu Type=p100 File=/dev/nvidia2 Cores=1,3,5,7,9,11,13,15,17,19,21,23

NodeName=gpuhost[001-077] Name=gpu Type=p100 File=/dev/nvidia3 Cores=1,3,5,7,9,11,13,15,17,19,21,23

I think you have a comma between the File and Cores/CPUs

Sean

[Ryan Novosielski]

Best off running nvidia-persistenced. Handles all of this stuff as a side effect, and also enables persistence mode, provided you don’t configure it otherwise. 

--
____
|| \\UTGERS,       |---------------------------*O*---------------------------
||_// the State     |         Ryan Novosielski - novo...@rutgers.edu
|| \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS Campus
||  \\    of NJ     | Office of Advanced Research Computing - MSB C630, Newark
    `'

[Ian Mortimer]

On Tue, 2018-07-24 at 02:19 +0000, Ryan Novosielski wrote:

> Best off running nvidia-persistenced. Handles all of this stuff as a
> side effect, and also enables persistence mode, provided you don’t
> configure it otherwise. 

Yes. But you have to ensure it starts before slurmd.


--
Ian

[Ryan Novosielski]

While true, I don’t find I need to take any special precaution on my machines. Probably prudent to set a systemd dependency though.

[Alex Chekholko]

Hello all,

My error was indeed just the comma in my gres.conf.  I was confused because I had the same file on my running nodes but that's just because slurmd started before the erroneous comma was added to the config.

So the error message was in fact directly correct, it could not find the device called "/dev/nvidia[0-1],CPUs="0-9""

I have a separate question.  None of my GPUs are in 'persistence mode'.  But the users have not encountered any problems.  Reading through the docs, it looks like it may have some minor effect on startup times.  Most of our GPU jobs are long (many hours, sometimes days).  Do people tend to use "persistence mode" for their GPU compute nodes?

Regards,

Alex