[slurm-users] slurmrestd unable to find plugin rest_auth/jwt

[Jon Marshall via slurm-users]

Hi,

I feel like I must have missed something obvious here, so hopefully someone can spot it! We recently updated to Slurm 24.11.6 on our cluster and I made sure to compile --with slurmrestd and --with jwt, as there are a few things I've been meaning to try out with the REST API.

>From the rpmbuild config.log I get:

  $ ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc/slurm --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-systemdsystemunitdir=/usr/lib/systemd/system --enable-pkgconfig --with-jwt

--

configure:21020: checking for jwt.h

configure:21020: gcc -c -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer  conftest.c >&5

configure:21020: $? = 0

configure:21020: result: yes

configure:21028: checking for jwt_add_header in -ljwt

configure:21051: gcc -o conftest -DNUMA_VERSION1_COMPATIBILITY -O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -Wl,-z,lazy -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer  -Wl,-z,relro  -Wl,-z,lazy conftest.c -ljwt  -ldl -lrt -lm -lresolv  >&5

configure:21051: $? = 0

configure:21061: result: yes

--

config.status:1746: creating src/plugins/auth/jwt/Makefile

--

config.status:1746: creating src/slurmrestd/plugins/auth/jwt/Makefile

--

ac_cv_header_jwt_h=yes

--

ac_cv_lib_jwt_jwt_add_header=yes

--

JWT_LDFLAGS='-ljwt'

And:

configure:26873: checking whether to compile slurmrestd

configure:26904: result: yes

--

configure:26915: checking for slurmrestd default port

configure:26933: result: 6820

And indeed I get a slurmrestd package out of it, which I've successfully installed with no errors. I've created a JWT key and rolled it out to the controller node, but when I go to check auth types I get:

slurmrestd -a rest_auth/jwt

slurmrestd: fatal: Unable to find plugin: rest_auth/jwt

Running an strace on this process shows that it does in fact stat the .so that does in fact exist:

stat("/usr/lib64/slurm/auth_jwt.so", {st_mode=S_IFREG|0755, st_size=283728, ...}) = 0

And I'm kind of stumped now. I've checked through all the resolved bug tickets to see if I can spot what is wrong but I'm getting nothing, and I can't find anything in the mailing lists either, other than things I've checked. Has anyone else had similar issues, or can someone see the blindingly obvious thing I'm missing?

Cheers

Jon

Jon Marshall

High Performance Computing Specialist

 

IT and Scientific Computing Team

 

Cancer Research UK Cambridge Institute

Li Ka Shing Centre | Robinson Way | Cambridge | CB2 0RE

Web | Facebook | Twitter

 

[Ole Holm Nielsen via slurm-users]

Hi Jon,

Did you check the software prerequisite minimum versions listed in this
page: https://slurm.schedmd.com/rest_quickstart.html#prereq

IHTH,
Ole

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Jon Marshall via slurm-users]

Hi Ole,

Sorry, missed your reply to this - I did indeed and the machine I am building the packages on definitely has jwt-devel packages installed, as noted in the rpmbuild config.

Basically it looks like it is getting built, and when the program runs it looks up auth_jwt.so which definitely exists, but then it craps out saying it can't find it.

Cheers

Jon

---

From: Ole Holm Nielsen via slurm-users <slurm...@lists.schedmd.com>
Sent: 14 August 2025 06:57
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>
Subject: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt

 

Hi Jon,

Did you check the software prerequisite minimum versions listed in this

page: https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fslurm.schedmd.com%2frest_quickstart.html%23prereq&c=E,1,vGDIY8GprI0f-1TQWtwOkaK69kYmK8KVm0ufe88JHRyla7Bow83wWpslJHkR079oQUKb7TVLX9sUcl3ojNLC0Om_3pyDuLw0CybskwWuDzo1&typo=1

> stat("/usr/lib64/slurm/https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fauth_jwt.so&c=E,1,pzCV_3-iTZai02HQXL4dMosthihwRaYiqdXWywyaBQ04wuqxYDySRp3UVZedug3FJ93qz-ObhCNbuUYk7Yno9bqH3sec4RzUX6HW5VTing,,&typo=1", {st_mode=S_IFREG|0755,

[Jon Marshall via slurm-users]

Further to this, it appears that I'm getting confused here - there is an auth_jwt.so, but I actually want rest_auth_jwt.so which is not getting built via rpmbuild. If I manually compile and install with --enable-slurmrestd and --with_jwt=/usr, then the required so is generated and installed.

I will continue to investigate the build logs for rpmbuild to see if I can figure out where it is going wrong.

Cheers

Jon

---

From: Jon Marshall via slurm-users <slurm...@lists.schedmd.com>
Sent: 27 August 2025 12:32
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>; Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>

[Jon Marshall via slurm-users]

For anyone interested, I managed to resolve this by running:

rpmbuild -ta slurm-24.11.6.tar.bz2 --with slurmrestd  --define "_with_jwt --with-jwt=/usr"

This has now successfully built the rest_auth_jwt.so and I've got the daemon running.

Cheers

Jon

---

From: Jon Marshall <Jon.Ma...@cruk.cam.ac.uk>
Sent: 27 August 2025 14:16
To: slurm...@lists.schedmd.com <slurm...@lists.schedmd.com>; Ole Holm Nielsen <Ole.H....@fysik.dtu.dk>; Jon Marshall <Jon.Ma...@cruk.cam.ac.uk>
Subject: Re: [slurm-users] Re: slurmrestd unable to find plugin rest_auth/jwt