[slurm-users] slurmrestd configuration

[Simone Riggi]

Dear all,

I write you about how to properly setup slurmrestd. 

I followed a recent post (https://groups.google.com/g/slurm-users/c/VDeFmSDao1U/m/Yu0j-VNcBgAJ) but slurmrestd fails with:

slurmrestd: fatal: No authentication plugins to load

Indeed when I run slurmrestd -a list, no plugin is detected. 

Can you provide some additional documentation explaining what authentication plugins I should use, how to install and configure them? I was not able to understand from the current documentation if this works only with jwt or there is a default plugin.

 

I try to describe what I did:

OS: Centos 7

Slurm: v20.11.5

1) Installed the dependencies, among which libjwt. I verified it is installed under /usr/lib64:

rpm -qa | grep jwt
libjwt-1.9.0-1.of.el7.x86_64
libjwt-devel-1.9.0-1.of.el7.x86_64

2) Installed slurm with:

rpmbuild -ta slurm-20.11.5.tar.bz2 --with mysql --with slurmrestd --with jwt

3) Added these options to /etc/slurm/slurm.conf

AuthType=auth/munge
AuthAltTypes=auth/jwt

4) Started daemons. I attach the logs: 

== slurmctld ==

2021-04-07T20:34:10.148] slurmctld version 20.11.5 started on cluster lofar-gpu-clust
[2021-04-07T20:34:10.153] No memory enforcing mechanism configured.
[2021-04-07T20:34:10.219] Recovered state of 1 nodes
[2021-04-07T20:34:10.219] Recovered information about 0 jobs
[2021-04-07T20:34:10.219] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions
[2021-04-07T20:34:10.219] Recovered state of 0 reservations
[2021-04-07T20:34:10.219] read_slurm_conf: backup_controller not specified
[2021-04-07T20:34:10.219] select/cons_tres: select_p_reconfigure: select/cons_tres: reconfigure
[2021-04-07T20:34:10.219] select/cons_tres: part_data_create_array: select/cons_tres: preparing for 1 partitions
[2021-04-07T20:34:10.219] Running as primary controller
[2021-04-07T20:34:10.220] No parameter for mcs plugin, default values set
[2021-04-07T20:34:10.220] mcs: MCSParameters = (null). ondemand set.

== slurmd ==

[2021-04-07T20:35:37.939] slurmd version 20.11.5 started
[2021-04-07T20:35:37.940] slurmd started on Wed, 07 Apr 2021 20:35:37 +0200
[2021-04-07T20:35:37.940] CPUs=40 Boards=1 Sockets=4 Cores=10 Threads=2 Memory=128529 TmpDisk=51175 Uptime=5391270 CPUSpecList=(null) FeaturesAvail=(null) FeaturesActive=(null)

== slurmrestd ==

$ systemctl start slurmrestd

slurmrestd[1987721]: fatal: No authentication plugins to load

Same result with this (as root user) following the old post: 

$ slurmrestd -f /etc/slurm/slurmrestd.conf 0.0.0.0:6820 

where /etc/slurm/slurmrestd.conf

include /etc/slurm/slurm.conf
AuthType=auth/jwt

Any hints?

Thanks very much for your help,

Regards,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Ward Poelmans]

Hi Simone,

On 8/04/2021 09:50, Simone Riggi wrote:

> where /etc/slurm/slurmrestd.conf
>
> include /etc/slurm/slurm.conf
> AuthType=auth/jwt

Did you add a key?

AuthAltParameters=jwt_key=/etc/slurm/jwt.key

It needs to be present on the slurmdbd and slurmctld nodes.

Ward

[Ole Holm Nielsen]

On 4/8/21 9:50 AM, Simone Riggi wrote:
> I write you about how to properly setup slurmrestd.

...

> 2) Installed slurm with:
>
> rpmbuild -ta slurm-20.11.5.tar.bz2 --with mysql --with slurmrestd --with jwt

I don't see this "--with jwt" in the slurm.spec file:

[slurm-20.11.5]# grep "# --with" slurm.spec
# --with cray %_with_cray 1 build for a Cray Aries system
# --with cray_network %_with_cray_network 1 build for a non-Cray system
with a Cray network
# --with cray_shasta %_with_cray_shasta 1 build for a Cray Shasta system
# --with slurmrestd %_with_slurmrestd 1 build slurmrestd
# --with slurmsmwd %_with_slurmsmwd 1 build slurmsmwd
# --without debug %_without_debug 1 don't compile with debugging symbols
# --with hdf5 %_with_hdf5 path require hdf5 support
# --with hwloc %_with_hwloc 1 require hwloc support
# --with lua %_with_lua path build Slurm lua bindings
# --with mysql %_with_mysql 1 require mysql/mariadb support
# --with numa %_with_numa 1 require NUMA support
# --without pam %_without_pam 1 don't require pam-devel RPM to be installed
# --without x11 %_without_x11 1 disable internal X11 support
# --with ucx %_with_ucx path require ucx support
# --with pmix %_with_pmix path require pmix support

/Ole

[Ward Poelmans]

Hi Ole,

On 8/04/2021 10:09, Ole Holm Nielsen wrote:
> On 4/8/21 9:50 AM, Simone Riggi wrote:
>>
>> rpmbuild -ta slurm-20.11.5.tar.bz2 --with mysql --with slurmrestd
>> --with jwt
>
> I don't see this "--with jwt" in the slurm.spec file:

It's not yet there: https://bugs.schedmd.com/show_bug.cgi?id=11114

But when libjwt is installed, the configure script will pick it up.

Ward

[Simone Riggi]

Thanks very much for your reply. 

- I added this line to the slurm.conf file:

AuthAltParameters=jwt_key=/var/spool/slurmctld/jwt_hs256.key

I restarted the daemons but this does not change slurmrestd error. I guess this is needed anyway after plugins are found, so thanks for pointing that out.

- I see effectively that --with jwt is not listed. I wonder how to build (using rpmbuild) slurm auth plugins? 

  In general I didn't understand from the doc what plugins slurmrestd expects by default and where it searches it. From -a option it seems it searches libraries inside a directory rest_auth/[PLUGIN_LIB_NAME]. But it seems these plugins are not built by rpmbuild or are not installed or are installed in a non-standard directory. 

The rpmbuild log is available here (https://drive.google.com/file/d/1jrm55DH50Sr3EX6WZXj-x0z5ArPETgSQ/view?usp=sharing), maybe you can understand if there is something wrong.   

Thanks a lot again for your help,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Ward Poelmans]

Hi Simone,

On 8/04/2021 15:53, Simone Riggi wrote:

> - I see effectively that --with jwt is not listed. I wonder how to build
> (using rpmbuild) slurm auth plugins? 
>   In general I didn't understand from the doc what plugins slurmrestd
> expects by default and where it searches it. From -a option it seems it
> searches libraries inside a directory rest_auth/[PLUGIN_LIB_NAME]. But
> it seems these plugins are not built by rpmbuild or are not installed or
> are installed in a non-standard directory. 

I see 'slurmrestd -a list' do stat on all libraries under
/usr/lib64/slurm and it opens two files:
/usr/lib64/slurm/rest_auth_local.so
/usr/lib64/slurm/rest_auth_jwt.so

Both of these come from the slurm RPM in my case
(slurm-20.11.5-1.el7.x86_64)

Ward

[Simone Riggi]

Thanks very much. 

I found both files under: 

/root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/auth/jwt/.libs/rest_auth_jwt.so

/root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/auth/local/.libs/rest_auth_local.so

It seems they weren't installed (any hints why?). By the way I tried to copy both in the /usr/lib64 and now slurmrestd returns:

$ slurmrestd -a list

slurmrestd: Possible REST authentication plugins:
slurmrestd: rest_auth/local
slurmrestd: rest_auth/jwt

However when I try to run it (slurmrestd -a rest_auth/local or slurmrestd -a rest_auth/local) I get: 

slurmrestd: fatal: No OAS plugins to load. Nothing to do

Some other plugins to be built and installed? 

Thanks again,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Simone Riggi]

Ok,

I think I have found the plugins in:

/root/rpmbuild/BUILD/slurm-20.11.5/src/slurmrestd/plugins/openapi/

I did:

make

make install

and now I have:

$ slurmrestd -s list

slurmrestd: Possible OpenAPI plugins:
slurmrestd: openapi/v0.0.35
slurmrestd: openapi/v0.0.36
slurmrestd: openapi/dbv0.0.36

Last one is not working:

$ slurmrestd -f /etc/slurm/slurmrestd.conf 0.0.0.0:6820 -a jwt -s openapi/dbv0.0.36

slurmrestd: fatal: slurm_openapi_p_init: slurm not configured with slurmdbd

The second one is apparently running but I do not see any logs:

$ slurmrestd -f /etc/slurm/slurmrestd.conf 0.0.0.0:6820 -a jwt -s openapi/v0.0.36

Besides I have no idea at the moment of what plugins to choose and what they do. 

How can I test (e.g. an endpoint with curl) if the rest API is working including authentication? 

Cheers,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Simone Riggi]

Hi again,

I think I managed to make it working with: 

$ slurmrestd -f /etc/slurm/slurmrestd.conf -a rest_auth/jwt -s openapi/v0.0.36 -vvv 0.0.0.0:6820

Sample request:

$ curl -H "X-SLURM-USER-NAME:riggi" -H "X-SLURM-USER-TOKEN:${SLURM_JWT}" http://localhost:6820/slurm/v0.0.36/ping

{
   "meta": {
     "plugin": {
       "type": "openapi\/v0.0.36",
       "name": "REST v0.0.36"
     },
     "Slurm": {
       "version": {
         "major": 20,
         "micro": 5,
         "minor": 11
       },
       "release": "20.11.5"
     }
   },
   "errors": [
   ],
   "pings": [
     {
       "hostname": "XXXX",
       "ping": "UP",
       "status": 0,
       "mode": "primary"
     }
   ]

where SLURM_JWT was generated with:

$ scontrol token lifespan=7200 username=riggi

How can I configure and test the other auth method (local)? I am using jwt at the moment.

I would like a user to be always authorized to use the rest API. 

Thanks,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Ward Poelmans]

Hi Simone,

On 8/04/2021 23:23, Simone Riggi wrote:
> $ scontrol token lifespan=7200 username=riggi
>
> How can I configure and test the other auth method (local)? I am using
> jwt at the moment.
> I would like a user to be always authorized to use the rest API. 

local means socket (so you don't need a token). Start it with:
slurmrestd unix:/var/run/slurmrestd.socket

Ward

[Simone Riggi]

Thanks Ward. 

I did the following:

$ slurmrestd unix:/var/run/slurmrestd.socket -a rest_auth/local -s v0.0.36 -vvv

and to test the response:

$ curl --unix-socket /var/run/slurmrestd.socket localhost/slurm/v0.0.36/ping     (here "localhost" can be whatever character, need to check --unix-socket curl option!)

$ echo -e "GET /slurm/v0.0.36/ping HTTP/1.1\r\n" | socat unix-connect:/var/run/slurmrestd.socket STDIO

$ echo -e "GET /slurm/v0.0.36/ping HTTP/1.0\r\n" | nc -U /var/run/slurmrestd.socket

All of them are working.

So in this case the only requirement for a user is having the read/write permission on the socket? 

My goal at the end would be to let a Dockerized client application communicate with Slurmrestd server (running on the host). Guess I need to mount the socket in the container but haven't tried yet.    

Thanks,

Simone

****************************************************************

Simone Riggi, PhD

INAF, Osservatorio Astrofisico di Catania
Via S. Sofia 78
95123, Catania - Italy
phone:  +39 095 7332 extension 282 (or 310)
e-mail: simone...@gmail.com,

            sri...@inaf.it,
            sri...@pec.it
skype: simone.riggi
****************************************************************

[Ward Poelmans]

Hi Simone,

On 9/04/2021 18:03, Simone Riggi wrote:

> All of them are working.
> So in this case the only requirement for a user is having the read/write
> permission on the socket? 

Correct. The authentication is done as you know the user with a socket.

> My goal at the end would be to let a Dockerized client application
> communicate with Slurmrestd server (running on the host). Guess I need
> to mount the socket in the container but haven't tried yet.    

I guess so. No experience with that. But keep in mind that inside the
container you have a different user namespace so you need to make sure
the uid/gid match.

Ward