[slurm-users] Slurmrestd JWT authentication [SEC=UNOFFICIAL]

733 views

Skip to first unread message

Justin Freeman

unread,

Mar 7, 2022, 6:05:07 PM3/7/22

to slurm...@lists.schedmd.com

Hi slurm-users,

Hoping you can point me in the right direction….I am trying to setup the Slurm REST API under parallelcluster v3.1.1. I am following the instructions at https://slurm.schedmd.com/jwt.html. I have added the AuthAltTypes=rest_auth/jwt and the AuthAultParameters=jwt_key=… to /opt/slum/etc/slurm.conf but restarting slurmctld fails with:

slurmctld: debug: slurmctld log levels: stderr=debug3 logfile=debug3 syslog=quiet

slurmctld: debug: Log file re-opened

slurmctld: pidfile not locked, assuming no running daemon

slurmctld: Stack size set to 10485760

slurmctld: debug: slurmscriptd: Got ack from slurmctld, initialization successful

slurmctld: debug: slurmctld: slurmscriptd fork()'d and initialized.

slurmctld: slurmctld version 21.08.5 started on cluster parallelcluster

slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/cred_munge.so

slurmctld: debug: _slurmscriptd_mainloop: started

slurmctld: debug3: Called _msg_readable

slurmctld: debug: _slurmctld_listener_thread: started listening to slurmscriptd

slurmctld: cred/munge: init: Munge credential signature plugin loaded

slurmctld: debug3: Success.

slurmctld: debug3: Called _msg_readable

slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/auth_munge.so

slurmctld: debug: auth/munge: init: Munge authentication plugin loaded

slurmctld: debug3: Success.

slurmctld: debug3: Trying to load plugin /opt/slurm/lib/slurm/rest_auth_jwt.so

slurmctld: debug3: Couldn't find sym 'auth_p_create' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_destroy' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_verify' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_get_uid' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_get_gid' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_get_host' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_pack' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_unpack' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_thread_config' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_thread_clear' in the plugin

slurmctld: debug3: Couldn't find sym 'auth_p_token_generate' in the plugin

slurmctld: fatal: plugin_load_and_link: Plugin loading failed due to missing symbols. Plugin is corrupted.

Looks like I am missing something when trying to enable JWT authentication with the slurm services. I am using the slurm install that comes out of the box with parallelcluster 3.1.1. I was wondering if there are any suggestions about how to fix/resolve this error?. Thanks.

Slurmrestd auth types:

$ slurmrestd -a list

slurmrestd: Possible REST authentication plugins:

slurmrestd: rest_auth/local

slurmrestd: rest_auth/jwt

Also, step 2 in the JWT instructions at https://slurm.schedmd.com/jwt.html is a little confusing – the key is saved to the StateSaveLocation but the rest of the instructions reference /etc/slurm? Am I missing something here (likely!) ?

dd if=/dev/random of=/var/spool/slurm/statesave/jwt_hs256.key bs=32 count=1

chown slurm:slurm /etc/slurm/jwt_hs256.key

chmod 0600 /etc/slurm/jwt_hs256.key

chown root:root /etc/slurm

chmod 0755 /etc/slurm

Thanks!

Regards,

Brian Andrus

unread,

Mar 7, 2022, 10:15:52 PM3/7/22

to slurm...@lists.schedmd.com

Double-check you have all the packages.

When slurm is built, slurmrestd is a separate package and is only built if the whole set was directed to do so. If they did not build it, you will need to do so yourself. This will mean using your custom built files throughout.