[slurm-users] Slurmrestd authentication failed: Unspecified error

[Chenyang Yan]

Hello,

I have met a similar issue with slurmrestd authentication failed error, similar question: https://lists.schedmd.com/pipermail/slurm-users/2021-June/007480.html

I have installed `slurm 21.08.6` on CentOS 7.9.2009 container, basic service is running fine
```
[root@slurmctl supervisor]# ls -l /.dockerenv
-rwxr-xr-x. 1 root root 0 Mar 17 23:17 /.dockerenv
[root@slurmctl supervisor]# srun --partition normal hostname
slurmctl
```

Slurmrestd is compiled with `--enable-slurmrestd` successfully, slurmrestd JWT configuration is as follows:
```
[root@slurmctl slurmctld]# dd if=/dev/random of=/var/spool/slurm/jwt_hs256.key bs=32 count=1
[root@slurmctl slurmctld]# scontrol show config |grep -i auth
AuthAltTypes            = auth/jwt
AuthAltParameters       = jwt_key=/var/spool/slurm/jwt_hs256.key
AuthInfo                = (null)
AuthType                = auth/munge
[root@slurmctl slurmctld]# ll -l /var/spool/slurm/jwt_hs256.key
-rw-r--r--. 1 root root 32 Mar 17 23:21 /var/spool/slurm/jwt_hs256.key

# start slurmrestd process
[root@slurmctl slurmctld]# SLURMRESTD_SECURITY=disable_user_check /usr/sbin/slurmrestd -vvvv 0.0.0.0:19090

# check process and environ
[root@slurmctl slurmctld]# ps -ef |grep slurmrestd
root      1235   236  0 23:26 ?        00:00:00 /usr/sbin/slurmrestd -vvvvv 0.0.0.0:19090

[root@slurmctl slurmctld]# cd /proc/1235/
[root@slurmctl 1235]# cat environ | tr '\0' "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl
SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord
```

I have generated correct token to request, but slurmrestd log message reported authentication failed: Unspecified error
```
[root@slurmctl 1235]# scontrol token username=slurm
SLURM_JWT=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw
[root@slurmctl 1235]# token="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NDc1NjM2MjAsImlhdCI6MTY0NzU2MTgyMCwic3VuIjoic2x1cm0ifQ.151oD4rdm_AuDFUWc24eKaXgTPAQE_v1ugBzzA8ulNw"
[root@slurmctl 1235]# curl 172.17.0.4:19090/openapi -H "X-SLURM-USER-TOKEN: $token" -H "X-SLURM-USER-NAME: slurm"
Authentication failure


slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[172.17.0.1]:38090] attempting user_name slurm token authentication pass through
slurmrestd: error: operations_router: [[172.17.0.1]:38090] authentication failed: Unspecified error
slurmrestd: debug2: _on_message_complete_request: [[172.17.0.1]:38090] on_http_request rejected: Unspecified error
```

But! I have found that I'm setting `SLURM_JWT` environment variable for process, whatever token value is authenticated normally
```
# start process with SLURM_JWT
[root@slurmctl 1235]# cd /proc/2108/
[root@slurmctl 2108]# cat environ |tr "\0" "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl
SLURM_JWT=randomtoken
SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord

# request OK
[root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"
[
 ]
[root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"
[
 ]
```

So, I'm confused about JWT authentication.
Q1: What is used for the `SLURM_JWT` environment variable, is it required for JWT? Related search from github source repo: https://github.com/SchedMD/slurm/search?q=SLURM_JWT

Q2: How to use slurmrestd JWT authentication?

--------
Thanks, Chenyang Yan

[Guillaume COCHARD]

Hello,

We had the same error and we fixed it by adding `Environment="SLURM_JWT=daemon"` to the [Service] section of the unit file (in our case /usr/lib/systemd/system/slurmrestd.service ).

We have a bug (feature?) that makes us unable to use root or slurm user as user for slurmrestd service so maybe you'll encounter that as well. According to the release notes ( https://github.com/SchedMD/slurm/blob/master/NEWS ) this should be starting on version 22.05.0pre1 but it is already the case in our version 21.08.6. We have an administrator user for this can of usage.

Our service file is like this:

$ cat /usr/lib/systemd/system/slurmrestd.service
[Unit]
Description=Slurm REST daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurm.conf

[Service]
User=slurmadm
Type=simple
EnvironmentFile=-/etc/sysconfig/slurmrestd
Environment="SLURM_JWT=daemon"
ExecStart=/usr/sbin/slurmrestd -v -a rest_auth/jwt localhost:6820
ExecReload=/bin/kill -HUP $MAINPID

[Install]
WantedBy=multi-user.target

Guillaume

---

De: "Chenyang Yan" <memory...@gmail.com>
À: slurm...@schedmd.com
Envoyé: Samedi 19 Mars 2022 14:09:06
Objet: [slurm-users] Slurmrestd authentication failed: Unspecified error

[Chenyang Yan]

Guillaume, thanks for your reply and your workaround.

Do you try adding `SLUTM_JWT=daemon` fo slurmrestd process, and then any value is is authenticated successfully for request header `X-SLURM-USER-TOKEN`, `X-SLURM-USER-NAME` .

You can see the information in my last email:

```

# start process with SLURM_JWT
[root@slurmctl 1235]# cd /proc/2108/
[root@slurmctl 2108]# cat environ |tr "\0" "\n"
TERM=xterm
TINI_VERSION=v0.18.0
SHLVL=1
HOSTNAME=slurmctl

SLURM_JWT=randomtoken                   <===== SLURM_JWT environ

SUPERVISOR_ENABLED=1
SUPERVISOR_PROCESS_NAME=slurmrestd
PWD=/
SUPERVISOR_SERVER_URL=unix:///var/run/supervisor/supervisor.sock
SUPERVISOR_GROUP_NAME=slurmrestd
PATH=/root/.pyenv/shims:/root/.pyenv/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin
HOME=/root
SLURMRESTD_SECURITY=disable_user_check
_=/usr/bin/supervisord

# request OK
[root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: slurm"
[
 ]
[root@slurmctl 2108]# curl 172.17.0.4:19090/slurm/v0.0.35/jobs -H "X-SLURM-USER-TOKEN: everythingvalue......" -H "X-SLURM-USER-NAME: errorvalue"
[
 ]

```

So, I'm confused for slurmrestd JWT authentication.

--------------------

Thanks, Chenyang Yan