[slurm-users] External Authentication Integration with JWKS and RS256 Tokens

[Laurence]

Hi,

I am trying to configure SLURM to use external authentication for JWT as described in the documentation.

https://slurm.schedmd.com/jwt.html

JWT Authentication worked when I tested the setup for standalone use but am having difficulty with tokens from our oauth provider.

My first question is has anyone successfully done this? My second question is on the example code to verify the jwt key. Is the example up to date as it doesn't work for me. The final question is does anyone have any suggestions on the concrete error reported in the slurmctld log.

slurmctld: error: failed to verify jwt, rc=22
slurmctld: error: could not find matching kid or decode failed

Thanks,

Laurence

[Ümit Seren]

If you use AzureAD as your identity provider beware that their JWKS json doesn't contain the alg parameter.

We opened an issue: https://bugs.schedmd.com/show_bug.cgi?id=16168 and it is confirmed.

As a workaround you can use this jq query to add the alg to the jwks json that you get from AzureAD: 

curl -s https://login.microsoftonline.com/TENANT/discovery/v2.0/keys | jq '.keys |= map(.alg="RS256")' > $TMPFILE

Hope this helps

Best

Ümit

[Laurence Field]

Hi Ümit,

Thanks for your reply. We are using Keycloak and the JWKS does contain this parameter. I will continue to debug but any suggestions would be greatly appreciated.

Cheers,

Laurence

[Laurence Field]

Hi,

After verifying the JWT and JWKS with some Python code, it magically seems to work. At least the error has changed to auth_p_verify: jwt_get_grant failure. This suggests I need to update something in the authorization policy. Will do that now but if anyone has done this before and can give me some hints, they would be most welcome.

Cheers,

Laurence

[Ümit Seren]

Looks like you are missing the username field in the JWT token: https://github.com/SchedMD/slurm/blob/slurm-22-05-8-1/src/plugins/auth/jwt/auth_jwt.c#L419

You have to make sure that your JWT token contains the SLURM username as an attribute (https://slurm.schedmd.com/jwt.html#compatibility). 

[Laurence Field]

Hi Ümit,

Thanks for the reply. Yes, it looks like this is the issue. Although from the master branch it suggests that the claim_field can also be used but this is not in the version we have deployed.

Cheers,

Laurence

[Laurence]

Coming back to this, it is failing again and I don't know why.

slurmctld: error: failed to verify jwt, rc=22
slurmctld: error: could not find matching kid or decode failed

The kids seem to match and python code I have verifies the jwt with the jwks. Does anyone have any ideas on what the issue might be? The jwks can be found at the following URL.

https://auth.cern.ch/auth/realms/cern/protocol/openid-connect/certs

Cheers,

Laurence

[Ümit Seren]

I would suggest du increase the log verbosity of slurmrest and see if there is more information in the log file