[slurm-users] Kernel keyrings on Slurm node inside Slurm job

[Matthias Leopold]

Hi,

I want to access the kernel "user" keyrings inside a Slurm job on a
Ubuntu 20.04 node. I'm not an expert on keyrings (yet), I just
discovered that inside a Slurm job a keyring for "user: invocation_id"
is used, which seems to be shared across all users of the executing
Slurm node (other users can access/destroy my keys).

The structure in a session run from Slurm looks like this (when using
cifscreds):

Session Keyring

989278347 --alswrv 0 0 keyring: _ses

446567140 ----s-rv 0 0 \_ user: invocation_id

638050420 ----sw-v 35816 10513 \_ logon: cifs:d:itsc-test2


The structure in a SSH session looks like this (when using cifscreds):

Session Keyring

932177825 --alswrv 1000 1000 keyring: _ses

826996940 --alswrv 1000 65534 \_ keyring: _uid.1000

1006610690 ----sw-v 1000 1000 \_ logon: cifs:d:itsc-test2


I researched about this invocation_id and found a section on
"KeyringMode=" in systemd.exec man page, but that didn't really help me.

Can you explain to me how it would be possible to get "private" keyrings
inside a Slurm job on the executing node?

thx
Matthias

[Yair Yarom]

Hi,

I think you should look at pam_keyinit and add it to the slurm pam (the one used with the UsePAM configuration).

We currently don't do this, but it's on the todo list to check it out... (so I'm not sure if it will work, or if it's the right way to do this).

--

  /|       |
  \/       | Yair Yarom | System Group (DevOps)
  []       | The Rachel and Selim Benin School
  [] /\    | of Computer Science and Engineering
  []//\\/  | The Hebrew University of Jerusalem
  [//  \\  | T +972-2-5494522 | F +972-2-5494522
  //    \  | ir...@cs.huji.ac.il
 //        |

[Matthias Leopold]

Thanks for the hint. I wasn't aware of UsePAM. At first it looks
tempting, but then I read some bug reports and saw that it's an
"alternative way of enforcing resource limits" and is considered an
"older deprecated functionality".

https://bugs.schedmd.com/show_bug.cgi?id=4098

That doesn't sound too good.

I noticed that I can get a session keyring in an interactive job when I
run "srun --pty keyctl session". That works for my tasks (putting cifs
credentials there), but now I have to find out how to use this in batch
jobs.

Matthias

Am 24.08.22 um 10:43 schrieb Yair Yarom:

> Hi,
>
> I think you should look at pam_keyinit and add it to the slurm pam (the
> one used with the UsePAM configuration).
> We currently don't do this, but it's on the todo list to check it out...
> (so I'm not sure if it will work, or if it's the right way to do this).
>
>
> On Tue, 23 Aug 2022 at 16:36, Matthias Leopold
> <matthias...@meduniwien.ac.at

> // \ |ir...@cs.huji.ac.il <mailto:ir...@cs.huji.ac.il>
> // |
>

--
Matthias Leopold
IT Systems & Communications
Medizinische Universität Wien
Spitalgasse 23 / BT 88 / Ebene 00
A-1090 Wien
Tel: +43 1 40160-21241
Fax: +43 1 40160-921200

[Ole Holm Nielsen]

On 8/25/22 11:15, Matthias Leopold wrote:
> Thanks for the hint. I wasn't aware of UsePAM. At first it looks tempting,
> but then I read some bug reports and saw that it's an "alternative way of
> enforcing resource limits" and is considered an "older deprecated
> functionality".
>
> https://bugs.schedmd.com/show_bug.cgi?id=4098

Warning: Do NOT configure UsePAM=1 in slurm.conf (this advice can be found
on the net). See
https://wiki.fysik.dtu.dk/niflheim/Slurm_configuration#configure-prologflags

/Ole

[Yair Yarom]

I hope UsePAM won't get deprecated. I can understand the dangers, and indeed to use it for limits seems weird (nowadays), but it's a nice hook to have and we use it for other purposes: pam_setquota for /tmp quota per user; Setting the per user /run/user/ directory (usually systemd sets this up, but systemd doesn't play nicely with slurm); Fixing some cgroup mess we have in our system; And calling pam_loginuid.

For a different solution - maybe calling keyctl in a TaskProlog can solve this issue.

--

  /|       |
  \/       | Yair Yarom | System Group (DevOps)
  []       | The Rachel and Selim Benin School
  [] /\    | of Computer Science and Engineering
  []//\\/  | The Hebrew University of Jerusalem
  [//  \\  | T +972-2-5494522 | F +972-2-5494522
  //    \  | ir...@cs.huji.ac.il

 //        |