[slurm-users] scrun: Failed to run the container due to GID mapping configuration

[Toshiki Sonoda (Fujitsu) via slurm-users]

Dear All,

We set up scrun (slurm 23.11.5) integrated with rootless podman, referring to the official documentation.

https://slurm.schedmd.com/containers.html#podman-scrun

https://slurm.schedmd.com/scrun.html#SECTION_Example-%3CB%3Escrun.lua%3C/B%3E-scripts

However, runc/crun prints the error message and fails to run the container.

Looking at the error message from runc, it seems that there is a problem with the GID mapping configuration.

- runc

  [hoge@slm-master hoge]$ podman run centos echo "Hello World"

  time="2024-04-03T02:39:36Z" level=error msg="runc run failed: cannot specify gid=5 mount option for rootless container: User namespaces enabled, but no group mapping found."

  srun: error: slm-worker: task 0: Exited with exit code 1

- crun

  [hoge@slm-master hoge]$ podman run centos echo "Hello World"

  srun: error: slm-worker: task 0: Exited with exit code 1

  mount `devpts` to `dev/pts`: Invalid argument

When we removed the process of linux.gidMappings configuration from scrun.lua,

runc failed to run the container due to the different error, but crun succeeded.

- scrun.lua (The process of linux.gidMappings configuration)

... snip ...

  -- Provide default group map as root if one not provided

  -- mappings fail with build???

  if (true or config["linux"]["gidMappings"] == nil)

  then

      config["linux"]["gidMappings"] =

              {{containerID=process_group_id, hostID=math.floor(group_id), size=1}}

  end

... snip ...

- runc

  [hoge@slm-master hoge]$ podman run centos echo "Hello World"

  time="2024-04-03T02:37:18Z" level=error msg="runc run failed: User namespaces enabled, but no gid mappings found."

  srun: error: slm-worker: task 0: Exited with exit code 1

- crun

  [hoge@slm-master hoge]$ podman run centos echo "Hello World"

  Hello World

Therefore, we would like to know the following two questions.

  - The reason for the failure to run the container

  - Required or recommended configuration to run the container

Best Regards,

Toshiki Sonoda

[Markus Kötter via slurm-users]

Hi,

On 04.04.24 04:46, Toshiki Sonoda (Fujitsu) via slurm-users wrote:
> We set up scrun (slurm 23.11.5) integrated with rootless podman,

I'd recommend looking into nvidia enroot instead.

> https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf




MfG
--
Markus Kötter, +49 681 870832434
30159 Hannover, Lange Laube 6
Helmholtz Center for Information Security