[slurm-users] MIG and eBPF issues (Slurm 24.11.6)

[wk5ng--- via slurm-users]

Hi all,

I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly.

What I'm seeing in slurmd logs:
[2025-11-24T23:32:50.197] [331.interactive] cgroup/v2: cgroup_p_constrain_apply: CGROUP: EBPF Closing and loading bpf program into /sys/fs/cgroup/system.slice/slurmstepd.scope/job_331
[2025-11-24T23:32:50.197] [331.interactive] error: load_ebpf_prog: BPF load error (Operation not permitted). Please check your system limits (MEMLOCK).

I've tried increasing the system limits for MEMLOCK by setting DefaultLimitMEMLOCK=infinity in /etc/systemd/system.conf, and I've copied my slurmd.service file below where I've set Delegate=yes and LimitMEMLOCK=infinity. Previously only Delegate=yes wasn't set (I had rifled through the cgroupv2 documentaton for Slurm and found that setting), but in both cases I see the same BPF load error.

Just wondering if this was something that other people had come across before and maybe I'm doing something silly here. I've checked that my slurm.conf has the corresponding parameters set according to Slurm's own documentation for cgroup.conf and my cgroup.conf is also copied below.

Some portion of the gres.conf is also copied below, and even though I tried AutoDetect=nvml for this node, it's still doesn't work, which was why I changed to manually setting it based off the output of slurmd -G.

Maybe I should try switching back to cgroupv1 and see if that helps fix things, but I'm not sure at this point if MIG and Slurm are compatible using cgroupv1.

I can send other parts of logs, configuration files etc. Any help would be greatly appreciated!

###### slurmd.service file
[Unit]
Description=Slurm node daemon
After=network.target munge.service
ConditionPathExists=/etc/slurm/slurm.conf

[Service]
Type=forking
EnvironmentFile=-/etc/sysconfig/slurmd
ExecStart=/usr/sbin/slurmd -d /usr/sbin/slurmstepd $SLURMD_OPTIONS
ExecReload=/bin/kill -HUP $MAINPID
PIDFile=/var/run/slurmd.pid
KillMode=process
LimitNOFILE=51200
Delegate=yes
LimitMEMLOCK=infinity
LimitSTACK=infinity

[Install]
WantedBy=multi-user.target

###### cgroup.conf
CgroupPlugin=autodetect
ConstrainCores=yes
ConstrainDevices=yes
ConstrainRAMSpace=yes

##### gres.conf
NodeName=gpu-3 AutoDetect=nvml Name=gpu
NodeName=gpu-4 Name=gpu MultipleFiles=/dev/nvidia0,/dev/nvidia-caps/nvidia-cap30,/dev/nvidia-caps/nvidia-cap31

--
slurm-users mailing list -- slurm...@lists.schedmd.com
To unsubscribe send an email to slurm-us...@lists.schedmd.com

[Brian Andrus via slurm-users]

How are you running the lxd container? Have you setup the device as a
passthru to the container?

I have not run lxd containers under slurm (I use apptainer/podman) but I
have used lxd VMs as nodes with no issues, even using GPUs/IB cards in them.

Brian Andrus

[wk5ng--- via slurm-users]

Yes we have setup the NVIDIA GPUs as passthrough to the container based on LXD's documentation:
https://documentation.ubuntu.com/lxd/latest/reference/devices_gpu/
We also have nvidia.runtime set to True to pass the host NVIDIA and CUDA runtime libraries into the container:
https://documentation.ubuntu.com/lxd/latest/reference/instance_options/#nvidia-and-cuda-configuration

With lxd VMs, have you had experience with MIG in the VMs? For example, it explicitly says that the MIG GPU type is not supported for VMs in LXD documentation, but at the same time, maybe I just need to pass in the GPUs, then enable MIG within the lxd VM.

[Christopher Samuel via slurm-users]

On 11/26/25 11:32 am, wk5ng--- via slurm-users wrote:

> I'm having some trouble getting Slurm 24.11.6 to work with MIG, and the slurmd logs seem to point to an issue with eBPF. For some context, this is an LXD unprivileged container where I'm trying to get MIG to work with Slurm. Other compute nodes without MIG work fine and isolate the GPUs accordingly.

To clarify, are you running the slurmd inside the container, or is this
outside of it?

Have you tried running other eBPF programs inside the container?

For cgroups v2 Slurm uses eBPF as v2 removed the ability to constrain
devices.

All the best,
Chris
--
Chris Samuel : http://www.csamuel.org/ : Philadelphia, PA, USA

[wk5ng--- via slurm-users]

Slurmd is running inside the container, and no I've not tried running other eBPF programs inside the container - is there something that you would recommend trying just to see if eBPF is functional?

One thing that I've tried is just running:
bpftool prog show
This works on the bare metal server running the LXD containers but within the containers, I had to install linux-tools-generic and linux-tools-common, then running the same command shows:
root@gpu-4:~# bpftool prog show
Error: can't get next program: Operation not permitted

I think this possibly points to a limitation of unprivileged containers (or more likely, a misconfiguration of the container on my end due to my ignorance). For example, we added the GPUs and MIGs using lxc config device add, but maybe we also need to add lines specifying lxc.cgroup2.devices.allow based on https://linuxcontainers.org/lxc/manpages/man5/lxc.container.conf.5.html. Also possibly, I should look into privilege escalation (?) https://documentation.ubuntu.com/lxd/latest/explanation/bpf/ but I don't quite understand this yet.