Singularity container instance with CPU affinity

[C. Sauvanaud]

Dear all,

For testing purpose I need to run a Singularity container with only a subset of cores from my machine. 

For this purpose I chose to use the "taskset" command and to use is as follows:

  taskset --cpu-list 0-6 singularity instance.start /image.img myinstance

In this case I restricted my container instance to use 7 cores (child processes included) and the command "taskset -c -p PID" confirms this well.

However, once I am inside my container instance, I get to run heavy jobs such as "stress-ng --cpu 22 -t 1m" (running a CPU load on 22 cores for 1 minute) and the jobs are actually using more cores than the 7 I allowed. 

Are there any reason why I might not be able to restrict the number of cores of a container instance?

Best,

Carla 

[Thomas Hartmann]

Hi Carla,

probably cgroups are better in constraining resource usage.
It should be possible to runwith cgexec singularity on a simple cgroup
cpu controller rule set [1] and the kernel should take care.

It worked for me in a quick test with an on-the-fly cgroup [2] - but for
something consistent, you probably want to define proper rule sets via
/etc/cgrules.conf

Cheers,
Thomas


[1]
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/resource_management_guide/starting_a_process

https://www.antonlindstrom.com/2015/11/22/limit-processes-with-cgexec.html


[2]
> cgcreate -g cpu:singufoo
> echo 200 > /sys/fs/cgroup/cpu/singufoo/cpu.shares
> cgexec -g cpu:singufoo singularity shell /your/container/path/
> cat /sys/fs/cgroup/cpu,cpuacct/singufoo/cpu.shares
200
> cat /sys/fs/cgroup/cpu,cpuacct/singufoo/tasks
20381

# which is the shell's PID in the container

btw: watch out for hierarchies in resource usages, i.e., if you nest
cgroups where 200 become from 200 from 1000 from the whole CPU share
etc. - I just stumbled over this one... ;)

> --
> You received this message because you are subscribed to the Google
> Groups "singularity" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to singularity...@lbl.gov
> <mailto:singularity...@lbl.gov>.

[Carla Sauvanaud]

Hi Thomas,

Thank you for you answer.
My issue here is that I have an idle machine on which I want to make
some tests with fewer cores.
Therefore, by using cpu cgroups, my container instance is using by
default all cores available as long as there is no other process asking
for CPU time :/
One solution would be to use cpu cgroups and then to run a job with a
CPU intensive infinite loop in parallel I guess.
However I wanted to understand why "taskset" is not doing what I expect
from it.

Best,
Carla

[C. Sauvanaud]

Hi all, 

The issue here is that I run the 'stress-ng' command from a new shell that I create as follows:

  singularity shell instance://myinstance

This particular process does not inherit from the CPU affinity.

Best,

Carla

[Thomas Hartmann]

Hi Carla,

ah, OK - then the cpuset controller might be the right one. cpuset
should be able to bind groups to specific cores and reserve cores
exclusively. However, I have not used the cpuset controller myself yet
but only cpu/cpuacct - so #YMMV ;)

Cheers,
Thomas

[Priedhorsky, Reid]

> For testing purpose I need to run a Singularity container with only a subset of cores from my machine.
> For this purpose I chose to use the "taskset" command and to use is as follows:
> taskset --cpu-list 0-6 singularity instance.start /image.img myinstance
>
> In this case I restricted my container instance to use 7 cores (child processes included) and the command "taskset -c -p PID" confirms this well.
> However, once I am inside my container instance, I get to run heavy jobs such as "stress-ng --cpu 22 -t 1m" (running a CPU load on 22 cores for 1 minute) and the jobs are actually using more cores than the 7 I allowed.

Have you validated that those 22 processes are really running on 22 different cores rather than the 7 allowed (at 3–4 processes per core)?

I ask because asking the kernel “how many cores are installed” and “how many cores am I allowed to use” are different questions, and in my experience it’s easier to ask the former.

HTH,
Reid

[Carla Sauvanaud]

Hi Reid,

Yes indeed they were running on 22 different cores.
But this is OK now because I understood that it was related to me
running the "stressng" command inside a shell created from "singularity
shell ..." that was not given the CPU affinity as I expected.

Best,

Carla