[lxc-users] Strange CPUSET CGROUP behaviour

[Ben Green]

Hi all,

We are starting to use LXC in production here, after solving our final
problem (inconsistent and disappearing cgroup space - Thread name
"Disappearing Cgroups" on this list) completely with the help of Serge
E. Hallyn.

Now something odd has come up. I have one server on the Debian kernel
4.19.0-9-amd64 and another on 4.19.0-10-amd64. On the 4.19.0-9-amd64
kernel we cat this:

# cat /sys/fs/cgroup/cpuset/forcontainers/tasks
1216
5065
12372
etc.

But on the other one with the 4.19.0-10-amd64 we get this:

#  cat /sys/fs/cgroup/cpuset/forcontainers/tasks

No output! Also on the 4.19.0-10-amd64 machine it is not possible to add
to the list of tasks:

# echo $$ >> /sys/fs/cgroup/cpuset/forcontainers/tasks
-bash: echo: write error: No space left on device

However, we've had this happen before, and thought that adding
'swapaccount=1' to the grub command line arguments had fixed it. Seems
it hadn't. I'm not even sure therefore that this is consistent between
kernels. It appears it might be entirely random.

Has anyone else experienced this phenomena? Any clues as to what might
be happening?

It seems that the cgroup has a messed up mount, but it looks fine to me:

  cgroup on /sys/fs/cgroup/cpuset type cgroup
(rw,nosuid,nodev,noexec,relatime,cpuset)

Remove and remake the /sys/fs/cgroup/cpuset/forcontainers directory and
the problem persists.

--
Cheers,
Ben Green

[Christian Brauner]

On Thu, Sep 03, 2020 at 03:27:37PM +0100, Ben Green wrote:
> Hi all,
>
> We are starting to use LXC in production here, after solving our final
> problem (inconsistent and disappearing cgroup space - Thread name
> "Disappearing Cgroups" on this list) completely with the help of Serge
> E. Hallyn.
>
> Now something odd has come up. I have one server on the Debian kernel
> 4.19.0-9-amd64 and another on 4.19.0-10-amd64. On the 4.19.0-9-amd64
> kernel we cat this:
>
> # cat /sys/fs/cgroup/cpuset/forcontainers/tasks
> 1216
> 5065
> 12372
> etc.
>
> But on the other one with the 4.19.0-10-amd64 we get this:
>
> #  cat /sys/fs/cgroup/cpuset/forcontainers/tasks
>
> No output! Also on the 4.19.0-10-amd64 machine it is not possible to add
> to the list of tasks:
>
> # echo $$ >> /sys/fs/cgroup/cpuset/forcontainers/tasks
> -bash: echo: write error: No space left on device

That usually indicates that cpuset.cpus and/or cpuset.mems is empty and
that cgroup.clone_children in the parent cgroup isn't set to 1. LXC
should usually take care of this though.

Christian

[Ben Green]

So apparently this is a known kernel issue. Solution is to write values into cpuset.mems and cpuset.cpus.

https://serverfault.com/questions/579555/cgroup-no-space-left-on-device

Cheers,

Ben

_______________________________________________
lxc-users mailing list
lxc-...@lists.linuxcontainers.org
http://lists.linuxcontainers.org/listinfo/lxc-users

-- 
Cheers,
Ben Green