Tuning pods running on Kubernetes with the containerd runtime for a high number of connections

[Rilind Harizaj]

Hi all,

I am currently in the process of modifying open file handles limit for RabbitMQ as recommended in the Deployment Guidelines:
https://www.rabbitmq.com/docs/production-checklist#resource-limits-file-handle-limit

I am running an EKS Cluster with version 1.32, and have successfully modified the open file handle limits of the EC2 instances that the EKS cluster uses as nodes. 

However, the container specific settings still show the default open file handle limit, and I am trying a workaround, in which I add the command to modify the open file handle limit in the initContainer, like so:

spec:

  ....

  override:

    statefulSet:

      spec:

        template:

          spec:

            containers: []

            initContainers:

              - name: setup-container

                command: ["sh", "-c", "cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins ; echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf && sed -e 's/default_user/username/' -e 's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf && chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf ; sleep 30; echo '* soft nofile 2000000' >> /etc/security/limits.conf; echo '* hard nofile 2000000' >> /etc/security/limits.conf; echo '* soft nproc 2000000' >> /etc/security/limits.conf; echo '* hard nproc 2000000' >> /etc/security/limits.conf"]

Does anyone whether there is another (better) way to do this?


To the extent of my knowledge this largely depends on the container runtime used, in this case containerd.


Best

Rilind

[Vilius Šumskas]

Hello,

 

open file handle limit must be set on the host running the container, not inside the container. For example, on GKE Clusters running Container-Optimized OS it is 1048576. On EKS it should be around 512K by default. If you are seeing lower limits you have something customized or maybe you are hitting AWS bug like https://github.com/awslabs/amazon-eks-ami/issues/1551

 

--

    Vilius

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/58782937-8c8d-4907-9260-732ddbe0528dn%40googlegroups.com.

[Michal Kuratczyk]

Containered 2.0 changed the behaviour in this area, see https://github.com/GoogleCloudPlatform/k8s-node-tools/tree/master/containerd-nofile-infinity for example.

I ran into this problem after upgrading GKE to 1.33. I was able to increase the limit from within the container. After that, I started experimenting with solutions to this,

see https://github.com/rabbitmq/rabbitmq-server/commit/46f83bba23f25ab042f848bc705bb6168e101442 (this has not been merged/released in any RabbitMQ version).

Changing the default behaviour is always tricky though:

1. we can try to set a higher limit (say, 65k) and ignore errors if we can't do it - this would help some users, eg. those upgrading GKE to 1.33+; however, if someone sets the limit

    to a higher value, we would actually lower the limit for them

2. we can try to set the limit and fail to boot if we can't

3. we can try to bump the limit but only if it's low. Eg, if ulimit is below 65k, try to set it to 65k and ignore errors

Right now I'm leaning towards the last option - best effort that should help avoid issues for most users unaware of open file limits in their environment,

but shouldn't affect anyone who explicitly sets high limits. Hopefully a failed attempt to change the limits won't trigger alarms in any security tools.

If anyone has opinions, I'm interested to hear them.

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB8134A642F37E56B3C0C559D6922BA%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Vilius Šumskas]

For GKE 1.33 it was a temporary issue, see https://cloud.google.com/kubernetes-engine/docs/troubleshooting/known-issues#containerd-ulimit-reduced . It was fixed on host level since then. Also, you most probably won‘t be able to increase ulimit above the 1M maximum without modifying fs.nr_open on those GKE Linux nodes https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config#sysctl-options

 

I imagine environment variable is useful, but I would vote to leave the limit as is if env. variable is not set. Alternatively, detect too low limit and show a warning during startup, without modifying it. Most of the time this should be decision at the OS/operations level and only for particular cases.

 

--

    Vilius

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/CALWErWLBennftVU%2BwGZseAgvVv9pj0bOhMW_9cmBToOxZuZgvw%40mail.gmail.com.

[Michal Kuratczyk]

Good to know about GKE. I reported the problem to Google Cloud Support but back then they just acknowledged the

default changed and didn't express any intent to restore it. I'm glad they did.

Adding a dedicated env variable but only setting the limit when it's explicitly set was the approach

preferred by some of the team members. I guess that's what we'll settle on then.

Best,

To view this discussion visit https://groups.google.com/d/msgid/rabbitmq-users/AM8PR01MB8134F5AFEF8406F67C10CD4B922BA%40AM8PR01MB8134.eurprd01.prod.exchangelabs.com.

--

Michal

RabbitMQ Team

[Rilind Harizaj]

Hello,

see: https://github.com/GoogleCloudPlatform/k8s-node-tools/tree/master/containerd-nofile-infinity

This is a result of containerd 2.0's behavior, so I will be actively looking into how one can achieve the same in an EKS environment.

Best

Rilind

[Rilind Harizaj]

Hi,

since the rabbitmq 4.1-management image uses ubúntu as the underlying linux distro, I believe that the max hard limit on open files I am hitting relates to a hard-coded value in the source code of the os:

https://askubuntu.com/questions/1510935/unable-to-increase-max-number-of-open-files-per-process-in-22-04

From the post:
" The default kernel compile time open file limit is defined as fs/file.c:unsigned int sysctl_nr_open __read_mostly = 1024*1024; and boot time limits will be restricted to that limit, as you have discovered. So the short answer to the question for regular users is that you can't."

Since in the rabbitmq containers, there is no root access which would allow to change the values in the containers as can be done in a workaround for Ubuntu users, increasing the limit beyond this values is currently not possible.

This claim is supported by the output of the ulimit -Hn command on my system: 

rabbitmq@xxxxxxxx:/$ ulimit -Hn
1048576 (which is 1024 * 1024)

High concurrency in the context of Native MQTT is this not fully possible for with this rabbitmq-image.

(P.S: While I conducted some research, I cannot say for sure that I have successfully identified the root cause.)

Best

Rilind

On Tuesday, August 12, 2025 at 3:39:53 PM UTC+2 Michal Kuratczyk wrote:

[Rilind Harizaj]

Hi all,

I have conducted some research on the subject and would like to suggest the following solution, which has preliminarily worked for me:

• EKS Version: 1.33

• containerd info:

  • Client:
    Version: 1.7.27

    Server:
    Version: 1.7.27
    
    

• AMI Information (partial):

  • Amazon Linux release 2023.8.20250721 (Amazon Linux)
    NAME="Amazon Linux"
    VERSION="2023"
    ID="amzn"
    ID_LIKE="fedora"
    VERSION_ID="2023"
    PLATFORM_ID="platform:al2023"
    PRETTY_NAME="Amazon Linux 2023.8.20250721"
    ANSI_COLOR="0;33"

 

Step-by-step guide:

Install git and go, on all eks nodes.

yum install git go -y

Make sure to clone the NRI project and build the folder for running the ulimits-adapter-plugin, which is the one we are interested in.

git clone https://github.com/containerd/nri.git cd nri/

Modify the current /etc/containerd/config.toml to include the following settings:

[plugins."io.containerd.nri.v1.nri"] # Enable NRI support in containerd. disable = false # Allow connections from externally launched NRI plugins. disable_connections = false # plugin_config_path is the directory to search for plugin-specific configuration. plugin_config_path = "/etc/nri/conf.d" # plugin_path is the directory to search for plugins to launch on startup. plugin_path = "/opt/nri/plugins" # plugin_registration_timeout is the timeout for a plugin to register after connection. plugin_registration_timeout = "5s" # plugin_requst_timeout is the timeout for a plugin to handle an event/request. plugin_request_timeout = "2s" # socket_path is the path of the NRI socket to create for plugins to connect to.

socket_path = "/var/run/nri/nri.sock"

Restart the containerd service

systemctl restart containerd

Make sure you are within the NRI repository with the build file from the previous make command in there. Run:

./build/bin/ulimit-adjuster -idx 10 -verbose true

Now all that is left is to schedule a pod or a kubernetes object according to the spec below:

apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 3 selector: matchLabels: app: myapp template: metadata: labels: app: myapp annotations: ulimits.noderesource.dev/container.sleep: | - type: memlock hard: 987654 soft: 645321 - type: RLIMIT_NOFILE hard: 10000000 soft: 10000000 - type: nproc hard: 9000 spec: containers: - name: sleep image: ubuntu:latest command: - /bin/bash - -c - "ulimit -a; ulimit -Ha; sleep inf"

terminationGracePeriodSeconds: 3

Output of command:

Soft Limits (added from me for readability)

real-time non-blocking time (microseconds, -R) unlimited

core file size (blocks, -c) unlimited

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 30446

max locked memory (kbytes, -l) 630

max memory size (kbytes, -m) unlimited

open files (-n) 10000000

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 10240

cpu time (seconds, -t) unlimited

max user processes (-u) 0

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

Hard Limits (added from me for readability)

real-time non-blocking time (microseconds, -R) unlimited

core file size (blocks, -c) unlimited

data seg size (kbytes, -d) unlimited

scheduling priority (-e) 0

file size (blocks, -f) unlimited

pending signals (-i) 30446

max locked memory (kbytes, -l) 964

max memory size (kbytes, -m) unlimited

open files (-n) 10000000

pipe size (512 bytes, -p) 8

POSIX message queues (bytes, -q) 819200

real-time priority (-r) 0

stack size (kbytes, -s) 10240

cpu time (seconds, -t) unlimited

max user processes (-u) 9000

virtual memory (kbytes, -v) unlimited

file locks (-x) unlimited

The important part is the annotation provided here. Every pod scheduled on a node that has NRI setup will be able to define its resource limits, without being superimposed by the opinionated settings of containerd 1.7.27 on AWS EKS (or the settings that influence it).

 

 

Links:

NRI:

Containerd v1.7 NRI Support:

Best

Rilind

On Tuesday, August 12, 2025 at 3:39:53 PM UTC+2 Michal Kuratczyk wrote: