Max File Descriptors in official Docker image (tag: 3.7.12-management) and periodic high CPU usage

[er...@24x8.com]

The official Docker image sets the number of file descriptors to 1048576 by default. I'm trying to figure out how to reduce this (my environment doesn't need anywhere near that). I'm starting the RMQ image as part of a Docker Swarm, using the build-in "docker stack deploy" functionality. When started that way (from hours of research by multiple people), there is no way to set the open file limit (it can be done using a standard "docker run" and with the older version of swarm that used the Python script, but not with "docker stack deploy"). We've also found references that suggest that the docker containers "inherit" the open file limit (i.e. ulimit -n) from the host, but that does not appear to be the case (we're running on Ubuntu 18.04). So my primary question is how to set the open file limit on the standard Docker image when using "docker stack deploy"?

The follow-on question to this is "why do I care that the open file limit is 1048576"? Mostly, I don't. But having so many files sucks up a lot of unnecessary CPU cycles when erl_child_setup runs every few seconds. Research on this uncovers several mentions of this being related to the max number of file descriptors, with suggestions on reducing that limit in order to speed up that operation. So, I'd also like to understand what erl_child_setup does (I couldn't find anything that actually explains its purpose) and whether there may be a way to reduce the CPU usage other than reducing the open file limit?

[Michael Klishin]

erl_child_setup is an auxiliary Erlang process [2] that is a helper for forking with fork(2) and friends.

beam.smp is the actual VM.

I'm not a kernel hacker but increasing the limit should only preallocate a larger collection of file descriptors. Why would those

entries be iterated over if they are not used?

[1] explains how the limit is controlled. Our team does not maintain the Docker image, so I'm not in a position to recommend

what'd be the optimal solution.

I haven't seen any evidence that suggests that simply increasing the limit has any meaningful impact on CPU usage.

If anything I'd expect I/O thread pool size [3] to have a greater effect than the limit since it controls an actual number of threads

(that will be idle most of the time without a heavy I/O load).

Before you reduce CPU usage it would help to understand what consumes CPU time. Have you tried profiling the process,

running it under strace or inspecting its own scheduler stats? (with `rabbitmq-diagnostics runtime_thread_stats`)

rabbitmq-top [4] can also help here and on a mostly idle system, see [5].

1. https://www.rabbitmq.com/networking.html#open-file-handle-limit

2. https://github.com/erlang/otp/blob/master/erts/emulator/sys/unix/erl_child_setup.c

3. https://www.rabbitmq.com/configure.html#customise-environment

4. https://www.rabbitmq.com/memory-use.html

5. https://github.com/rabbitmq/rabbitmq-server/issues/1713#issuecomment-461501717

On Wed, Feb 27, 2019 at 6:52 PM <er...@24x8.com> wrote:

The official Docker image sets the number of file descriptors to 1048576 by default. I'm trying to figure out how to reduce this (my environment doesn't need anywhere near that). I'm starting the RMQ image as part of a Docker Swarm, using the build-in "docker stack deploy" functionality. When started that way (from hours of research by multiple people), there is no way to set the open file limit (it can be done using a standard "docker run" and with the older version of swarm that used the Python script, but not with "docker stack deploy"). We've also found references that suggest that the docker containers "inherit" the open file limit (i.e. ulimit -n) from the host, but that does not appear to be the case (we're running on Ubuntu 18.04). So my primary question is how to set the open file limit on the standard Docker image when using "docker stack deploy"?

The follow-on question to this is "why do I care that the open file limit is 1048576"? Mostly, I don't. But having so many files sucks up a lot of unnecessary CPU cycles when erl_child_setup runs every few seconds. Research on this uncovers several mentions of this being related to the max number of file descriptors, with suggestions on reducing that limit in order to speed up that operation. So, I'd also like to understand what erl_child_setup does (I couldn't find anything that actually explains its purpose) and whether there may be a way to reduce the CPU usage other than reducing the open file limit?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Eric Lenington]

I've verified through experimentation that erl_child_setup does use more CPU the more open (but unused) file descriptors are allocated to the container. When using Docker "swarm" mode (via "docker stack deploy") there is no way to set ulimits, but there is using "docker run". So to test, I launched two containers with "docker run", one with no ulimits set (resulting in over 1000000) and one setting the max open files param to 8000. The one with no limit regularly saw erl_child_setup running with 50%+ CPU load, while the one with 8000 still saw erl_child_setup run, but using less than 1% CPU.

Also, if you ps aux | grep erl_child_setup, you see that the param passed to this process is the max file descriptors, so whatever this process is doing, it's doing it once for every possible file descriptor. Note that during this entire experiment, RabbitMQ is completely idle, with no active connections.

I've tried over a dozen different ways to set the max file descriptors when using "docker stack deploy", but none have had any effect.

Where does the RabbitMQ docker image get it's value from?

If there a way to set the max file descriptors as part of a RabbitMQ config file or environment var?

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/7FiPdGJ6aco/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.

[Michael Klishin]

> if you ps aux | grep erl_child_setup, you see that the param passed to this process is the max file descriptors, so whatever this process is doing, it's doing it once for every possible file descriptor

Can you share some evidence of this?

The runtime can find out what its limit is but I honestly have no idea what it might be trying to do. What specifically is the parameter?

The limit is a kernel setting. It cannot be set via RabbitMQ control file because applications cannot set a limit higher than the hard limit

in the kernel. In addition, modern distributions often use systemd and that has its own mechanism of managing

this limit, so even calling `ulimit -n` in a RabbitMQ startup script won't have any effect (we used to recommend that and it no longer works).

If you use the Ubuntu variant of the image, see [1]. I know nothing about Alpine, unfortunately. I don't see anything that

sets the limit in [2][3]. I suspect there may be something in the base Ubuntu image.

I'd ask on a relevant Docker forum and perhaps in an issue in docker-library/rabbitmq (the maintainers seem to use it for questions

and usually respond quickly). I definitely think there should be a way to alter this limit if there isn't already.

1. https://www.rabbitmq.com/install-debian.html#kernel-resource-limits

2. https://github.com/docker-library/rabbitmq/blob/7b31215f4c8ac87158947b1dfbfd10ad3b62946d/docker-entrypoint.sh

3. https://github.com/docker-library/rabbitmq/blob/7b31215f4c8ac87158947b1dfbfd10ad3b62946d/3.7/ubuntu/Dockerfile

[Eric Lenington]

The only evidence I have is the output of ps aux, which is simply "erl_child_setup 1048576" (or whatever the file descriptor limit is--when I changed it using "docker run" to 8192, the output of ps aux changed accordingly.

What exactly does erl_child_setup do?

I'm actually trying to reduce the number of file descriptors, not increase them. Given that, is there a way for RabbitMQ to affect it?

Yes, I've gone through those files, too (and several others). I can't find anything that sets the value (and in fact, the underlying Ubuntu image is set to 1024 (per ulimit -a).

I have questions on Docker forums. I'll try to post something on the docker-library/rabbitmq Github.

[Michael Klishin]

erl_child_setup is a part of Erlang, not RabbitMQ, so you will have a more informed answer

quicker on erlang-questions [1].

It does take a max file descriptor limit [2] and runs select(2) on them continuously [3]. I won't claim I understand

the intent fully but now it makes some sense.

There are no plans to get RabbitMQ into the system limit management game. It's a responsibility of the deployment operator

and the space is too messy and fragmented across operating systems.

You can try exporting a lower ERL_MAX_PORTS. In fact, I found a blog post from 2012 which I won't share (because too much

could have changed in the runtime in ~ 7 years) which recommends using lower values to reduce memory consumption.

Alternatiively to ERL_MAX_PORTS you can pass the same value as an additional VM argument [4][5].

But the right thing to do is to tweak the limit in the Docker image, and if that's not possible or very involved, make it easy to do so.

1. http://erlang.org/mailman/listinfo/erlang-questions

2. https://github.com/erlang/otp/blob/master/erts/emulator/sys/unix/erl_child_setup.c#L411

3. https://github.com/erlang/otp/blob/master/erts/emulator/sys/unix/erl_child_setup.c#L474

4. `RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS` in http://www.rabbitmq.com/configure.html#supported-environment-variables

5. http://erlang.org/doc/efficiency_guide/advanced.html#ports

[Grigory Starinkin]

Hi,

It might be that CPU usage originates from busy-waiting schedulers (see https://stressgrid.com/blog/beam_cpu_usage/ for performance implications graphs).

Would you be able to try setting RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS variable to: “+sbwt none” and see if it helps in your setup?

Might also worth to give these parameters a try:

• +sbwtdcpu none
• +sbwtdio none

This thread might be related as well: https://groups.google.com/forum/#!searchin/rabbitmq-users/sbwt|sort:date/rabbitmq-users/onyNwSEsaeg/rDu97n9aGAAJ 

Kind Regards,

Grigory Starinkin

[Michael Klishin]

Hi Grigory,

But why would that correlate with the max open file handle limit? Simply having a higher limit (not actual connections) should not

increase the number of Erlang processes on the node in any way.

[Eric Lenington]

It shouldn't, but it does (even on an otherwise idle server). The max file descriptors setting for the container is passed (as the only) command line argument to erl_child_setup. Yesterday, we were finally able to find a way to change this value in a docker stack environment (see below). Once we did that, the argument passed to erl_child_setup when down (to the max file descriptors value we set: 64000), and the CPU load dropped to match.

The place we found to set the file descriptor limit was in /etc/docker/daemon.json. This file exists on the host and controls for "nofile" setting (among other things) for all containers run by that docker daemon. This isn't the most ideal solution, since the value applies to all containers, not just the RabbitMQ container, but it gets us to a workable solution until it's (hopefully) possible to set ulimits on a per-container basis in the future (specially when using docker stack deploy, where this is not supported in the current docker release). Here's the content of our JSON file:

{

  "default-ulimits": {

    "nofile": {

      "Name": "nofile",

      "Hard": 64000,

      "Soft": 64000

    }

  }

}

[Grigory Starinkin]

Hi Michael,

That’s a very good question. I think I missed it.

Eric,

Is there an easy way to reproduce this? How do you test this?

Thank you,

[Eric Lenington]

Just set your ulimit for max file descriptors to 1048576 (ulimit -n 1048576 in your startup script and probably restart your server, as the RabbitMQ daemon needs to pick this up). That was the default setting in a new docker deployment, which is what caused this issue to begin with (typically, it's much lower than this--like 1024--and needs to be increased).

--Eric

[Michael Klishin]

My response was to Grigory's hypothesis around a runtime scheduler flag used by default at the moment

(by the Erlang VM, not set by RabbitMQ). It helps with a very large number of processes (connections, queues) and

there should be no correlation with the max file handle limit.

The findings above confirm that erl_setup_child for some reason runs select(2) on every possible file descriptor there is.

I don't know why that's the case. That's a great question for erlang-questions

I will add a note about default-ulimits in Docker to the docs with a link to [1]. That's a great find, Eric!

1. https://docs.docker.com/engine/reference/commandline/dockerd/

[Michael Klishin]

[1] is updated. Hopefully it will safe Docker users some time in the future. Thank you, Eric, for sharing your findings

and an example small enough to include into the docs!

1. https://www.rabbitmq.com/install-debian.html#kernel-resource-limits