Memory and CPU consumption issue native mqtt (3.12)

[Rony Christian]

I have install rabbitmq 3.12 for native mqtt, as per theory it consume less memory as compare to previous version. I'm using rabbitmq 3.10 and it's consume tons of memory. After getting news of rabbitmq native mqtt, I've upgrade to 3.12 but still memory consumption are same as previous version.

I've more than 1,00,000+ connection at same time.

All featured flags are enabled.

I'm stuck in this from last 4 weeks, please let me know if I miss something. Sharing here my configuration (rabbitmq.conf):

#listeners.tcp = none
listeners.ssl.default = 5671
management.ssl.port   = 15671
vm_memory_high_watermark.relative = 0.6
collect_statistics_interval = 60000

#mqtt.listeners.tcp = none
mqtt.listeners.ssl.default = 8883
mqtt.listeners.tcp.1 = 1883
mqtt.prefetch         = 10
mqtt.vhost            = /
mqtt.exchange         = amq.topic
mqtt.allow_anonymous  = true
#mqtt.durable_queue_type = quorum

ssl_options.cacertfile = /etc/ssl/ca-bundle.pem
ssl_options.certfile = /etc/ssl/cert.pem
ssl_options.keyfile = /etc/ssl/privatekey.pem
ssl_options.verify     = verify_none
ssl_options.fail_if_no_peer_cert = false

management.ssl.cacertfile = /etc/ssl/ca-bundle.pem
management.ssl.certfile   = /etc/ssl/cert.pem
management.ssl.keyfile    = /etc/ssl/privatekey.pem

[Michal Kuratczyk]

You have not provided the information I asked for last week.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/d9059260-c80d-46d9-aeaf-1629b265696an%40googlegroups.com.

--

Michał

RabbitMQ team

[David Ansari]

Make sure to disable management metrics:

management_agent.disable_metrics_collector = true

and to use small TCP buffer sizes.

(This applies to both 3.11 and native MQTT in 3.12.)

More information in https://blog.rabbitmq.com/posts/2023/03/native-mqtt and https://next.rabbitmq.com/mqtt.html#scalability

[Rony Christian]

Thank you for your response.

I've attached here all the information.

here is my server details:

2 cores and 16 GB memory

rabbitmq 3.12 

Erlang/OTP 25 [erts-13.0.4]

using mqtt

following plugins are enable:

rabbitmq_management

rabbitmq_management_agent

rabbitmq_mqtt

rabbitmq_peer_discovery_aws

rabbitmq_peer_discovery_common

rabbitmq_web_dispatch

After disabling management_agent.disable_metrics_collector, memory and CPU consumption are in control. I've added following configuration:

mqtt.tcp_listen_options.sndbuf = 1024
mqtt.tcp_listen_options.recbuf = 1024
mqtt.tcp_listen_options.buffer = 1024
management_agent.disable_metrics_collector = true

I have some doubts regarding buffer size. Can I reduce buffer size and does it reduce memory uses?

Performance are significantly very good, as compare to my previous configuration.

But, when I check memory breakdown (rabbitmq-diagnostics memory_breakdown), other_proc consume a lot of memory.

Reporting memory breakdown on node rabbit@rabbitmq-1...
other_proc: 3.3927 gb (67.36%)
allocated_unused: 1.0056 gb (19.97%)
binary: 0.2024 gb (4.02%)
connection_other: 0.0952 gb (1.89%)
mnesia: 0.0864 gb (1.71%)
other_system: 0.0818 gb (1.62%)
metrics: 0.05 gb (0.99%)
other_ets: 0.0491 gb (0.97%)
code: 0.0359 gb (0.71%)
plugins: 0.0356 gb (0.71%)
atom: 0.0015 gb (0.03%)
msg_index: 0.0001 gb (0.0%)
mgmt_db: 0.0 gb (0.0%)
quorum_ets: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
connection_readers: 0.0 gb (0.0%)
connection_writers: 0.0 gb (0.0%)
connection_channels: 0.0 gb (0.0%)
queue_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

Note: only rabbitmq are install on this server.

Here is output of rabbitmq-diagnostics observer

Currently I've around 30,000 connection and memory uses is 6.5 GB and CPU are running on average of 30%-40%.

Please suggest where I'm doing wrong.

Thanks and Regards

[David Ansari]

> Erlang/OTP 25 [erts-13.0.4]

I recommend to use Erlang 26 instead of 25.

> I have some doubts regarding buffer size. Can I reduce buffer size and does it reduce memory uses?

Yes, it does, please read https://rabbitmq.com/networking.html#tuning-for-large-number-of-connections-tcp-buffer-size
Note that buffer sizes of 1024 bytes are very likely too low for production. In the blog post, I used such low buffer sizes just to show the gain of native MQTT in 3.12 vs non-native MQTT in 3.11. TCP buffer sizes (and also RabbitMQ management metrics) cause an equally high amount of memory in both 3.11 and 3.12.

> Currently I've around 30,000 connection and memory uses is 6.5 GB and CPU are running on average of 30%-40%

The screenshot you pasted shows a total of 3.9 GB, not 6.5 GB?

I can't see any process which requires excessive amount of memory in your screenshot and the screenshot already lists processes ordered by decreasing memory consumption.

Are all 30,000 connection using SSL?

Are the connections sending or consuming messages?

2 CPUs seems too low for me for a RabbitMQ node that's supposed to handle 30,000 connections concurrently. 0.6 - 0.8 CPU usage is not that much depending on what else is going on in the cluster.

[Rony Christian]

Thanks for your quick response.

Here is some latest statistic:

Currently total connection are 38,000+

Are all 30,000 connection using SSL?

>> Yes

Are the connections sending or consuming messages?

>> Yes

output of rabbitmq-diagnostics memory_breakdown

Reporting memory breakdown on node rabbit@rabbitmq-1...

other_proc: 5.1774 gb (67.77%)
allocated_unused: 1.6302 gb (21.34%)
binary: 0.257 gb (3.36%)
connection_other: 0.1279 gb (1.67%)
mnesia: 0.1248 gb (1.63%)
other_system: 0.1063 gb (1.39%)
metrics: 0.0713 gb (0.93%)
other_ets: 0.066 gb (0.86%)
plugins: 0.0415 gb (0.54%)
code: 0.0359 gb (0.47%)
atom: 0.0015 gb (0.02%)

msg_index: 0.0001 gb (0.0%)
mgmt_db: 0.0 gb (0.0%)
quorum_ets: 0.0 gb (0.0%)
quorum_queue_procs: 0.0 gb (0.0%)
quorum_queue_dlx_procs: 0.0 gb (0.0%)
stream_queue_procs: 0.0 gb (0.0%)
stream_queue_replica_reader_procs: 0.0 gb (0.0%)
connection_readers: 0.0 gb (0.0%)
connection_writers: 0.0 gb (0.0%)
connection_channels: 0.0 gb (0.0%)
queue_procs: 0.0 gb (0.0%)
queue_slave_procs: 0.0 gb (0.0%)
stream_queue_coordinator_procs: 0.0 gb (0.0%)
reserved_unallocated: 0.0 gb (0.0%)

1. Why other_proc consume this much memory? Any solution to reduce this?

2. Can I upgrade erlang to 26 without affecting rabbitmq?

3. Any recommendation for buffer size, like what are the value of buffer size according to my need and configuration?

4. Any server recommendation (please suggest we need it for 1,00,000+ connection on production)?

5. Do I disable management plugin to reduce cpu and memory uses? 

Thanks & Regards

[Michal Kuratczyk]

Seems like those GBs of memory are used by ssl_gen_statem processes, which are Erlang processes to handle TLS.

As you can see, their overhead is quite small per-process (278kb in your environment), but given the number of connections,

it adds up. Most likely a large part of that can be reclaimed by garbage collection / process hibernation - they don't actually

need 278kb, they just accumulated some garbage when the connection was established and because, I assume, they are

mostly idle, garbage collection isn't triggered for them.

Things you can try:

1. Assuming the theory is correct, `rabbitmqctl force_gc` should release quite a lot of memory

2. If you have a test environment, try to request process hibernation after some time of idleness. Note that you have to use advanced.config for this

and you need to put all your ssl_options in it (otherwise ssl_options from advanced.config will overwrite your current options). It'd look something like this:

```

[
 {rabbit, [
  {ssl_options, [
 {hibernate_after, 1000},
 {keyfile,
  "/etc/.../server_key.pem"},
 {certfile,
  "/etc/.../server_certificate.pem"},
 {cacertfile,
  "/etc/.../ca_certificate.pem"},
 {verify,verify_none}]}
 ]}
].

```

The key part is the `hibernate_after`, but as I said - you need the rest as well (you can run `rabbitmqctl environment` to get your current settings

in the advanced/Erlang term format).

If you could try the above, it'd be great to confirm this solves the problem. On our side, we'll look into configuring this hibernate_after setting

by default, so that in the future, these processes use less memory without any manual adjustments.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8f3738ec-e476-48ce-851f-72abe6f35398n%40googlegroups.com.

--

Michał

RabbitMQ team

[David Ansari]

1. Please try out what Michał suggested. Alternatively, wait for 3.13 RC 2 (should ship in a few days) which contains https://github.com/rabbitmq/rabbitmq-server/pull/9761 and hibernates ssl_gen_statem processes by default. I expect the memory usage to go down in your use case (at least for connections that are idle most of the time).

2. Yes

3. It depends on how often MQTT messages are sent on the connection and how large MQTT messages are. Hence, no specific recommendations from my side. Choose some values that are lower than the defaults. 

4. What means "1,00,000+" ? Do you mean 10,000+ or 100,000+ or 1,000,000+ ? All recommendations are in https://next.rabbitmq.com/mqtt.html#scalability and https://next.rabbitmq.com/networking.html#tuning-for-large-number-of-connections

5. Correct, I was mainly thinking of memory, but CPU will reduce too.

[Rony Christian]

Hi,

Facing issue with high CPU usage from last few days. sometime cpu goes more then 100%.

currently I've 49000+ connections

I've run rabbitmq-diagnostics runtime_thread_stats, below is the output:

server details:

2 core and 16 GB memory

rabbitmq 3.12 

Erlang/OTP 25 [erts-13.0.4]

using mqtt

Please help me to control cpu consumption. 

[Rony Christian]

hi,

On my production server, facing high CPU usage issue from last few days.

I've run rabbitmq-diagnostics runtime_thread_stats, below are the output.

Server configuration: 

2 cores and 16 GB ram

rabbitmq 3.12 

Erlang/OTP 25 [erts-13.0.4]

using mqtt

live connection: 48000+

CPU consumption is more then 80% (sometimes goes 100% +)