Memory usage other_ets

[Andrea R.]

Hi all,
I've a very basic installation of rabbit on a Ubuntu 22.04 server for a streaming message service.
The server is very small, 4Gb RAM and a watermark for memory of 0.4 but we exchange few message at minute.

I've an issue with the other_ets memory that seam to increase day by day until it full the memory and the server stop to ingest new messages from producer.

The status of RabbitMQ is available at link https://pastebin.com/UN1VVv1L
Someone can suggest me what is the problem? Searching on Google I not found any valid suggestion on this.

Regards
Andrea

[Andrea R.]

Sorry, I make a mistake and the screenshot about the memory usage not work. Now you can find general memory usage and ets

[nsgc]

could it be a log rotation problem in ets?

[Andrea R.]

How it's possible to rotate the ets? I don't find the settings about it.

[Karl Nilsson]

It looks like a component (seshat) that is responsible for managing counters is using a fair bit of ETS memory. 

I assume you use short lived connections and/or other entities?

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/231ab1a9-5d76-4e99-9a53-e437b8aacdc1n%40googlegroups.com.

--

Karl Nilsson

[kjnilsson]

I noticed you are on RabbitMQ 3.10.1 from May this year. You need to upgrade to a more recent version (latest 3.10.x would be good) as I realised we fixed this in June.

Cheers

Karl

[Andrea R.]

Thanks for your reply.

Yes, on our platform the behavior is that consumers have short connections, because they only connect to the queue for as long as they need to and then they are destroyed.

Now I've upgraded the system to the version 3.10.8 and we try if it solve. This test take long time because the issue of memory need some week before the problem occurs

[kjnilsson]

Great yes this should fix it but let us know if not.

[Andrea Romanello]

Hi to all,

I came back after the upgrade to the version 3.10.8 and some time of monitoring.

I'm very sad, but the problem is already present and we need to restart our server after two week of running.

This is the memory occupation before the restart.

As you see, the memory usage for others is very high compared to others.

There something that we can do to solve this problem?

Kind regards

Andrea

You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/Pt8tZQIE8mo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8cafc317-f64f-485a-8e7c-c23c8eb4ec09n%40googlegroups.com.

[Karl Nilsson]

Can you show us the per ETS table memory view for this as well?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CABS4Ym--UUs5%3DBpv6kVEKQa3_%3DbKPgKigvZEj8D32WnosmFwsg%40mail.gmail.com.

--

Karl Nilsson

[Steve Horne]

Hi all. We're seeing a similar issue since we upgrade to 3.10.7 and started making heavy use of stream queues (not clear if related). Can't upload attachments and don't know how to get the per ETS table memory view in text, here are the top three entries transcribed manually:

Name                                     | Owner Name                     | Memory  | Size    | Type  | Named  | Protection | Compressed
-----------------------------------------------------------------------------------------------------------------------------------------------------------
anonymous                            | seshat_counters_server    | 8.6 GiB | 1922606 | set   | false  | public     | false
channel_stats_deliver_stats  | rabbit_mgmt_storage        | 538 MiB | 34564   | set   | true   | public     | false
rabbit_stream_coordinator    | ra_coordination_log_ets    | 98 MiB  | 111397  | set   | false  | public     | false

Any advice appreciated as currently we're having to periodically restart RabbitMQ in production to keep memory at a sane level.

Best regards,

Steve Horne

[Karl Nilsson]

Are you using steams over amqp or the stream protocol/plugin? 

Can you upgrade to the latest 3.10 just to make sure the fix shipped later than I thought. 

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/96d4ed93-45d4-416b-89b4-464ea07d85c7n%40googlegroups.com.

--

Karl Nilsson

[kjnilsson]

OK I think we found it, expect a fix in the near future. Here is the github issue you can follow:

https://github.com/rabbitmq/rabbitmq-server/issues/6317

[Andrea R.]

Hi Karl,

me too i'm using AMQP protocol with RabbitMQ 3.10.8. The queue was created with type Stream.

Thank you for bug reporting.

Regards

Andrea

[kjnilsson]

Ok the bug I found and raised is only if you're using the stream protocol, _not_ when using AMQP. So if this still happens when using only AMQP (which I think is what you are using).

Can you tell me more about how your consuming applications are using streams over AMQP. Do they disconnect frequently? If so do they send a basic.cancel first or just terminate the channel? The more information the better.

[Steve Horne]

Still seeing the issue with 3.10.10. The use-case for streams is pub/sub notifications between services, we're calling BasicCancel as part of service shutdown but the memory usage remains very high.

[Steve Horne]

And yes sorry we're using AMQP not the stream protocol.

[kjnilsson]

try 3.10.11, 

https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.10.11

[Steve Horne]

Memory usage is significantly better since we upgraded to 3.10.11. We'll continue to monitor but this looks like our issue might be resolved. Many thanks @kjnilsson for the fix (and indeed for all your work on this great piece of software).

Steve

[Andrea R.]

Hi @kjnilsson

also on my environment seem the version 3.10.11 solve the problem.

Thank you for support!

[Luke Bakken]

Thanks everyone for letting us know that upgrading has resolved your issues.

[Hritik kumar]

Hey everyone,

We've encountered an unexpected issue with one of our nodes experiencing a significant spike in memory utilization. Upon investigation, we've found that the majority of the memory is being consumed by other_ets
Any insights or advice on resolution are appreciated.

Environment:
RabbitMQ: 3.11.5
Erlang: 25.3.2.10

queue type: quorum

rabbitmq conf
listeners.tcp.1 = 0.0.0.0:5672
mqtt.listeners.tcp.1 = 0.0.0.0:1443
mqtt.tcp_listen_options.backlog = 9182
mqtt.tcp_listen_options.recbuf  = 191072
mqtt.tcp_listen_options.sndbuf  = 191072
mqtt.tcp_listen_options.keepalive = true
mqtt.tcp_listen_options.nodelay   = true
mqtt.tcp_listen_options.exit_on_close = true
mqtt.tcp_listen_options.send_timeout  = 120
mqtt.allow_anonymous  = false
mqtt.default_user     = guest
mqtt.default_pass     = guest
mqtt.vhost            = /
mqtt.exchange         = amq.topic
mqtt.subscription_ttl = 86400000
management.listener.port = 15672
management.listener.ip   = 0.0.0.0
log.file.level = info
vm_memory_high_watermark.relative = 0.6
handshake_timeout = 20000
tcp_listen_options.backlog = 9182
tcp_listen_options.recbuf  = 191072
tcp_listen_options.sndbuf  = 191072
tcp_listen_options.keepalive = true

[Luke Bakken]

Two issues:

• Please do not reply to an old discussion
• RabbitMQ 3.11.x is not eligible for community support - https://github.com/rabbitmq/rabbitmq-server/blob/main/COMMUNITY_SUPPORT.md

You should upgrade.