3.10.7: queue memory usage

[Дмитрий Николаевич]

Hello everybody.

OS: debian 10
RabbitMQ 3.10.7, Erlang 25.0.4 [jit]

We have next problem after upgrade to 3.10.7.

At some moment rabbitmq starts consume memory of 6..10GB,
however its queues are empty at this time, no processing is on.
Web UI and `rabbitmq-diagnostics memory_breakdown` inform that most memory is allocated for `queue_procs`. Broker restart does not release memory.
The only operation that helps us is `purge` of queues, though they are empty.

Another negative factor is huge perf decrease at this state, though watermark is far.


. 

[Michal Kuratczyk]

Hi,

Please provide full details about these queues (most importantly: queue type).

Check `rabbitmq-diagnostics observer` and share a screenshot (we should see if that's 1 queue or more).

What version have you upgraded from? How long ago?

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/dcb5e79d-5ead-4456-9e1f-93f0eb836596n%40googlegroups.com.

--

Michał

RabbitMQ team

[Дмитрий Николаевич]

`durable` is true, type is `classic`, messages are not persisted.
there is no special policy. processes of all these queues has high memory usage.
`rabbitmq-diagnostics observer` is not possible to apply now, this sutuation has gone away after `purge` command.
previous version was 3.8.3, as I know this situation did not happen for this version.

I have  log, there are some messages:

2023-04-17 14:34:05.983482+00:00 [info] <0.679.0> Starting message stores for vhost 'XXX'
2023-04-17 14:34:05.983760+00:00 [info] <0.684.0> Message store "YYY/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2023-04-17 14:34:05.989884+00:00 [info] <0.679.0> Started message store of type transient for vhost 'XXX'
2023-04-17 14:34:05.990194+00:00 [info] <0.688.0> Message store "YYY/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2023-04-17 14:34:05.993349+00:00 [info] <0.679.0> Started message store of type persistent for vhost 'XXX'
//it takes almost one minutes and then alarm is ON
2023-04-17 14:35:07.849301+00:00 [info] <0.410.0> vm_memory_high_watermark set. Memory used:10386735104 allowed:10000000000
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0> memory resource limit alarm set on node 'rabbit@ZZZ-rabbitmq'.
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0>
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0> **********************************************************
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0> *** Publishers will be blocked until this alarm clears ***
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0> **********************************************************
2023-04-17 14:35:07.849626+00:00 [warning] <0.408.0>

четверг, 27 апреля 2023 г. в 14:32:14 UTC+3, Michal Kuratczyk:

[Michal Kuratczyk]

We can't guess what's going on based on pieces of information like this.

We would love to fix this if it turns out to be a RabbitMQ bug but you'd need to put some effort into reproducing it

to provide us with the steps.

Note: while I don't think we've ever heard about an issue like this before, we are completely rewriting classic queue implementation.

In RabbitMQ 3.12, to be released soon, classic queue implementation will be very different, especially if you set x-queue-version=2.

Therefore, whatever happened here, will likely behave differently.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ef3683af-26b2-4702-bf5b-8a2ea3ffa75cn%40googlegroups.com.

--

Michał

RabbitMQ team

[Дмитрий Николаевич]

Case repeated again.

пятница, 28 апреля 2023 г. в 11:06:39 UTC+3, Michal Kuratczyk:

[Michal Kuratczyk]

I'd recommend trying lazy queues or upgrading to 3.12 + classic queues v2 as soon as available. Historically, queues kept a lot of stuff (including messages) in memory, because disks were slow.

These days disks are much faster so we are moving towards writing stuff to disk - storage operations are no longer a problem and used less memory is a huge advantage.

In 3.12 all classic queues will behave like lazy (even classic queues v1) but v2 should offer much better performance overall.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7869158f-5c9b-42f7-addd-5f3debd3f26an%40googlegroups.com.

--

Michał

RabbitMQ team

[Mc Polu]

Hello, apologies for hijacking the thread, this:

=> In 3.12 all classic queues will behave like lazy (even classic queues v1)

Is alarming for use cases where low latency is a must. Will there be a way to keep queues in memory, even if it is not the default, please?

Thank you.

[Michal Kuratczyk]

Hi,

Queues will still keep a small number (based on the consumption rate) of messages in memory. Based on our testing, classic queues v2 in 3.12 provide lower latency in virtually all test scenarios,

while using less memory at the same time. Having said that, 3.12 RC2 has just been released and we would very much appreciate you testing and letting us know your results.

Alternatively (or in parallel), if you can provide perf-test flags that mimic your workload, we can check how it goes.

Here are some numbers I'm currently collecting from 3.11.0 and 3.12-rc2  for a blog post that will be published soon.

You can see more results here: https://grafana.rabbitmq.com/d/1OSIKB4Vk/rabbitmq-performance-tests?orgId=1&var-namespace=performance-tests&var-instance=All&var-percentile=0.99&var-rabbitmq_cluster=All&var-PVC=All&var-workload_name=All&var-msg_size=All&from=1683979755000&to=1683984910203

And here's 5 queues, each with 1 publisher publishing 10000 1kb messages per second and 1 consumer consuming all of them for a total of 50000 1kb messages flowing through the node per second

(perf-test -x 5 -y 5 -r 10000 -c 500 -qp fivers-%d -qpf 1 -qpt 5 -qa x-max-length=1000000 -ad false -f persistent -s 1000 -qa x-queue-version=...):

Classic queues v1 in 3.11 don't even play in the same league. Here I focused on non-lazy v1 in 3.11 vs 3.12 (what you'd get by simply upgrading without any other changes):

3.11 struggles to even sustain the expected 50k msg/s. And 3.12 uses less memory at the same time (the queues are empty in this case - all messages get consumed immediately, so memory usage is low anyway):

Please don't take this as the definite reassurance that your use case will be perfectly supported. Probably the biggest challenge for the RabbitMQ team is lack of feedback from users - we hear about issues, but we don't hear about things working well.

Very few users share their workload details with us, and therefore we mostly test with hypothetical scenarios. By all means, please test it yourself and/or share details with us. Sharing use case information is a great way to contribute to RabbitMQ!

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/53374836-d12a-40ca-a022-800b878f80c1n%40googlegroups.com.

--

Michał

RabbitMQ team

[Mc Polu]

Hello, thank you, we will be testing during the first three weeks of June and will let you know of the results.