delayed messages every 15 seconds

[Uli Waibel]

Hi all, 

I have a Spring Boot application that writes data to Rabbit-MQ queues and a second Java Spring Boot application that reads data from the queues. There are 8 queues, each of which is written with 50 messages per second. On the other hand, there are 8 listeners that each read from a queue.
The average transfer time is approx. 3 ms from send to receive a single message. However, the transfer time always increases to approx. 60 ms for a short time after approx. 15 seconds.
The application runs in a Kubernetes cluster, as well as Rabbit-MQ which is set up as a 3-node cluster.
Does anyone have an idea what could be causing this peak every 15 seconds? The version is RabbitMQ 3.12.13

Best regards 

Uli

[Uli Waibel]

An update to the original post

all queues are added to a exchange of type "Direct"

Queues are defined as "durable"

/Uli

[Michal Kuratczyk]

Some questions to clarify:

1. What is the queue type?

2. Is it precisely 50 msgs/s or just roughly?

3. What happens after the peak? Does it go back to ~3ms for another 15 seconds?

4. How large are the messages?

In general - 4.0 is the version with community support at this point, so you definitely should upgrade,

which could alleviate the problem on its own. Additionally, 4.0 introduced a feature you might be interested in

https://www.rabbitmq.com/docs/local-random-exchange

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion, visit https://groups.google.com/d/msgid/rabbitmq-users/6a4a4e9f-255f-4495-a6e7-2f6dce8de87bn%40googlegroups.com.

--

Michal

RabbitMQ Team

This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

[Uli Waibel]

Hi Michal, 

thank you for the fast response

the message size is approx. 1k. 

Queue type is a durable queue

50 /s is precisely (+- 1%)

after the peak it goes imediatelly down to approx. 3ms

we think to upgrade to v4.x but we had issues after an upgrade and so we decided to go back to v3.x. But definitely we will upgrade in near future

/Uli

[Michal Kuratczyk]

"durable" is not really a queue type. That'd be "classic" (v1 or v2), quorum or stream.

So the precision of 50 msgs/s is most likely what leads to spikes "every 15s". I don't think anything in particular happens every 15s inside RabbitMQ,

it's probably just that 50 msgs/s * message size * 15 * 8  (so 6MB or so) is exactly the amount of data that causes something to happen.

Since you don't know the queue type, most likely it's a classic queue v1. Some ideas for debugging:

* check the publisher confirm latency - does it spike as well?

* publish faster/slower to see if the spike frequency changes (that'd confirm that it's not so much "every 15s" as "every 6MB" or "every 6000 messages" or something)

* convert the queues to classic v2 (more performant in general and a different on-disk representation may change this behaviour)

* if you are using mirroring (HA policies) - see if that happens without them

But generally speaking: RabbitMQ 4.0 (which only supports v2 for classic queue) + perhaps local-random-exchange is likely the solution anyway.

Best,

To view this discussion, visit https://groups.google.com/d/msgid/rabbitmq-users/d4bcefb6-eba3-4bdd-8d4e-8668cffd30e3n%40googlegroups.com.

[Michal Kuratczyk]

Oh, and don't forget this can be a consumer-side problem as well. If the consumer's prefetch fills up,

RabbitMQ will not deliver it immediately.

[Uli Waibel]

thank you for the feedback

queue type is "classic", could see it in the Admin-UI

we created a special test clients where we have removed all business logic except of the message publish / consume part --> same result

we ran the same test on single node instance (no HA) - almost same result

will try to switch to the classic v2 queue types along with an upgrade to v4

try to use the local-random-exchange 

regards

Uli