RabbitMQ .NET Client fails to perform various operations on broker due to timeout

[Simone Musetti]

Erlang version: 21.2

RabbitMQ version: 3.7.12

Rabbit .Net Client: 6.1.0

Environment:

- Virtual machine with Windows Server 2016

- No antivirus, neither any endpoint policies

Welcome everyone.

Lately I'm facing this problem with the broker. I have a cluster with two nodes. When performing operations on broker I get timeout exceptions on QueueDeclare and QueueDelete. Both nodes are up & running. Internet connection is good between them and they are not partitioned.

I don't get why the broker is timing out. 

This series of operations, in order, take up to 20 seconds in total (or fails) to be performed on an idled broker and for a single message:

- channel creation

- queue declare

- exchange declare

- queuebind

- basicpublish

- queuedelete

- exchangedelete

- channel closing

Here the exceptions I get, per operation:

QueueDeclare operation

Timeout operazione. -->    in RabbitMQ.Util.BlockingCell`1.WaitForValue(TimeSpan timeout)

   in RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)

   in RabbitMQ.Client.Impl.ModelBase.QueueDeclare(String queue, Boolean passive, Boolean durable, Boolean exclusive, Boolean autoDelete, IDictionary`2 arguments)

   in RabbitMQ.Client.Impl.AutorecoveringModel.QueueDeclare(String queue, Boolean durable, Boolean exclusive, Boolean autoDelete, IDictionary`2 arguments)

QueueDelete operation

Timeout operazione. -->    in RabbitMQ.Util.BlockingCell`1.WaitForValue(TimeSpan timeout)

   in RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout)

   in RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body)

   in RabbitMQ.Client.Framing.Impl.Model._Private_QueueDelete(String queue, Boolean ifUnused, Boolean ifEmpty, Boolean nowait)

   in RabbitMQ.Client.Impl.AutorecoveringModel.QueueDelete(String queue, Boolean ifUnused, Boolean ifEmpty)

Do you might have any clue about the timeout I get during these operations executions?

Thank you in advance for any response.

Simone

[Warren Pang]

Do you have any issues with busy IO, memory leak, or network filter? We expect error logs both server’s and client’s.

Thanks 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/fadf26be-5b81-496c-9529-06dddf052f6ao%40googlegroups.com.

[simone....@gmail.com]

Thank you for your response.

No memory leak, VM is almost idle, no antirivurs (neither Windows Defender or Windows Firewall). 

Server logs on debug level are on attachment.

The many closing connection that you might find are related to the problem: my application has a 15 seconds timeout on every RabbitMQ operation. The problem is, the broker is not responding in that amount of time.

Thank you, Simone.

[simone....@gmail.com]

To be honest, after a while both machines started to run correctly. In about two hours, the problem restarted.

[Tessa Plum]

simone....@gmail.com wrote:
> To be honest, after a while both machines started to run correctly. In
> about two hours, the problem restarted.

I'd suggest you monitor the IO issue. If you have been using RMQ in VM
environment, IO issue can cause the similar problem. For example, a
snapshot process can make server hang up for long time.

B. regards,
Tessa

[simone....@gmail.com]

Hi, thank you for your response.

What do you mean by monitoring IO? The mnesia errors on the logs are related to a machine restart that I performed. Logs for the .NET Client are relative to a precedent period of time. 

No snapshot has been taken during the test.

In any case I will try to perform an IO monitoring.

Thank you, Simone.

[simone....@gmail.com]

Hi,

sorry for the high amount of time I took to answer.

Just performed IO monitoring, nothing relevant to be added here: machines are never put under pressure.

However lately I found some problems in performing basic operations after a nertwork partition recovered succesfully. In logs I can see:

operation basic.publish caused a channel exception not_found: no exchange 'exchange' in vhost 'vhost'

operation queue.declare caused a channel exception not_found: failed to perform operation on queue 'queue1' in vhost 'vhost' due to timeout

I think the problem might be related to the one I was talking previously.

I checked and:

• there are no problems about machine overload
• network partition has been recovered succesfully: both nodes see each other
• the error is related to operations performed on the local node (locahost) and not on the other one
• these operations have been performed using the latest .NET client version
• no VM snapshots are taken
• the installation is local, on my personal PC, so no WAN

 Do you have any idea how the broker might fail performing these operations? 

Thank you again for your time, Simone.

[Amari CH]

simone....@gmail.com wrote:
> However lately I found some problems in performing basic operations
> after a nertwork partition recovered succesfully. In logs I can see:

> *operation basic.publish caused a channel exception not_found: no
> exchange 'exchange' in vhost 'vhost'*
> *operation queue.declare caused a channel exception not_found: failed to
> perform operation on queue 'queue1' in vhost 'vhost' due to timeout*
> *
> *

what are OS and rabbitmq/erlang version?

--
Amari CH
https://maddoghost.com/

[simone....@gmail.com]

Thank you for the response.

OS: WS 2016 (both machines)

RabbitMQ: 3.7.12

Erlang: 21.2

[Amari CH]

Hello

It seems like you were using the version too old.
I would suggest you to upgrade them to the latest.

RabbitMQ version: 3.8.8
Erlang configuration: Erlang/OTP 23 [erts-11.0.3]

Thus you can get better performance in recent windows OS.

hope this helps.

[Simone Musetti]

Thank you for your reply. I'll evaluate the upgrade. I just don't get how this can be related to performance issues. VMs are almost idle, they only host the RabbitMQ instance.

I found this issue already on these versions:

- 3.6.6, 3.7.12

--
You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/NEwpGe1e1AQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f858cce0-4b7d-4c52-16ff-251cdfcd9aaa%40maddoghost.com.