Is related between C# RMQ client exception "Pipelining of requests forbidden" and RabbitMQ server warning "busy_dist_port"

[Xiaoyan Song]

Our C# 5.1.0 client throw many exception " Pipelining of requests forbidden" as below.

I noticed rabbitMQ log shows some warning of  "rabbit_sysmon_handler busy_dist_port <0.5054.0> [{initial_call,{gm,init,1}},{erts_internal,dsend_continue_trap,1},{message_queue_len,0}]" and Grafana shows some data buffered in distribution links queue as attached pic.

Can someone tell me if "Pipelining of requests forbidden" related to  "busy_dist_port"?

Or just caused by channel thread safe issue.

[System.NotSupportedException]
Pipelining of requests forbidden
StackTrace:
    Core.Library.RabbitMQ.Receiver.ListenToQueue    D:\cd011a\_w\1513\s\library\Core.Library.RabbitMQ\Receiver.cs [71]
    Core.Library.RabbitMQ.Receiver.ConfigureChannel    D:\cd011a\_w\1513\s\library\Core.Library.RabbitMQ\Receiver.cs [243]
    RabbitMQ.Client.Framing.Impl.Model.BasicQos
    RabbitMQ.Client.Impl.ModelBase.ModelRpc
    RabbitMQ.Client.Impl.ModelBase.Enqueue
    RabbitMQ.Client.Impl.RpcContinuationQueue.Enqueue

Thanks a lot

xiaoyan

[Luke Bakken]

Hello,

That's interesting. They might be related. Have you confirmed that the timestamps for both events is at about the same time?

More than likely your C# code is sharing the IModel instance across threads.

Thanks -

Luke

[Xiaoyan Song]

Luker, 

thanks a lot for your reply! 

I find one of the patterns:

The "[System.TimeoutException] The operation has timed out" is more related to  Grafana "data buffered in distribution links queue" metrics.

After got "System.TimeoutException", then, client got several exceptions of "[System.NotSupportedException] Pipelining of requests forbidden".

Currently, I have set "RABBITMQ_DISTRIBUTION_BUFFER_SIZE=1048576" 1GB in rabbitmq-env.conf. (Rabbitmq 3.11.2 + Erlang 25.1.2.)

[System.TimeoutException] The operation has timed out. StackTrace: 

Core.Library.RabbitMQ.Receiver.ListenToQueue D:\cd011a\_w\910\s\library\Core.Library.RabbitMQ\Receiver.cs [62]

Core.Library.RabbitMQ.Receiver.DeclareQueue D:\cd011a\_w\910\s\library\Core.Library.RabbitMQ\Receiver.cs [230]

RabbitMQ.Client.Impl.AutorecoveringModel.QueueDeclare 

RabbitMQ.Client.Impl.ModelBase.QueueDeclare 

RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply 

RabbitMQ.Util.BlockingCell.GetValue

In addition, I still review my code about Imodel thread safe issue. 

I add lock whenever access Imodel insance in my code for thread safe now. 

Not sure if C#SDK auto recovery mechanism maybe conflict with my lock? 

After extracted code pattern, I could share here. 

BR

xiaoyan

[Xiaoyan Song]

 Luker,

I attached 3 pictures about client/server log and grafana buffer metrics. (this server rmq 3.8.3, erlang 22.3, RABBITMQ_DISTRIBUTION_BUFFER_SIZE=256MB), 

One question:

How to decide the value of RABBITMQ_DISTRIBUTION_BUFFER_SIZE?

What is the proper reference metric for it?

Thanks in advance for your help.
BR

[Michal Kuratczyk]

Hi,

RabbitMQ 3.8 is out of support - upgrade ASAP.

Distribution buffer is an Erlang setting https://www.erlang.org/doc/man/erl.html:

A larger buffer limit allows processes to buffer more outgoing messages over the distribution. When the buffer limit has been reached, sending processes will be suspended until the buffer size has shrunk. The buffer limit is per distribution channel. A higher limit gives lower latency and higher throughput at the expense of higher memory use.

If you see dist_port_busy logs, then you can definitely try a higher value.

You can also look for why it is used so much. Most common reasons are:

* mirrored classic queues (which are deprecated!)

* using the Management API (many requests collecting data about many queues, etc)

Best,

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/bb195ff9-2b10-48ee-a299-a4a6a8b1015fn%40googlegroups.com.

--

Michał

RabbitMQ team

[Xiaoyan Song]

Michal,

Thanks for your reply! 

Could u pls instruct more about 

1. is "Pipelining of requests forbidden" related to "busy_dist_port" and more internal details are appreciated.

2. If set dist_buf_busy_limit to 2GB, how to measure latency impact? Which metrics can indicate this?

Based on https://www.erlang.org/doc/man/erl.html#+zdbbl, the distribution buffer busy limit (dist_buf_busy_limit) in kilobytes. Valid range is 1-2097151. Defaults to 1024.

The max value is 2GB. Currently I set to 1GB, then will try 2GB. 

Currently, we have below queues. and we have disabled queue metrics "sudo rabbitmqctl eval 'application:set_env(rabbitmq_prometheus, return_per_object_metrics, false).".

All 3159 queues are mirrored classic queues now, ha=all (3nodes). (To be upgrade to 3.11.2 with qurom queue next year). 

Connections: 3284
Channels: 4761
Exchanges: 38
Queues: 3161
Consumers: 3834 

BR

xiaoyan

[Michal Kuratczyk]

Hi,

Mirrored queues are almost certainly what causes extensive usage of distribution links (the algorithm used for mirroring is quite ineffective, which is one of the reasons why we developed quorum queues).

Erlang distribution (and its buffers) is used for all operations/communication inside a RabbitMQ cluster. Therefore measuring publisher confirm latency for a replicated queue is an example of something you can easily measure (https://perftest.rabbitmq.com/ is your friend) with different buffer settings, since publishing to such a queue requires internal cluster communication. You can also use https://grafana.com/grafana/dashboards/11352-erlang-distribution/ to monitor your distribution links.

I'm not familiar with "pipelining is forbidden", so I can't answer this part.

Best,

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/7f7b82e7-3651-4c0c-ac9a-89d2d8fccaffn%40googlegroups.com.

--

Michał

RabbitMQ team

[Luke Bakken]

Hello,

Yes, please share a complete set of code that is like yours via a git hosting provider like GitHub or GitLab.

I should be able to clone, compile and run the code. I can then review it.

How are you running the code? Windows Service, IIS, something else?

Thanks,

Luke