Poor C# BasicPublish Performance?

[Brian Kinder]

Hi All,

I've been doing some fairly heavy performance tests with rabbit to see if it can suit my needs as a messaging layer to replace an existing system, and after some testing I'm hitting some apparent latency on the sending side that I'm trying to track down. I'm using many separate publisher/consumer applications in C# to emulate load from our application, accross about 65 queues. As part of their logging, the senders track how long it takes the IModel.BasicPublish call to happen and I've logged fairly frequent spikes of > 100ms.

Any idea what could cause it to get hung up like that? subsequent messages will go through fine, and there doesn't seem to be any pattern as to when these spikes occur. Is BasicPublish asynchronous under the hood?

Testing plan:

Here's my plan setup running at roughly 27,000 messages/second. I should note that overall rabbitmq is doing a fairly solid job keeping up with overall throughput, its just individual messages getting delayed that is the problem.

The Test Server (its late, pretty sure this is a cluster of 4 servers each with these specs ill get back and confirm on this tomorrow):

 -Ubuntu 14.04.1 16 core, 16 GB memory

 -Rabbitmq 3.6.12 with Erlang 19.3.6.2.

The Test Client Software:

  The test software consists of two C# programs using the rabbitmq client library. One program is a dedicated producer, the other a consumer. 

    The producer sends 1.5KB messages at a set rate/second after a warm-up period on a single thread. Each producer only sends to a single exchange

    The consumers subscribe to a single queue and process messages with the async consumer

    Both consumers and producers track their messages sent/received and log results to a database for easy perusal

The Test Client Systems:

10x Quad Core, 8GB memory systems

 -Each system is running 3-4 producers at 350 msgs/second to a topic exchange (C) with 26 routing keys to queues C_1-C_26 (6 clients send 1050/s, 4 clients send 700/s)

 -Each system consumes a single queue B_1-B_10

3x Quad Core, 8GB memory systems

 -Each system is running 9-10 producers at 115 msgs/s to a topic exchange (A), with 26 routing keys to queues A_1-A_26 (2 clients send 1150, one sends 1035, 115/s per route)

4x Hex-core, 16GB memory systems, 2 of these systems are producing a total of 3000msgs/sec (3x300+2x1200), while 2 are producing 4200 msgs/sec(2x300+3x1200)

 -Each system is running 2-3 producers at 300 msgs/second to a topic exchange (B), with 10 routing keys B_1-B_10

 -Each system is running 2-3 producers at 1200 msgs/second to a direct exchange (D), with one queue D_1

 -Each system is running 13 consumers. half of which are queues from A, half are queues from C (covering the entire A_1-A-26, and C_1-C-26)

1x Hex-core 16GB system that just consumes a total 3,000 messages/second sent by the 

 -This system is running 4 consumers that consume D_1

[Karl Nilsson]

Hi,

Are any of your connections in "flow" status? If they enter flow temporarily you could well notice that as publish latency.

https://www.rabbitmq.com/flow-control.html

Cheers

Karl

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Pivotal/RabbitMQ

[V Z]

No flow control or blocked. no resource alarms. All connections are green (running). There is no backup (queue depth is always zero per management UI), consumer utilization is at 100%, deliver rate matches publish rate one-for-one, messages are sent non-persistent, no publisher confirms, and auto ack on consumer end. Queue are not mirrored. It is a 4 node cluster. Access is via HAProxy.

[Karl Nilsson]

Maybe it is blocking on Socket.Send as it won't return until all bytes have been sent - you could try to inspect the socket's send queue to see if it also spikes.

On 21 December 2017 at 16:01, V Z <uvzu...@gmail.com> wrote:

No flow control or blocked. no resource alarms. All connections are green (running). There is no backup (queue depth is always zero per management UI), consumer utilization is at 100%, deliver rate matches publish rate one-for-one, messages are sent non-persistent, no publisher confirms, and auto ack on consumer end. Queue are not mirrored. It is a 4 node cluster. Access is via HAProxy.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Pivotal/RabbitMQ

[V Z]

Thanks, Karl. Could you please educate us on how to inspect the socket send queue?

[Luke Bakken]

Hi Brian -

On another message thread, I suggested using PerfTest to see if you could duplicate this latency (https://groups.google.com/d/topic/rabbitmq-users/EhPPy2zSXQA/discussion)

I don't know if you are the same person as "V Z", but if you are, did you give PerfTest a try? You should be able to configure it to mirror your test scenario exactly.

I suggest using the latest RC candidate - https://github.com/rabbitmq/rabbitmq-perf-test/releases/tag/v2.0.0.RC4

You can run producers and consumers on different machines using this version of PerfTest - just be sure that clocks are as synchronized as possible and that you use the --use-millis argument.

Thanks,

Luke

[Karl Nilsson]

On linux (and other unixes) netstat will give you this info (Send-Q). On windows it seems more difficult! Are you using windows for this test?

On 21 December 2017 at 17:05, V Z <uvzu...@gmail.com> wrote:

Thanks, Karl. Could you please educate us on how to inspect the socket send queue?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Pivotal/RabbitMQ

[Brian Kinder]

Yes, all our senders and receivers are running from windows server 2012

On Friday, December 22, 2017 at 2:19:28 AM UTC-6, Karl Nilsson wrote:

On linux (and other unixes) netstat will give you this info (Send-Q). On windows it seems more difficult! Are you using windows for this test?

On 21 December 2017 at 17:05, V Z <uvzu...@gmail.com> wrote:

Thanks, Karl. Could you please educate us on how to inspect the socket send queue?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Karl Nilsson

Pivotal/RabbitMQ

[Michael Klishin]

Please use PerfTest to establish your baseline to use a point of reference.

We recently introduced a way to batch N messages into a single socket write:
https://github.com/rabbitmq/rabbitmq-dotnet-client/pull/368

Basic benchmarks suggest that it makes a noticeable difference. Not sure if there's a .NET client milestone

release that includes it but we can definitely publish one for others to try.

Of course, application-level batching and compression are options available to any version of any client.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

We didn't get to try PerfTest because we distributed producers across more machines, and the latency got lower to the point that satisfies our needs. Therefore, it seems like the latency was attributed to out-of-capacity on the producers end. Thanks for your help.