Trying to understand messaging latency

[V Z]

We ran a very simple test: best effort publishing (no confirms), no persistence, auto ack, prefetch=1000 (Rabbit 3.6.12, Elang 19.3.6.2). 

For the most part we see 3.5 ms between the publish and consume (different processes on the same machine). Minimum is about 2.5 ms. However, sometimes, this difference is up to 100 ms.

Is there a way to understand why the latency spikes sometimes? It may not be in Rabbit; could be in the app, but if it's in Rabbit, what to look for? Thanks

[Michael Klishin]

It could be VM context switching. I recall a couple of threads in the last year or so about scheduler-to-core binding making a difference

for some.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

Here is what I cannot wrap my head around -- 

The producer keeps publishing non-stop (1,000/sec in our case). The consumer keeps consuming. As observed via management UI, there is no queuing, consumer utilization is at 100%, no disk I/O, no flow control or blocked connections.

If there was some kind of pause in the broker (say, for 100 ms), wouldn't all messages from that point on be delayed? In our case only few out of 100,000 published have this latency, and only 100 have latency of about 20 ms. How can one message be delayed, and the message arriving after it not? Does this pattern (of some messages being delayed) suggest that the latency is introduced by the producer, or am I jumping to conclusion?

Would top plug-in help with understanding any latencies? If so, what would we be looking for? Thanks

[Luke Bakken]

Hi VZ -

If you'd like to try out a different settings based on what Michael suggested, add the following lines to /etc/rabbitmq/rabbitmq-env.conf (then restart):

RABBITMQ_SCHEDULER_BIND_TYPE=u
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS='+sbwt very_short'

This will allow the operating system to decide which CPU cores the scheduler threads are bound to, and will set the schedulers to busy wait for a shorter-than-normal time if they don't have work. It may have an effect, and it may not (http://erlang.org/doc/man/erl.html#+sbt)

There are quite a few other Erlang VM arguments listed on that page that you could try out.

You should also give our PerfTest application a try to see if you see the same latency bumps: https://github.com/rabbitmq/rabbitmq-perf-test

I would be interested to know what PerfTest shows in your environment.

Thanks,

Luke

[V Z]

Thanks, Luke, but, unfortunately, our Rabbit runs in PCF, so modifying the start-up script may be problematic.

I can try PerfTest. Does it time stamp every message, calculate difference between consumption and publication, then plots/prints latency distributions?

I wonder, though, if my reasoning is on point that if one message gets delayed, all messages after it would be delayed by at least as much assuming steady arrival?

[Luke Bakken]

> our Rabbit runs in PCF

Are you hosting your own PCF? Just curious.

PerfTest does exactly that to calculate latency - that's why I suggested it!

As for one message having higher latency I think that is probably due to how your consumer is calculating latency. If you can provide the code to your producer and consumer maybe I can reproduce what you are seeing.

If you are using one queue, it makes sense that a RabbitMQ-internal delay in one message might be reflected in subsequent messages. But, I am not nearly as familiar with RabbitMQ internals as the rest of my team so I hope one of them weighs in on that.

Thanks,

Luke

[V Z]

Yes, we have PCF on premise. I will try with PerfTest. I did not know it had that feature. Thanks