Did I squeeze the last drop out of Rabbit?

[V Z]

I have a rather standard, nothing special, as far as I am concerned, 2 core (Intel 6 2.6GHz) Linux VM, plenty of RAM. I can publish and consume at about 40,000 100-byte messages per second with no confirms and no acks (Rabbit 3.6.2, Erlang 18.2.1).

While this is happening 'top' shows 99% CPU utilization with rabbit process at 199% CPU (99% x 2 cores?). All publishers' connections and channels are in flow control (yellow square). If I disable stats via rate_mode=none, I could get 2,000 more messages per second, but then I don't see what's going on in broker. Other than that, I didn't change any Rabbit or Erlang defaults.

Is that it, or can something be tweaked to get more throughput?

[V Z]

There is no message build up during the test, i.e. consumer is keeping up.

[Michael Klishin]

* Use a CPU with more cores

* Use more connections, channels, queues (which will actually use the cores above)

* Use more nodes [1]

* If you can trade off message ordering for throughput, you can try rabbitmq-sharding

* Enable HiPE (a JIT compiler in the VM)

* No shortage of different VM flags to try, e.g. scheduler-to-core binding [2]

Note that we know no specifics of your test except for the number of cores. With 2 cores, 1 node, 2 connections

and 1 queue (which is the most common benchmarking scenario we see, so I'll assume that) there is less room

for tweaking. The above is as much as can be recommended without knowing more about the workload.

1. https://cloudplatform.googleblog.com/2014/06/rabbitmq-on-google-compute-engine.html

2. https://github.com/rabbitmq/rabbitmq-server/issues/612

On Sat, Jun 18, 2016 at 4:15 AM, V Z <uvzu...@gmail.com> wrote:

There is no message build up during the test, i.e. consumer is keeping up.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

Thanks. It is not that I was complaining; just wondering how much I can squeeze out of that.particular VM as a benchmark. I understand that throughput can be increased by adding more resources. I will look into HiPE.

I forgot to mention that I was publishing to a fanout exchange with single queue bound to it.

When I ran 2 producers from 2 different machines, each publishing to its own fanout exchange with own queue, so 2 queues total, message rate got split -- roughly 20,000 messages per second per connection, i.e. total throughput did not change, and both connections were in flow control. When I made second producer publish concurrently on 2 channels, message rate got split further -- 20,000 msgs/sec for the first producer publishing on a single channel, and 10,000 msgs/sec for each of the two channels used by the second producer. So, again, the same 40,000 msgs/sec total throughput, and all three channels in flow control.

Does this sound normal? I just want to make sure I did not mess something up in my Java client.

[Michael Klishin]

On a 2 core machine this is likely an effect of the (Erlang VM) scheduler.

Every channel in RabbitMQ is an Erlang process (a very lightweight thread if you will),

and so is every queue. Each of them gets some execution time from the runtime which only has 2 schedulers (things that are

assigned to execute processes) on a 2 core machine by default.

The reason why all 3 channels go into flow control is this: there's still one queue, which can only use a single core on the hot

code path. So when it gets to its maximum throughput, it starts throttling all channels using an internal credit flow mechanism.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[V Z]

There were actually 2 queues -- one producer publishing to one queue via one fanout exchange over one connection, and the other producer publishing to another queue via a different fanout exchange over one or two connections on one or two threads. In all cases throughput was diluted across all channels.

There was a post on this forum (https://groups.google.com/forum/m/#!starred/rabbitmq-users/tO8vaRp7bCA) that suggested that publishing over several channels was alleviating flow control. I don't understand how that could be because in my case I also publish over multiple channels, and in fact over multiple connections from multiple machines, unless the client in that post was getting bogged down and slowing down, so not improving throughput...