RabbitMQ 3.12.2 Federation Throughput/Rate Slow

[Jeff S]

Hi.

Using PerfTest (latest stable version) from a separate beefy 16 CPU box pushing to a single RabbitMQ 3.12.2 node (no cluster), classic queues, able to get about 35-50K messages/sec.

I then, enabled federation to another beefy (16 cpu, 32 gb ram, ssd) node and set up 1500 consumers to only consume via PerfTest and I see federation only pushing max of 8.5-9K messages per sec. So the upstream is able to receive 35-50K messages/sec but then its fedearion is only pushing 8.5K msg/sec.

99th percentile Latency between the 2 (Single, non clustered ) nodes is .3 ms. No network issues or saturation. The problem is purely within the RabbitMQ ecosystem. The closest thread came from Jack Vanlightly (who since left the RabbitMQ team) but I'm seeing the same with 3.12.2. Installed in various environments and the throughput is consistently the same from federation. Please help. Here is the 1st perftest command I run. I also have another on the downstream with publishers set to 0 and consumers set to 1500 (on a completely different beefy server. Both RabbitMQ nodes do not have anything else installed besided prometheus, management and federation plugins.

sudo java -jar perf-test.jar --uri amqp://some_server_dns --metrics-prometheus-port 8080 --queue-pattern 'perf-test-%d' --queue-pattern-from 1 --queue-pattern-to 500 --producers 500 --consumers 1500  -u "throughput-test-1" --id "test-1" -f persistent -q 40 --rate 1000 -z 180 --use-millis

[Michal Kuratczyk]

You might be running into this issue: https://github.com/rabbitmq/rabbitmq-server/pull/9150

We had an attempted fix but it had its own issues and I believe we currently don't have an alternative.

You can check if that your issue with either:

* rabbitmq-diagnostics consume_event_stream (you'll see a lot of consumer.created/consumer.deleted events)

* or tcpdump/wireshark (you'll see a lot of Basic.Consume/Basic.Cancel commands)

A larger prefetch on the downstream might be a sufficient workaround.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/297210e1-b993-46b3-be8b-eb1ad644123bn%40googlegroups.com.

--

Michał

RabbitMQ team