Rabbitmq latency

[Tal Ben Shabtay]

Hi all,

i have started using rabbitmq as my main message bus solution,

overall i really like rabbitmq , ez installation and java api and lots of features.

however , i have been messing around a bit with the latency to see if rabbitmq can cause any issues as my application is latency sensitive (0-5 ms is fine but from there it starts to be messy).

so as far as my understanding goes , rabbitmq can support to up to 5-7k messages per second throughput, but i can see in my solution

that i have between 20-50ms latency on a 400 messages per second throughput thru two servers on a lan in AWS.

my machine configuration -

producer - 4 cores , 8 gb (EC2) which includes also the rabbitmq installation
consumer - 16 core , 30gb (EC2)
im using the java api with qos of (0,0,true) for each channel and 5 threads on a fixed thread pool per connection and auto ack.

rabbitmq configuration :

durable queues, exchanges.
messages are non persistant.
12 fanout exchanges , each with about 3 queues , each with 3 bindings routing keys , channel per binding.

cat /etc/rabbitmq/rabbitmq-env.conf

export RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="+A 128"

cat /etc/rabbitmq/rabbitmq.config

[
        {kernel,[
                        {inet_default_connect_options, [{nodelay, true}]},
                        {inet_default_listen_options,  [{nodelay, true}]}
                ]
        },
        {rabbit,[
                        {hipe_compile,true},
                        {tcp_listen_options, [
                                                {backlog,   128},
                                                {nodelay,   true},
                                                {sndbuf,    196608},
                                                {recbuf,    196608}
                                             ]
                        }
                ]
        }


].

can you please share with me your settings and your latency results ?

something just doesnt make sense , it cant be that 20ms is lost thru rabbitmq with basic fanout exchange.

also , i have seen on the web people talk about batch ack that can improve performance , can you please share how to accomplish this ?

or if possible for certain queues / messages i dont need ack at all , just fire and forget.

thanks.

[Michael Klishin]

See the Erlang VM scheduler flags. They could have significant effects in virtualized environments.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kyle O'Donnell]

One of our developers wrote a patch for the way GC was handled, which was causing HUGE latency spikes (every 60 seconds)

Here is our config. The framemax should be lower but one of our apps needs this...sadly.

[
{kernel,
[{inet_dist_listen_min, 10100},
{inet_dist_listen_max, 65000},

{inet_default_connect_options, [{nodelay, true}]},
{inet_default_listen_options, [{nodelay, true}]}
]
},
{rabbit, [
{hipe_compile,true},

{tcp_listen_options, [binary,
{sndbuf, 4096},
{recbuf, 4096},
{buffer, 4096},
{packet, raw},
{reuseaddr, true},

{backlog, 128},
{nodelay, true},

{exit_on_close, false}]},
{loopback_users, []},
{background_gc_enabled, false}, <patch
{vm_memory_high_watermark_paging_ratio, 0.99},
{vm_memory_high_watermark, 0.6},
{frame_max, 524288}
]
}
].

This is specific to our hardware, specifically the +S and +sct values, everything else has helped a great deal with latency. We're sub 1ms latency..not sure how many msgs/sec probably ~10k.

(Intel(R) Xeon(R) CPU E5-2698 v3 @ 2.30GHz)

RABBITMQ_SERVER_ERL_ARGS="\
+K true +A5 +P 1048576 -smp enable +S 32:32 \
-kernel inet_default_connect_options [{nodelay,true}] \
+sct L0-7,16-23c0-7,16-23p0N0:L8-15,24-31c8-15,24-31p1N1 \
+sbt db +swt low +sbwt long"

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send an email to rabbitm...@googlegroups.com.

[Michael Klishin]

Note that you can simply override that variable instead of patching the file. Plus 98.9% of people should not worry about max frame size.

[Tal Ben Shabtay]

thanks Kyle and Michael ,

i have read about the configuration flags Kyle has shared, and except the +sct which is specific to he's system , 

i dont see any changes.

i did update my rabbitmq rpm to 3.6.1.1 (latest) and erlang was at R14 03 , i updated it to 18.2 

can it be that my java api usage is causing this ?

Producer

public void send(AbstractMessage msg) {
String routingKey = msg.getRoutingKey().getRoutingKeyString();

String msgJson = gson.toJson(msg);

if (!this.connection.isOpen()) {

logger.debug("Connection has died. creating new connection");

this.connection = buildConnection();

}

if (!this.channel.isOpen()) {

this.logger.debug("Channel is closed. creating new channel.");

this.channel = this.connection.createChannel();

this.channel.exchangeDeclare(this.exchange, "topic", true);

}

this.channel.basicPublish(this.exchange, routingKey, MessageProperties.TEXT_PLAIN, msgJson.getBytes());
}

public Connection buildConnection() {

return factory.newConnection(Executors.newFixedThreadPool(3), addressArr);

}

Consumer :

public void listen(String routingKey) {
                Connection connection = buildConnection();

                Channel channel = this.connection.createChannel();

channel.exchangeDeclare(this.exchange, "topic", true);

channel.queueDeclare(this.queueName, false, false, true, null);

                channel.basicQos(0, 0, true);

channel.queueBind(this.queueName, this.exchange, routingKey);

channel.basicConsume(this.queueName, true, buildConsumer());

}

private DefaultConsumer buildConsumer(Channel channel) {

return new DefaultConsumer(channel) {

@Override

public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] body)

throws IOException {

String msgJson = new String(body);

AbstractMessage absMsg = gson.fromJson(msgJson, AbstractMessage.class);

handler.handleMessage(absMsg);

}

};

}

[Michael Klishin]

You open new connections and channels, make sure they are not being opened more frequently than you think they are. Tracing new connections is easy in the log, for channels you'd have to inspect protocol traffic with Wireshark:

https://www.rabbitmq.com/amqp-wireshark.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Tal Ben Shabtay]

are you referring to the producer or consumer ?

in any case i dont see new connections being created on the rabbitmq web management ... 

i know the code here doesnt show it here but my topology is for the consumer connection per queue , channel per binding.

where for the producer is connection + single channel per producer.

something interesting i have noticed just now , upgrading my vm from 4 cores to 8 cores dropped the latency to 2-5 ms. while the machine wasnt even on high load while on 4 cores.

thats weird , load average was about 3.5 (which is fine on 4 core machine) , cpu load was at about 30% each core .

on 8 core, i can load average at 1.7 and cpu load is the same.

thoughts ?

[Michael Klishin]

Does your client run on the same host? Going from 4 to 8 cores halves the LA at the same CPU load… this is expected with a system that uses enough channels and queues (each of which can only use a single core).

[Tal Ben Shabtay]

well , the way i measured latency is to add timestamp to my message before sending and take another time stamp when receiving.

so at first i run the following test - 

producer + rabbit on one machine (4 cores at the start then switched to 8) and consumer on another machine (consumer was 8 cores) 

but i started getting doubt about this test as the clocks might not be in sync and thats whats probably causing the latency

so then i started testing also with a producer+consumer+rabbitmq on the same machine (at first it was 4 cores then switched to 8) .

[Michael Klishin]

Then you have at least 2 cores used by the clients, which leaves 2 to RabbitMQ.

[Tal Ben Shabtay]

even so the LA was fine for that amount of cores , but never mind that we got a bit side tracked..

i think i have made a problem with my testing, i included the json parsing in the measurements, 

after revisiting the test parameters for sending just a timestamp string , every msg is between 0-2ms . i can live with it.

so i think ill change my question to , 

what is the recommended way to seralize / deseralize objects using rabbitmq ? 

currently im using gson which causes latency spikes (jackson too) .

[Michael Klishin]

Wireshark can be a good tool to see latency without application processing steps (which is only worth so much but can be useful to validate assumptions).

RabbitMQ treats all messages as opaque blobs of bytes, so anything you please. JSON is hardly an efficiency champion: try MsgPack or Kryo if you are on the JVM.