Publish Performance using STOMP

[Perry Taylor]

I have implemented a client application using the STOMP protocol. It seems to work fairly well but I am struggling with an issue when publishing messages.                                                                                                                                                                        

When consuming messages STOMP requires that an ACK/NACK be sent to RabbitMQ to confirm receipt of the consumed message. This works quite well with acceptable performance.                                                                                                                                                                 

When it comes to publishing, STOMP does not provide any kind of ACK from RabbitMQ to confirm a message was successfully processed. The protocol specifies that RabbitMQ is required to send an ERROR frame if the operation was unsuccessful. This requires the client to actively listen for a possible ERROR frame. The question becomes how long to wait with no ERROR frame received before assuming an ERROR frame will not be coming. This has a huge impact on performance.

                                                                  

Is this a shortcoming of the STOMP protocol or is does this come from the AMPQ layer? How is this best approached?                                                                                      

Thanks.

Perry Taylor

[Luke Bakken]

Hi Perry,

I believe the "receipt" feature is what you're looking for - http://stomp.github.io/stomp-specification-1.2.html#RECEIPT

Use it in combination with persistent:true - https://www.rabbitmq.com/stomp.html#pear.mp

Thanks,

Luke

[Perry Taylor]

Thank you for that Luke.

We were already making the message persistent and have now implemented the receipt feature.  Seems to be working reliably but publish performance is very poor.  Is this to be expected or are other things that can be done to improve this?

Thanks.

Perry

[Michael Klishin]

Error communication to clients is different from protocol to protocol. In every protocol RabbitMQ

supports except for MQTT 3.1 (which doesn't support server-to-client error communication in most cases)

this communication is asynchronous in most cases (certainly for publishing).

What kind of timeout is used is up to you but timeouts < 5 seconds is generally

a bad idea because of the high probability of false positives.

How poor is "very poor"? What doees your benchmarking code look like? Are you sure your app is not blocking to wait for an ERROR frame? If so

that would ruin throughput and it is not something that RabbitMQ's or the STOMP plugin or the STOMP client used can possibly solve.

When it comes to publishing, RabbitMQ STOMP plugin builds on top of Publisher Confirms [3] which uses asynchronous acknowledgement delivery.

You can always use [2] for baseline performance. It is not a STOMP-based tool but it can be used to simulate a lot of workloads.

Lastly, single queue benchmarks are highly discouraged as RabbitMQ was not designed for the Single Giant Queue workload (we consider it to be an anti-pattern

for this reason and others). See [1].

1. https://www.rabbitmq.com/queues.html#runtime-characteristics

2. https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/

3. https://www.rabbitmq.com/confirms.html

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Perry Taylor]

Hello Michael..

Thank you for that information.  It was very informative.

We have been using Websphere MQSeries for many years.  We're seeing both publish and consume rates approaching 1,000 messages/second.  For RabbitMQ publish rates are reasonable.  This includes the client sending an ACK.  Publish rates are abysmal at about 200 recs/sec using RECEIPT acknowledgements.  In both brokers we are persisting to disk.

Putting messages in MQSeries is a synchronous thing yet we don't see the poor performance there.  I'm just struggling with how we can complete a unit of work, the successful processing of which includes sending a message to the broker, without having confirmation the message was successfully processed.  Perhaps our requirements are just not suited to RabbitMQ.

Thanks.

Perry

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Perry Taylor]

Correction:  Rather than..

"For RabbitMQ publish rates are reasonable"

I should have said...

"For RabbitMQ consume rates are reasonable"

My apologies.

Perry

[Michael Klishin]

I don't see why STOMP wouldn't be able to do a few thousands of messages per second, with acknowledgements or not.

Can you please share a little bit more about how you consume (or publish) and what the workload looks like?

Acknowledging batches from consumer end is going to be more efficient, not less, with any protocol. For publishers,

it's entirely client library specific as both publishing and acknowledgement flow are asynchronous, so unless you *choose* to block,

client library won't wait for a group of confirmations (in fact, it cannot know where the batch ends).

Lastly, you can use PerfTest (which won't use STOMP) to establish a baseline [1]. Very likely you will see at least 20K+ messages/second

with the most basic settings.

1. https://rabbitmq.github.io/rabbitmq-perf-test/stable/htmlsingle/

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3322b558-99fb-4edd-a946-991d51bd4e21%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

I double checked the STOMP spec [1] and there is nothing that mandates that RECEIPT frames should not be sent asynchronously. So they

are in RabbitMQ. I'm afraid I cannot suggest much more without seeing your code but it's certainly either the client you use or how you use it

that chokes the throughput.

1. http://stomp.github.io/stomp-specification-1.2.html#RECEIPT

[Perry Taylor]

I'm experimenting with two different prototype clients.  One is using STOMP from a language I'm pretty sure you would not know, UniVerse BASIC. It would be a lot of code and I can post it if you want to wade through it.  The other client prototype is in python using pika.  I'm seeing only slightly better performance there where I'm using a blocking connection with delivery_mode = 1, mandatory = True. I will attach that as its not very big.  Please forgive my amateurish python as I am new to it and learning as I go.

Thanks.

Perry

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/3322b558-99fb-4edd-a946-991d51bd4e21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Perry Taylor]

Here are some example results.

The queue ptq is durable, using the default exchange, no replication. Both clients are using the same data source for the messages.

UniVerse BASIC Client using STOMP using receipts on publish, ack: client-individual on consume:

Source File :  CLM.INSTANCE

Record stats:  count = 100,000 min = 21 max = 23 avg = 23

Broker:  RABBITMQ

Connecting to pttest on loudvmq001b.zcloud.com(61613) as sadvdoprabbitmqtest ...

CONNECTED, et = 0.083974

Sending 100,000 CLM.INSTANCE records to queue ptq ...

ET: 361.167181, rec/sec: 276.9

Subscribing to queue ptq ...

SUBSCRIBED, et = 0.00012

Retrieving messages from queue ptq...

recs: 100,000, et: 170.151906, rec/sec: 587.7

Unsubscribing from queue ptq ...

UNSUBSCRIBED, et = 0.060325

Disconnecting from pttest on loudvmq001b.zcloud.com(61613) ...

DISCONNECTED, et = 0.001189

And here are examples from my python publish-only client:

First with mandatory = False:

filespec   : PTEST.CLM.INSTANCE

hostname   : loudvmq001b.zcloud.com

vhost      : pttest

exchange   : 

queue      : ptq

Mandatory  : False

source file: CLM.INSTANCE

records    : 100,000

min length : 21

max length : 23

avg length : 22.57549

et         : 189.81155228614807

recs/sec   : 526.8383235665564

And with mandatory = True:

filespec   : PTEST.CLM.INSTANCE

hostname   : loudvmq001b.zcloud.com

vhost      : pttest

exchange   : 

queue      : ptq

Mandatory  : True

source file: CLM.INSTANCE

records    : 100,000

min length : 21

max length : 23

avg length : 22.57549

et         : 203.69526314735413

recs/sec   : 490.9294327952021

As you can see there is a pretty significant difference between STOMP with reciepts and pika using mandatory.

Thoughts?

Next,  I'm going to download and try out the perf-test to get some baselines.

Thanks.

Perry

[Luke Bakken]

Hi Perry,

Your Python code is waiting for a confirmation for each message, sequentially. This is guaranteed to limit publishing rate.

The correct method is to publish and retrieve confirmations asynchronously, which is not possible with BlockingConnection. You will have to either modify your Python code, or use PerfTest, which does support this pattern. For instance, the following PerfTest arguments will publish 10000 persistent messages with a maximum of 10 outstanding confirms:

--flag persistent --pmessages 10000 --producers 1 --consumers 0 --uri amqp://MYHOST:5672 --confirm 10

Thanks,

Luke

[Perry Taylor]

Okay finally got firewall rules in place so I can run PerfTest.

I can telnet to the port...

$ telnet loudvmq001a.zcloud.com 5672                                                                          

Trying 10.11.10.157...

Connected to loudvmq001a.zcloud.com.

Escape character is '^]'.

^]

telnet> quit

Connection closed.

But when I try to run PerfTest I get...

$ perf-test_linux_x86_64 --flag peristent --pmessages=1000 --producers 1 --consumers 0 --confirm 10 --uri amqp://loudvmq001a.zcloud.com/:5672

Main thread caught exception: java.net.SocketException: Address family not supported by protocol

[main] ERROR com.rabbitmq.perf.PerfTest - Main thread caught exception

java.net.SocketException: Address family not supported by protocol

What am I doing wrong?

Thanks.

Perry

[Luke Bakken]

Hi Perry,

You have a small error in your URI. It should be this:

--uri amqp://loudvmq001a.zcloud.com:5672

[Perry Taylor]

That doesn't seem to fix the problem...

$ perf-test_linux_x86_64 --flag peristent --pmessages=1000 --producers 1 --consumers 0 --confirm 10 --uri amqp://loudvmq001a.zcloud.com:5672                                                                                                     

Main thread caught exception: java.net.SocketException: Address family not supported by protocol

[main] ERROR com.rabbitmq.perf.PerfTest - Main thread caught exception

java.net.SocketException: Address family not supported by protocol

        at java.net.Socket.createImpl(Socket.java:460)

        at java.net.Socket.getImpl(Socket.java:520)

        at java.net.Socket.setTcpNoDelay(Socket.java:980)

        at com.rabbitmq.client.SocketConfigurators.lambda$static$0(SocketConfigurators.java:34)

        at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:59)

        at com.rabbitmq.client.impl.recovery.RecoveryAwareAMQConnectionFactory.newConnection(RecoveryAwareAMQConnectionFactory.java:62)

        at com.rabbitmq.client.impl.recovery.AutorecoveringConnection.init(AutorecoveringConnection.java:156)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1106)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1063)

        at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:1198)

        at com.rabbitmq.perf.MulticastSet.run(MulticastSet.java:115)

        at com.rabbitmq.perf.PerfTest.main(PerfTest.java:271)

        at com.rabbitmq.perf.PerfTest.main(PerfTest.java:369)

        at com.rabbitmq.perf.NativePerfTest.main(NativePerfTest.java:26)

What am I doing wrong?

Thanks.

Perry

[Luke Bakken]

Hi Perry,

I'm not really sure what's the issue in your environment. If I run PerfTest from a clone of the rabbitmq-perf-test repo like this, it works fine:

make ARGS='-flag peristent --pmessages=1000 --producers 1 --consumers 0 --confirm 10 --uri amqp://localhost:5672' run

I did a search on the exception, maybe you're running into this issue:

https://stackoverflow.com/questions/16373906/address-family-not-supported-by-protocol-family-socketexception-on-a-specific

Thanks,

Luke

[Perry Taylor]

Yeah its very odd.  I'l see what else I can come up with.  Thanks Luke.

-Perry

[Perry Taylor]

Well it seems that this problem is specific to the pre-compiled binary I downloaded.  I switched to using $ bin/runjava com.rabbitmq.perf.PerfTest ....   and the problem is gone.

I've been able to run some initial tests including the one suggested here.  I need to try to simulate what we would be using in terms of queue properties such as queue-mode=lazy which I am able to specify with --queue-args.  How do I get it to create a queue that is mirrored?  Is this possible?

Thanks.

Perry

[Luke Bakken]

Hi Perry,

PerfTest supports the --predeclared and --queue arguments to use a queue you have declared already (via the UI, for instance). You can use a policy to configure queues matching a certain regex to be lazy and to be mirrored.

Thanks -

Luke

[Arnaud Cogoluègnes]

PerfTest native binary is statically built and then contains just a
subset of the JDK. The error you get maybe due to missing ipv6 support
in the binary, but this needs more investigation.

The workaround is to switch to the usual distribution and run it with
a JVM installed.

The native binary is still considered experimental and can break with
configurations we have not tested yet, so thanks for reporting this.

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e19accb1-1103-4697-959d-7068eed992cf%40googlegroups.com.

[Arnaud Cogoluègnes]

Small typo: "--flag peristent" => "--flag persistent"

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

> To post to this group, send email to rabbitm...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/e19accb1-1103-4697-959d-7068eed992cf%40googlegroups.com.

[Perry Taylor]

Hey Luke...

I saw the predeclared option but was thinking it was something different.  That's exactly what I needed.  Thank you!

Perry

[Perry Taylor]

Thanks Arnaud!

-Perry

> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.