Issue with timeouts publishing to RabbitMQ

[Ryan Brown]

Hello,

We have written an application that publishes to rabbit using the headers exchange for routing to queues (approximately 1800 queues currently) and subscribes to these same queues to then deliver to our subscriber endpoints. It's essentially just a basic pub/sub rest interface to RabbitMQ. Recently, we have begun to see a significant ramp-up in volume and have experienced periodic failures where the error message indicates a timeout at the point we publish via the erlang rabbit client library. We are not doing confirmed publishing currently. However, we do wait for 'ok' return from rabbit to at least confirm rabbit has received the message.

While it appears to be related to volume, it does not seem that the volume we are seeing these failures at (~1000mps) should be causing issues with Rabbit. So, I wondering a couple of things.

1. Is there a way to determine if we are, indeed, overwhelming rabbit and need to scale-out? (Can we alert somehow if rabbit is blocking?)
2. Does the number of header tags effect the volume that RabbitMQ can handle? We are seeing an increase in the headers being used to apply more sophisticated routing for our clients. Curious if this could be somehow causing some of our issues.

Our current configuration is:

3 application servers (Erlang)

           ↓

1 F5 load-balancer

           ↓

4 RabbitMQ nodes clustered Active > Active > Active > Active

I'm sure I've left-out pertinent details. Feel free to ask. Any feedback is greatly appreciated.

[Michael Klishin]

On 14 October 2014 at 22:20:25, Ryan Brown (ryank...@gmail.com) wrote:
> However, we do wait for 'ok' return from rabbit to at least confirm
> rabbit has received the message.

Sorry but that doesn't confirm much if publisher confirms aren't used. 

> While it appears to be related to volume, it does not seem that
> the volume we are seeing these failures at (~1000mps) should
> be causing issues with Rabbit. So, I wondering a couple of things.
>
> Is there a way to determine if we are, indeed, overwhelming rabbit
> and need to scale-out? (Can we alert somehow if rabbit is blocking?)
> Does the number of header tags effect the volume that RabbitMQ
> can handle? We are seeing an increase in the headers being used
> to apply more sophisticated routing for our clients. Curious
> if this could be somehow causing some of our issues.

The headers exchange is by far the least efficient and optimised. Every "and" binding
can be easily translated to a topic exchange, for example, which are implemented
efficiently:

http://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/
http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/

I'd monitor channels using management UI to see if you may be running into flow control
often.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ryan Brown]

Thank you for the reply Michael. And thank you for the information on the headers vs. topic exchanges. I will research that for some future re-architecture work that is approaching. That doesn't, however, address my question of whether the inefficiency of the headers exchange could result in back-pressure from rabbit. (We are seeing flow control on ingress). One other note that may be of interest is that all of our queues are HA. I'm not sure if this could also cause some slowness as it writes-out the messages. Just a thought.

As for " Sorry but that doesn't confirm much if publisher confirms aren't used." I disagree only on one point. If we receive the ok, we can presume that Rabbit has at least received our message. (Or am I wrong in that presumption?) Otherwise, it would be no different than sending requests UDP and crossing our fingers.

Best.

-rb

--
-rb

[Michael Klishin]

On 14 October 2014 at 23:41:03, Ryan Brown (ryank...@gmail.com) wrote:
> That doesn't, however, address my question of whether the inefficiency
> of the headers exchange could result in back-pressure from rabbit.
> (We are seeing flow control on ingress).

It can.

> As for " Sorry but that doesn't confirm much if publisher confirms
> aren't used." I disagree only on one point. If we receive the ok,
> we can presume that Rabbit has at least received our message.
> (Or am I wrong in that presumption?)

Are you using direct client? Then possibly. With the network client, the data may
sit in OS buffers long enough for RabbitMQ to become unreachable.

[Ryan Brown]

Excellent Michael! First of all for the confirmation of my first suspicion. 

Secondly, we are using the network client. So that leads me to believe there is no real payoff to waiting for the ok. We do not want to use confirmed publishing as it just adds more overhead in publishing. However, this information raises some concern that possibly data could get lost if we don't do so. (although we have not actually seen this in practice with our current daily load of 17+M) Does pivotal recommend using confirmed publishing for applications that cannot lose messages? We are not talking financial transactions here. So, the occasional loss could be recovered/recreated. But, we can't have any significant amount of data loss or it would cause concern about the reliability of this system that has become a backbone for our entire product line.

Thanks again for the great information.

Best.

-rb

--
-rb

[Michael Klishin]

On 15 October 2014 at 00:15:47, Ryan Brown (ryank...@gmail.com) wrote:
> Secondly, we are using the network client. So that leads me to
> believe there is no real payoff to waiting for the ok. We do not
> want to use confirmed publishing as it just adds more overhead
> in publishing. However, this information raises some concern
> that possibly data could get lost if we don't do so. (although
> we have not actually seen this in practice with our current daily
> load of 17+M)

Network partition is a probability function. In some environments, the probability
is so low it may make sense to deliberately use riskier but higher throughput
solutions.

17M in 24 hours is about 200 messages/second — your RabbitMQ instance is probably
very lightly loaded, so you have some throughput head room.

Having your system in Erlang also gives you an option of switching to direct client
which avoids protocol serialisation overhead.

> Does pivotal recommend using confirmed publishing
> for applications that cannot lose messages? We are not talking
> financial transactions here. So, the occasional loss could
> be recovered/recreated. But, we can't have any significant
> amount of data loss or it would cause concern about the reliability
> of this system that has become a backbone for our entire product
> line.

You can't guarantee reliable delivery without getting a confirmation, so we definitely
recommend using them.
Dealing with acks and nacks w/o virtually no overhead is easier in Erlang
than most languages because [virtually all] Erlang systems are inherently event-driven and asynchronous.

 Several RabbitMQ plugins use publisher confirms under the hood (e.g. Shovel, Federation).

See "Acknowledgements and Confirms" on http://www.rabbitmq.com/reliability.html
and http://www.rabbitmq.com/confirms.html.

[Michael Klishin]

On 15 October 2014 at 01:23:53, Michael Klishin (mkli...@pivotal.io) wrote:
> 17M in 24 hours is about 200 messages/second — your RabbitMQ
> instance is probably
> very lightly loaded, so you have some throughput head room.

Actually, since the original message is about channels running into flow control,
this may not be a light load with headers exchanges.
I was talking about the general case here.

200 messages/sec is about 1/100th of what a several years old
development machine can do on basic benchmarks (using PerfTool)
with fanout of topic exchanges.

Perhaps tracking which processes run into internal flow control would be a nice
management UI improvement. 

[Laing, Michael]

You would get better throughput w 3 rabbitmq nodes instead of 4. I doubt if #4 is adding anything in terms of resilience but it is a drag on message processing.

You didn't say how big the messages are. We strip large message bodies and pass by URI reference.

You didn't say whether queues are backing up. We aim for 0 - we occasionally get hundreds of thousands as our instantaneous load can increase by 100x when news breaks. 

You didn't say whether your messages are persistent (delivery_mode=2). You get much better throughput with non-persistent messages. That's what we do.

Cheers,

ml

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send an email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Brown]

Michael,

Thank you for your response. I like your line here. Let me address each of your questions/statements individually:

"You would get better throughput w 3 rabbitmq nodes instead of 4. I doubt if #4 is adding anything in terms of resilience but it is a drag on message processing." [rb] Could you explain to me why this might be? I would assume (obviously incorrectly) that more RMQ nodes would just distribute the load more. I would think that incoming traffic would be distributed by the F5. Then, each RMQ node would then carry less of the burden of routing the individual messages to their respective queues. Am I following an over-simplified fallacy?

You didn't say how big the messages are. We strip large message bodies and pass by URI reference. [rb] Message bodies in our system are typically very small. They only average <25k

You didn't say whether queues are backing up. We aim for 0 - we occasionally get hundreds of thousands as our instantaneous load can increase by 100x when news breaks. [rb] This sounds very similar to our typical load. We keep them down to close to zero most of the time. Even with 1800+ queues. This does change when we receive batch runs of 1M+ messages throughout the night. To be clear though, our application is the client to each of these RMQ queues and attempts to deliver to an endpoint. If that endpoint does nto respond or returns a predefined code, we re-queue the message. We typically only see backed-up queues if one of our applications clients is down or not responding quickly. That said, I cannot say that each time this timeout occurs is when we have backed-up queues. However, I can say that we can reliably reproduce the issue when we artificially create "bad subscribers"

You didn't say whether your messages are persistent (delivery_mode=2). You get much better throughput with non-persistent messages. That's what we do. [rb] Our messages are persistent but that is due to our SLA of at-least-once delivery attempt that can survive a system re-boot.

Best.

-rb

--
-rb

[Michael Klishin]

> Could you explain to me why this might be? I would assume (obviously
> incorrectly) that more RMQ nodes would just distribute the load
> more. I would think that incoming traffic would be distributed
> by the F5. Then, each RMQ node would then carry less of the burden
> of routing the individual messages to their respective queues.
> Am I following an over-simplified fallacy?

Search for "data locality" in these threads:
https://groups.google.com/d/msg/rabbitmq-users/E9HRhnyCo5Q/qgr5Gkk6UpEJ
https://groups.google.com/d/msg/rabbitmq-users/RvIAEZeAyP4/XYcttozo8SkJ

Replicating to 3 mirrors is generally more work than to 2. Any client can
connect and publish to any node: the less intra-cluster traffic you have
the better.

[Laing, Michael]

It would also help if you: 1) describe the mirroring / high availability strategy you are using; 2) how you are distributing queues among nodes; and 3) how / if you are directing clients directly to the node that has the queue of interest.

ml

[Ryan Brown]

Michael,

Here are my best answers. Let me know if it is not what you are looking for. I apologize if my answers are ignorant as I am still figuraing-out many of these details.

1) describe the mirroring / high availability strategy you are using.

[rb] Clustered RabbitMQ nodes with mirrored queues (high-availability, via HA policy, with master and slave nodes).  Queues are also durable.

2) how you are distributing queues among nodes?

[rb] Due to the fact the F5 is handling all traffic there is really no predictability to how they get distributed.

3) how / if you are directing clients directly to the node that has the queue of interest. [rb] We are not. At least not intentionally. I think making the sub through the load-balancer makes this impossible. (????)

-rb

--
-rb

[Jason McIntosh]

A few things on this:

HA mode all means your data is REPLICATED to all nodes.  Configuraiton is ALWAYS replicated to all nodes.  All queues are MASTERED on a single node, so data manipulation is all done there, then replicated to the other nodes.

You can set an HA mode where you replicate to N number of nodes.  SO in a 4 node cluster, if you replicate to 2, that gives you some redundancy as well as the ability to scale your cluster horizontally.  But there IS still overhead due to cluster communication.  The more nodes in a cluster, the more overhead.  It's just far worse if you want to replicate the data in your queues to ALL nodes.

NOTE - connections to servers ARE balanced, but then they have to route the data from the master through existing connections to the "slave node" your client is connected to.  SO it's slower, though not appreciably from what I've seen (as long as your network, servers, etc. have capacity and are local to each other).  This does though let you scale connections and distribute request processing.  

With an F5, there isn't any easy way (at the moment) to keep your client connected to the same node that your server is running on. SO the answer to #3 on connecting to the "master node" is hit or miss.

Jason

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Jason McIntosh
https://github.com/jasonmcintosh/
573-424-7612

[Laing, Michael]

For rabbitmq, describe your infrastructure setup: physical/virtual, LAN/WAN, etc.

Might be useful to have that for your erlang apps too.

ml

[Ryan Brown]

Michael,

For both RMQ and the erlang app they are running on fairly substantial physical hardware (Currently. The plan is to move to ec2 but, these issues are raising concerns.). And they are all co-located in the same datacenter.

Thank you.

-rb

--
-rb

[Jason McIntosh]

Ryan:

To give you some metrics - we've been able to push on fairly beefy hardware around 20k messages a second.  We only start seeing flow control when Disk IO/RAM/CPU's  become an issue (network can be an issue too).  A question I'd have is when you see flow control - what is your server doing?  The ONLY time we've really had flow control is when we hit infrastructure limitations that suddenly caused rabbit to stop processing.  I'm going to toss some previous conversations in here:

http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2014-July/037142.html

The 20k messages a second was tested on a few servers with a SINGLE SSD, 8 cores, and I think it was 16GB of RAM, with durability but no confirms.  We were consuming at about the same rate all locally.  If you add a network layer in I saw things slow down a fair bit, but rabbit itself doesn't seem to flow control as long as it has available IO/CPU.  

OH and watch for server issues with rabbit being stuck on single cores (e.g. a queue really isn't multi-threaded), 

http://www.agoragames.com/blog/2011/06/24/of-penguins-rabbits-and-buses/

With the later releases of Erlang I've not seen many of those kinds of problems in a LONG while, but I had to toss it out there.

One thing - I've NOT tested with header type exchanges, so I can only relate tests I've done with direct or fanout type exchanges.

Jason

[Laing, Michael]

OK - with beefy hardware in a single DC, I would first make sure that that all the boxes are connected by very fast network links. The infra guys will probably SAY that they are but it is best to test - perhaps you have noisy neighbors or a weak link between racks.

And, without modifying your setup too much, I would implement an HA policy to limit the number of mirrors for each queue. Look here.

And at the same time, I would be watching memory/cpu/network/etc and drilling down more into why publishes are timing out - I'm not sure we've actually come to grips with that. There are many posts in this group that may help.

ml

[Ryan Brown]

Thank you Jason. this is the performance we have expected. And, honestly, have only just recently (in the past 3-4 months) started to see these timeouts. While we do have in increased load, it is nowhere near these levels. The only difference I have noted is that the increase has come predominantly from one publisher that happens to use 10+ headers and has over 100 subscribers to their messages.

Sounds to be like some isolated, in-depth, testing is required.

Thanks again!

-rb

--
-rb

[Ryan Brown]

Thank you Michael! I am definitely perplexed. The servers actually connected with fiber. However, I will run some tests.

I am going to set-up some tests comparing RMQ with the different HA policies as well.

Best,

-rb 

--
-rb

[Simon MacMullen]

On 17/10/14 12:38, Ryan Brown wrote:
> I am going to set-up some tests comparing RMQ with the different HA
> policies as well.

You may want to wait until you are running 3.3.x - if you are facing a
bottleneck in the server it is much easier to identify, see the blog
post I previously mentioned.

> The only difference I have noted is that the increase has come
> predominantly from one publisher that happens to use 10+ headers and
> has over 100 subscribers to their messages.

...but if I had to guess it might be that routing these messages is
proving expensive - the headers exchange is by far the least well optimised.

Cheers, Simon

[Laing, Michael]

Getting back to exchange type, switching to topic exchanges is something you probably will want to try in the medium term. They are highly performant in our experience.

With regard to using AWS, reliable networking among components is the achilles heel you must compensate for. We have a 'rabbits everywhere' policy, and use shovels (primarily) and federation - like rubber bands - to connect things into a redundant mesh.

ml

[Ryan Brown]

Thank you Simon. I will definitely take your advice. The upgrade may, unfortunately, take several weeks. I will post my various test results to my blog when completed.

-rb

--
-rb

[Ryan Brown]

Michael,

We are looking into unobtrusively switching to the topic exchange. This may prove slightly difficult due to some poor (in retrospect) decisions we made with our client contract to support routing flexibility. But, we will likely make that change and may just create a v2 contract and allow clients to move over time.

In regards to our AWS move. We have been using your great presentations as a guideline to help guide our re-arch. thank you so much for sharing your knowledge! Not to mention taking the time to answer questions like these.

Best,

-rb

--
-rb