RabbitMQ dropping messages?

[Ryan Brown]

Hello-

I realize this topic may sound inflammatory. I apologize for that in advance. However, we have begun to see what appears on the surface to be just that.

With our application we receive requests and publish them to RMQ via a headers exchange. (We do not currently use publish/confirms for performance concern reasons.) We then log that we have received the message. Then, when the message is received by the subscriber we log again. What I have begun to see, very infrequently, but enough to cause concern, is the first log message occurring saying that we have handed the message off to RMQ. However, we never see the message delivered back to the subscriber. Other messages before and after the "dropped" message are being handled as expected. And I have re-sent the exact same message through and it succeeds.

So, I have several questions.

1. When my application receives the 'ok' back from RMQ on a basic.publish, what precisely does that mean?
2. Under what circumstances might a message be dropped after this 'ok' is received?

This application has been in production for several years and we have just begun to have reports of these messages not making it through. That would suggest that we have done something to make this happen. However, I am having difficulty determining what that may have been without understanding what might cause this in the first place.

System Information:

• REST wrapper on RMQ written in Erlang (currently running R15B01)
• RMQ version 3.2.2
• amqp_client 2.7.1
• All queues are defined as durable and HA

Any information that might lead me in the correct direction would be greatly appreciated.

Thank you.

[Michael Klishin]

 On 7 March 2015 at 01:27:37, Ryan Brown (ryank...@gmail.com) wrote:
> When my application receives the 'ok' back from RMQ on a basic.publish,
> what precisely does that mean?

"When will messages be confirmed?" on http://www.rabbitmq.com/confirms.html.

> Under what circumstances might a message be dropped after this
> 'ok' is received?

Are messages published as persistent? Are they routed to the queue(s) you expect?
Do messages or queues have a TTL? (can a TTL policy by erroneously applied to them?)
Were they consumed by another consumer?
Can the queues
be exclusive? (they are never mirrored because they are tied to the connection)
Can something in the system be purging queues?

There can be all kinds of reasons why a consumer doesn't receive a particular message,
including swallowed exceptions in handleDelivery.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ryan Brown]

Michael,

Thank you for the quick response. To answer your questions:

Are messages published as persistent? << yes

Are they routed to the queue(s) you expect? << not that I can tell. Can we tell this from RMQ logs? We have only found these occurrences in retrospect days later as we process many millions of messages a day.
Do messages or queues have a TTL? (can a TTL policy by erroneously applied to them?) << no
Were they consumed by another consumer? << no
Can the queues
be exclusive? (they are never mirrored because they are tied to the connection) << please clarify. We have 6 consuming nodes and all receive messages from all queues.
Can something in the system be purging queues? << no. All messages before and after are successfully delivered 

[Alvaro Videla]

To debug message routing you could use the tracer plugin https://www.rabbitmq.com/firehose.html

Exclusive queues, means that the application code specifically declares the queue as being exclusive (nice tautology). See: https://www.rabbitmq.com/amqp-0-9-1-reference.html#queue.declare.exclusive So if your code doesn't set that flag, then your queues are not exclusive.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.