Issue with message count on rabbitmq

[rahul gupta]

Hi, 

SCENARIO:

a queue with some messages and no more messages are being produced on queue.

SimpleMessageListenerContainer to consume messages. AcknowledeMode is set to AUTO_ACK

Rabbitmq Admin url to find the total number of messages (Ready + Unacknowleded) using the api call '/api/queues/%2F/<queue_name>

LOGIC:

   SimpleMessageListenerContainer.setQueueName(<queue_name>);

  SimpleMessageListenerContainer.start();

  waitTillMessageCountIsZero() // i.e. all messages are consumed and acknowledged

  SimpleMessageListenerContainer.stop()

ifUnused = false, ifEmpty = true

  RabbitAdmin.deleteQueue(<queue_name>, ifUnused, ifEmpty);

ISSUE : 

When I try to delete the queue, the queue still has some messages in unacknowledged state and some consumers are also attached

I understand that a call to SimpleMessageListenerContainer.stop() will not block for `inFlight` messages to be consumed. The messages will be eventually consumed and acknowledeged even after calling stop()

However, I can not understand why the messageCount was zero in the first place when all messages are not yet acknowledged.

I am assuming that for a message, transition from Ready to Unack'd is atomic.

 Is my assumption correct? Is there some other lifecycle state for a message in between READY and UNACK'd.

Please throw some light on this.

Regards, 

Rahul

[Michael Klishin]

 On 26 May 2015 at 12:16:00, rahul gupta (rahu...@gmail.com) wrote:
> However, I can not understand why the messageCount was zero
> in the first place when all messages are not yet acknowledged.

If Spring AMQP (or your own code) uses queue.declare-ok to count the number of messages,
keep in mind that it returns a number of messages that are ready for delivery, so
that does not include messages *already delivered*.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[rahul gupta]

I am not using queue.declare-ok to count the number of messages

As already mentioned above, I am using Rabbit mq admin url i.e. <RABBIT_ADMIN_ENDPOINT>/api/queues/%2F/<queue_name>

It provides the complete queue stats. I am using "messages_ready" and "messages_unacknowledged" fields from the json response.

Thanks,

Rahul

[Michael Klishin]

On 26 May 2015 at 12:47:45, rahul gupta (rahu...@gmail.com) wrote:
> As already mentioned above, I am using Rabbit mq admin url i.e.

> /api/queues/%2F/

>
> It provides the complete queue stats. I am using "messages_ready"
> and "messages_unacknowledged" fields from the json response.

Then I'm not sure what your question is. The reason why we have message_ready
and messages_unacknowledged is because not everybody agrees on what should
be the "total number of messages" at any given moment.

[rahul gupta]

Michael, 

The issue is of intermittent nature.

When a message is delivered to a consumer, the message moves from READY state to UNACK'd state till the consumer acknowledges it.

Concern : While a message is being delivered to a consumer, the state transition from READY to UNACK does not seem to be "ATOMIC". I am observing an intermittent state where the rabbitmq shows zero messages(ready and unack both) and after a while, the messages are being shown in UNACK'd state.

It occurs very fast and can not be observed with naked eye. 

Before stopping SMLC, I need some sort of guarantee that all messages are already consumed(It is guaranteed that no more messages will be produced).

Please guide on how can I guarantee with SMLC whether all messages are consumed and acknowledged.

I hope I have made myself clear.

Regards, 

Rahul

--

Thanks and Regards,
---
Rahul Gupta

[Michael Klishin]

On 26 May 2015 at 13:41:03, rahul gupta (rahu...@gmail.com) wrote:
> Concern : While a message is being delivered to a consumer, the
> state transition from READY to UNACK does not seem to be "ATOMIC".

Atomic with respect to what? (what other events)

As far as what is going on in a single RabbitMQ message store (node), it
is effectively atomic.

 
> I am observing an intermittent state where the rabbitmq shows
> zero messages(ready and unack both) and after a while, the messages
> are being shown in UNACK'd state.

Permanently or temporarily ? How long is "a while"?

> It occurs very fast and can not be observed with naked eye.

Have you considered that an HTTP request that returns the data runs concurrently with
message deliveries? So there may be "in flight" messages at any time.

[Michael Klishin]

On 26 May 2015 at 13:41:03, rahul gupta (rahu...@gmail.com) wrote:

> Before stopping SMLC, I need some sort of guarantee that all
> messages are already consumed(It is guaranteed that no more
> messages will be produced).
>
> Please guide on how can I guarantee with SMLC whether all messages
> are consumed and acknowledged.

OK, this is a lot more specific.

To guarantee that, you need to first unbind your queue from every exchange
to make sure no deliveries are coming in (and then there's the default exchange…)

Then cancel your consumer. How that translates to Spring AMQP concepts, I'm not
sure, sorry .

[rahul gupta]

On May 26, 2015 4:17 PM, "Michael Klishin" <mkli...@pivotal.io> wrote:
>
> On 26 May 2015 at 13:41:03, rahul gupta (rahu...@gmail.com) wrote:
> > Concern : While a message is being delivered to a consumer, the
> > state transition from READY to UNACK does not seem to be "ATOMIC".
>
> Atomic with respect to what? (what other events)
>
> As far as what is going on in a single RabbitMQ message store (node), it
> is effectively atomic.
>  

By atomicity, i mean when a message is being delivered, queue stats should be Ready-1 and unack +1 atomically

> > I am observing an intermittent state where the rabbitmq shows
> > zero messages(ready and unack both) and after a while, the messages
> > are being shown in UNACK'd state.
>
> Permanently or temporarily ? How long is "a while"?
>

Its temporary.
A while is in ms. I make request every 10 ms for now.

> > It occurs very fast and can not be observed with naked eye.
>
> Have you considered that an HTTP request that returns the data runs concurrently with
> message deliveries? So there may be "in flight" messages at any time.
> --

Where are these ' in flight' messages accounted for? I guess, definitely not in 'ready'. How can i get a count of these messages?

[rahul gupta]

It is already guaranteed by the app that no more messages are being delivered.

I would like to call "cancel" on my consumer once it is certain that all packets are consumed which is appearing to be a false alarm in this case.

Any idea on how to get track of the "in flight" messages which are not counted in both ready and unack'd messages.

[Michael Klishin]

On 26 May 2015 at 15:13:37, rahul gupta (rahu...@gmail.com) wrote:
> Any idea on how to get track of the "in flight" messages which
> are not counted in both ready and unack'd messages.

You can't. The client can't know what may be coming on the wire.

I'd just give your consumer a grace period when shutting down. RabbitMQ Java
client does this: when you shut it down, there's a period of time before
consumer thread pool is shut down, to let consumers finish the work they may
be doing. Since we don't know what that work may be or how long it takes, we have
a timeout (10 seconds by default IIRC). 

[rahul gupta]

What we are concerned about here is not the client but someone else monitoring the overall queue status. Client obviously can't know what may be coming on the wire.

spring-amqp's SMLC does already provide a grace period for shutdown (5 seconds by default), but I want to delay the invocation of shutdown method itself till all messages are already consumed i.e. acknowledged.

This issue is arising because of the false information rabbitmq provides when a message is being moved from READY to UNACK state (i.e. in flight)

I am not sure whether it is a bug with rabbitmq server that the operation of moving a message from READY to UNACK is not atomic.

Any pointers on how to understand in detail the overall cycle of a message in rabbitmq?

[Michael Klishin]

On 26 May 2015 at 15:31:01, rahul gupta (rahu...@gmail.com) wrote:
> What we are concerned about here is not the client but someone
> else monitoring the overall queue status. Client obviously
> can't know what may be coming on the wire.
>
> spring-amqp's SMLC does already provide a grace period for shutdown
> (5 seconds by default), but I want to delay the invocation of shutdown
> method itself till all messages are already consumed i.e. acknowledged.
>
> This issue is arising because of the false information rabbitmq
> provides when a message is being moved from READY to UNACK state
> (i.e. in flight)

I am not convinced it is "false". Your monitoring system has a natural race
condition with messages on the wire.

> Any pointers on how to understand in detail the overall cycle of a message in rabbitmq?

Interactions with consumers is primarily handled by rabbit_queue_consumers:
https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_queue_consumers.erl#L124-125

Transitioning a message from Ready to Unacknowledged involves updating multiple state
fields. As far as CPUs go, that is not atomic. But as far as any external observer goes,
it is. Server cannot know if the messages it has sent out were delivered or not
because there are several layers of buffering (in rabbit_writer, networking stack in the OS
and device drivers) between them.

If you see some messages in unacknowledged state after your consumer has done all the work,
it almost certainly means some messages were not acknowledged. 99.9% of such issue reports
end up being an application-specific issue. 

[Gerard ONeill]

I was wondering, Rahul, whether or not you got your consumer to work.

Michael, this was almost 2 years ago, but it seems this thread ended without you knowing what Rahul was talking about -- perhaps you have insight now?

Rahul was not trying to eliminate all so called race conditions including when an inflight message might hit the queue.  He was talking about the race condition that may indicate to a consumer that there are 0 messages in the queue, but in reality there are 5 messages transitioning from ready to unacked.  To the consumer there looks like there are no messages ready for processing or being processed.  The inflight message does not affect this at all.

I'm getting 0 for my MessageCount in my api despite preloading the queue with 2 -5 messages, and found this thread -- I don't know right now if my problem is due to this 'race' condition, or if I'm not using the API correctly.  I'll still search the interwebs for others who may have had this problem.

G.