when there are no messages in the queue stop consuming

[owgit...@gmail.com]

Hello Team,

I am running this example asynchronous consumer: https://github.com/pika/pika/blob/master/examples/asynchronous_consumer_example.py 

I just need to modify this to have when there are no messages in the queue stop consuming and log how much time it took to consume all the messages.

Thanks, 

Shantha

[Hans-Martin Mosner]

In general, it's impossible to know when all messages have been consumed, as it might be possible that the producer is still running and just not fast enough filling the queue that you've been draining.

If you just want to stop when no more messages can be retrieved at some time, you can either set a short timeout after processing each message that will stop your consumer in casxe no new messages arrive within the timeout interval, or you can use basic.get to poll the queue (which is generally not recommended for production use, but neither is setting the prefetch count to 1).

If you know for certain that you'll be the only consumer, you can look at the message_count attribute of the Queue.DeclareOk frame that's being passed into the on_queue_declareok() method and use it to initialize a counter that you decrement after each received message, stopping when it reaches 0.

(This one I haven't tried yet:) If you have multiple consumers (a.k.a. worker threads), you can publish a "sentinel" message to the queue when you start, and consume messages until you receive such a sentinel message. Message order should be preserved, so this message should be seen only after all messages with payload have been received. It doesn't matter whether you as a consumer receive the same sentinel message that you created in the beginning or get one created by another consumer, since they only signal that you should stop consuming, and there's one for each consumer so every consumer should get one.

Cheers,

Hans-Martin

[Shantha Dodmane]

Hello Hans-Martin,

Thanks a lot for your response. 
Let me give you an overview of the problem statement I have.  Basically my producer will produce hundreds of messages every three hours at once. And my consumer is capable of processing those messages within one hour. 

But I need to log the exact time it took to process those hundreds of  messages every time.

Thanks,

Shantha Dodmane

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/82d61650-34bf-49de-a161-2954d9774d7fn%40googlegroups.com.

[Hans-Martin Mosner]

This is somehow turning into a one-on-one consulting session, should I quote my hourly rate? :-)

Your requirements are pretty incomplete and inexact, but you want an exact solution. That's not going to work.

You need to find out what the exact requirements are (i.e. what event triggers the hundreds of messages, can the consumer be started when the producer is known to have finished a batch, how precise must the timing be, when must the time be available, etc.). All of these determine what possible solutions can be found. It would also be interesting whether this is an exercise/test task or some real requirements. You're supposed to come up with solutions by yourself.

A good practive in the stack exchange group of sites is to require people who ask questions to show what they have already tried and how it didn't work. What have you already tried on your own? My timeout solution should work for your case, have you tried it?

Cheers,
Hans-Martin

[john pfersich]

+1

/*—————————————————-*/

Sent from my iPhone

https://boincstats.com/signature/-1/user/51616339056/sig.png

See https://objectnets.net and https://objectnets.org

https://datascilv.com and https://datascilv.org

On Jun 3, 2021, at 06:04, 'Hans-Martin Mosner' via rabbitmq-users <rabbitm...@googlegroups.com> wrote:



To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/185e2386-71ad-428c-93df-94337d40e9a6n%40googlegroups.com.