unacked messages

[Jiří Kolb]

Hello,

I have following simple architecture logstash -> rabbitmq -> graylog. I can see that messages are delivered to graylog without issues, but in rabbitmq I can see that messages are unacked. I guess that messages after delivering should be cleared from queue, but qued messages still show messages in queue. Can you please help?

Jiri

[Michael Klishin]

See tutorial 2: http://www.rabbitmq.com/getstarted.html.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Sean Nolan]

The tutorial shows how to have the consumer send acks but what about if the consumer says they are going to send acks but they don't send them. In that case on the server I see that the consumer requires ack but the unacked message count and process memory both just keep increasing for the consumer. In my case the messages are simply notifications so I would like to prevent the consumer from requiring acks at all - can I set a policy or do something else to achieve that? I've tried setting a message-ttl for all queues but I'm still seeing the unacked message count just keep growing.

Sean

On Saturday, February 18, 2017 at 1:37:30 AM UTC-6, Michael Klishin wrote:

See tutorial 2: http://www.rabbitmq.com/getstarted.html.

On Sat, Feb 18, 2017 at 10:27 AM, Jiří Kolb <jiri...@gmail.com> wrote:

Hello,

I have following simple architecture logstash -> rabbitmq -> graylog. I can see that messages are delivered to graylog without issues, but in rabbitmq I can see that messages are unacked. I guess that messages after delivering should be cleared from queue, but qued messages still show messages in queue. Can you please help?

Jiri

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Consider starting new threads for new questions.

Your question is answered in http://www.rabbitmq.com/confirms.html.

Not requiring acknowledgements means that RabbitMQ can potentially

outpace your consumer. When that happens consumer(s) can really quickly

run out of memory.

See for some numbers that explain how bumping QoS prefetch can

increase throughput without running the risk of automatic acknowledgements:

https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sean Nolan]

I don't think I asked a new question, I was adding to or expounding on the original question, which was "in rabbitmq I can see that messages are unacked. I guess that messages after delivering should be cleared from queue, but qued messages still show messages in queue." 

I'm saying that I see the same thing - the number of Unacked messages grows on the server and uses memory on the server because the consumer has issued a basic consume with noAck set to false, but then the consumer is not issuing a basic ack when they receive a message. I understand how it should work and why it would be dangerous for the consumer but in our use case I'm more concerned about making sure that misbehaved consumers can be prevented from causing my server to run out of memory. So the question is the same as the original - messages that are unacked stay in the queue after being successfully delivered - in my application they should be cleared after delivery regardless of ack, is there a way I can do that - either when the message is published, or preferably in a server level policy? If not then that's fine I'll figure out another way to monitor for misbehaved consumers and disconnect them, but if there is a simple way to do it in the server configuration then I'm not seeing it so I'm asking for help.

Thanks

Sean 

[Michael Klishin]

On 1 March 2017 at 23:19:15, Sean Nolan (sno...@nextradioapp.com) wrote:
> So the question is the same as
> the original - messages that are unacked stay in the queue after being
> successfully delivered - in my application they should be cleared after
> delivery regardless of ack, is there a way I can do that - either when the
> message is published, or preferably in a server level policy?

We have an entire documentation guide on acknowledgements:
http://www.rabbitmq.com/confirms.html

Have you read it? I believe it has an answer.

> On Tuesday, February 28, 2017 at 3:27:01 PM UTC-6, Michael Klishin wrote:
> >
> > Consider starting new threads for new questions.
> >
> > Your question is answered in http://www.rabbitmq.com/confirms.html.
> >
> > Not requiring acknowledgements means that RabbitMQ can potentially
> > outpace your consumer. When that happens consumer(s) can really quickly
> > run out of memory.
> >
> > See for some numbers that explain how bumping QoS prefetch can
> > increase throughput without running the risk of automatic acknowledgements:
> >
> > https://www.rabbitmq.com/blog/2014/04/14/finding-bottlenecks-with-rabbitmq-3-3/
> >
> >
> > On Wed, Mar 1, 2017 at 12:16 AM, Sean Nolan > > <>> wrote:
> >
> >> The tutorial shows how to have the consumer send acks but what about if
> >> the consumer says they are going to send acks but they don't send them. In
> >> that case on the server I see that the consumer requires ack but the
> >> unacked message count and process memory both just keep increasing for the
> >> consumer. In my case the messages are simply notifications so I would like
> >> to prevent the consumer from requiring acks at all - can I set a policy or
> >> do something else to achieve that? I've tried setting a message-ttl for all
> >> queues but I'm still seeing the unacked message count just keep growing.
> >>
> >> Sean
> >>
> >> On Saturday, February 18, 2017 at 1:37:30 AM UTC-6, Michael Klishin wrote:
> >>>
> >>> See tutorial 2: http://www.rabbitmq.com/getstarted.html.
> >>>

> >>> On Sat, Feb 18, 2017 at 10:27 AM, Jiří Kolb wrote:
> >>>
> >>>> Hello,

> To post to this group, send an email to rabbitm...@googlegroups.com.

[Sean Nolan]

I have read it and I hear you - you're saying the consumer isn't doing it right - I agree, but if the consumer isn't doing it right, I control the server but I don't control the consumer. So if the consumer sets noAck to false (ack is required) but then doesn't send basic acks - I see the Unacked count, and the In memory count and the Process memory usage all steadily increase for the consumers' queue (see attached). 

The documentation assumes that the consumer is issuing a basic ack - that's fine, that's what I would like to do, but I'm not able to control all my consumers AND for my use case the ack is not necessary, so I'm asking can I prevent the consumer from requiring an ack, if I can do that then they can't get it wrong.

Thanks

Sean

[Michael Klishin]

If you are not able to control your consumers the only thing you can do is to monitor channels

and queues using HTTP API and force close channels that have a higher than N number of unacknowledged messages.

Or set max queue length via a policy but clients with sufficient permissions can manage them, too.

It can be perfectly natural for consumers to slow down because their dependencies slow down

and thus for unacknowledged messages to pile up from time to time.

3.7.0 will provide a special kind of policies that users cannot manage.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Sean Nolan]

Thanks for confirming that - I'm sure others will find it useful to have it confirmed too.

Sean