Auto_ack and consumers

[aviv salem]

I know that when working with auto_ack=true, the prefetch size and count are ignored...

So as soon as the consumer gets a message it is acked, and another is sent. 

What I couldn't find, is how the server avoids overwhelming the client...

We're using pika, and the consumer generator... We noticed that the server sends all the messages to the client, even though the client is still running through the generator... I know it is the expected behavior, but my question is:

Will it stop at some point? Does the consumer have a way to tell the server that it's resources are low (for example to memory) and to stop sending messages for a while?

Does pika have some "max messages in generator memory" setting that allows me to throttle the rate of incoming messages? 

I know that it can be achieved with auto_ack=false and setting prefetch_size, but I'm asking about Auto_ack=true....

10x!

[Luke Bakken]

Hi Aviv,

I believe that the only "flow control" in this case is TCP back-pressure. So, if your client processes messages slowly, the TCP buffer will fill, TCP flow control will kick in, and the amount of data RabbitMQ is allowed to send over that socket will decrease, perhaps to zero. If your client prevents writing to the TCP socket for more than 30 seconds a timeout will be raised and RabbitMQ will close the socket.

Thanks,

Luke

[aviv salem]

Thanks for the answer...

If rabbitmq closes the socket, what happens to the messages already in memory (pika's consumer generator)?

Will the generator continue to give me messages until it runs out and then raise an exception? Or will it raise an exception immediately?(and so I'll lose the messages received so far...) 

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/bac7021b-40d8-4d86-95c5-c98ae6c46a2c%40googlegroups.com.

[Wesley Peng]

Hi

on 2019/9/9 16:41, aviv salem wrote:
> If rabbitmq closes the socket, what happens to the messages already in
> memory (pika's consumer generator)?
>

You would be able to re-connect to rabbitmq to produce/consume messages
etc. When connection is broken, the client library will raise an error,
you capture the error and try to re-connect to the cluster.

regards.

[Luke Bakken]

Hi Aviv,

The only way applications can determine that a TCP socket is closed is the next time they try to do something with it. In addition, prior to closing the TCP connection RabbitMQ will send a channel-level exception (via a Channel.Close message) to all channels on that connection, which clients will receive and raise as exceptions.

In the case of the consumer generator, Pika may try to check the socket for a heartbeat, or to send one, or might even get that Channel.Close message. If Pika churns through the messages without checking the socket, then they won't be lost. I haven't looked at exactly how the consumer generator is implemented, but I'm pretty sure a heartbeat check can be run "in the middle" of the generator, and it may also be possible that ChannelClosed will be raised mid-run.

Anything else in the TCP buffer will be lost, which is why auto-ack is not a safe way to consume messages. It's fast, but you can lose messages.

Thanks,

Luke

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.