Messages are piling up on the queue but not delivered to the consumer

[V Z]

Is there a known condition when messages, even though being in the ready state on the queue, are not delivered to the consumer? Certainly, it may be consumer's problem, but I am trying to cover all bases (Rabbit 3.6.6, Erlang 18.3.4.4, Java consumer using Rabbit client AMQP library)

We had a situation today where several consumers were connected to a queue per the management UI, messages were arriving on the queue, but nothing was delivered to the consumers until they got recycled. Once recycled and reconnected, messages were delivered. 

Is there something we could have checked or reviewed? Thanks!

[Michael Klishin]

Consumers using manual acknowledgements are not acknowledging at the rate developers expect them to.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gary Russell]

> until they got recycled.

In my experience, the most common reason for this is the consumer thread is "stuck" somewhere in user code (and missing manual acks, as Michael said).

Bouncing the server, or app, generally clears the issue (for obvious reasons).

Taking a thread dump when the server is hung is generally the best first step to debugging situations like this.

On Wed, Mar 1, 2017 at 7:15 PM, Michael Klishin <mkli...@pivotal.io> wrote:

Consumers using manual acknowledgements are not acknowledging at the rate developers expect them to.

On 2 Mar 2017, at 02:49, V Z <uvzu...@gmail.com> wrote:

Is there a known condition when messages, even though being in the ready state on the queue, are not delivered to the consumer? Certainly, it may be consumer's problem, but I am trying to cover all bases (Rabbit 3.6.6, Erlang 18.3.4.4, Java consumer using Rabbit client AMQP library)

We had a situation today where several consumers were connected to a queue per the management UI, messages were arriving on the queue, but nothing was delivered to the consumers until they got recycled. Once recycled and reconnected, messages were delivered. 

Is there something we could have checked or reviewed? Thanks!

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

[Michael Klishin]

RabbitMQ management UI provides metrics about delivery rate for each acknowledgement mode,

acknowledgement rate, redelivery rate, and a few others that can help narrow the issue down.

On Thu, Mar 2, 2017 at 4:18 AM, Gary Russell <grus...@pivotal.io> wrote:

> until they got recycled.

In my experience, the most common reason for this is the consumer thread is "stuck" somewhere in user code (and missing manual acks, as Michael said).

Bouncing the server, or app, generally clears the issue (for obvious reasons).

Taking a thread dump when the server is hung is generally the best first step to debugging situations like this.

On Wed, Mar 1, 2017 at 7:15 PM, Michael Klishin <mkli...@pivotal.io> wrote:

Consumers using manual acknowledgements are not acknowledging at the rate developers expect them to.

On 2 Mar 2017, at 02:49, V Z <uvzu...@gmail.com> wrote:

Is there a known condition when messages, even though being in the ready state on the queue, are not delivered to the consumer? Certainly, it may be consumer's problem, but I am trying to cover all bases (Rabbit 3.6.6, Erlang 18.3.4.4, Java consumer using Rabbit client AMQP library)

We had a situation today where several consumers were connected to a queue per the management UI, messages were arriving on the queue, but nothing was delivered to the consumers until they got recycled. Once recycled and reconnected, messages were delivered. 

Is there something we could have checked or reviewed? Thanks!

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[V Z]

Right, that is the obvious suspicion and the one that is the easiest to make but hard for consumers to accept. I wanted to make sure the queue process was not messed up somehow, but if it was, messages would not be appearing on the queue. Is that so?

[V Z]

One additional piece of information. There were 4 consumers on this queue, all running on separate VMs, and all stopped receiving messages at the same time until they were restarted. Does this make suspicion of application's problem less probable?

[Michael Klishin]

Possibly. Most consumers these days use shared external resources such as a database.

On Sat, Mar 4, 2017 at 5:31 AM, V Z <uvzu...@gmail.com> wrote:

One additional piece of information. There were 4 consumers on this queue, all running on separate VMs, and all stopped receiving messages at the same time until they were restarted. Does this make suspicion of application's problem less probable?

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.

To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[V Z]

That's a valid angle.

[Avi Block]

I'm having a situation where a queue is piling up, and the messages don't seem be delivered. Currently, the only consumer is the lodash rabbimq plugin.

I wrote a dead simple consumer in node, and verified that that messages do not come through.

const amqp = require('amqplib');

const q = 'central_access_logs';

async function main() {

   const conn = await amqp.connect("*****")

    );

    console.log('connected');

    const ch = await conn.createChannel();

    const ok = await ch.assertQueue(q);

    console.log(ok);

    const result = await ch.consume(q, function(msg) {

       console.log('any body home?' + JSON.stringify(msg));

       if (msg !== null) {

           console.log(msg.content.toString());

           ch.ack(msg);

       } else {

            console.log('cancelled');

           process.exit(0);

       }

   });

    console.log(result);

}

main();

This same code with any other queue works fine

On Saturday, March 4, 2017 at 10:43:49 PM UTC-5, V Z wrote:

That's a valid angle.