RabbitMQ 3.6.2 RC1 -- "ghost" consumers on lost channels

[John F]

We are heavily load testing this version in a test environment and we see several errors with this version (as well as previous 3.6.x versions).

Right now we have some consumers on a queue that looks like "ghost" consumers because they are there and they are on non-existing (unknown) channels. See screenshot.

The problem now is that we can't click the "unknown" channel link because it is non-existing but it seems that some of these "ghost" consumers are actually retrieving messages from the queue but our backend service doesn't receive it. So it looks like some internal logic around this is wrong.

[Michael Klishin]

How exactly are you testing this version?
Do those consumers show up in `rabbitmqctl list_consumers`?
Do they re-appear if you reset management database as described
at the bottom of http://www.rabbitmq.com/management.html?  

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Michael Klishin]

On 2 May 2016 at 16:17:15, Michael Klishin (mkli...@pivotal.io) wrote:
> How exactly are you testing this version?
> Do those consumers show up in `rabbitmqctl list_consumers`?
> Do they re-appear if you reset management database as described
> at the bottom of http://www.rabbitmq.com/management.html?

and one more question: what is in log files? 

[John F]

How exactly are you testing this version?

We have it setup in a 4 node cluster with load balancers in front of each pair (2 nodes). All nodes runs AMQP and STOMP with TLS/SSL enabled. STOMP is the primary protocol for most connections (hardware devices). AMQP is used for backend services. We use HA queues (which this one is).

We have some services that consumes and produces messages and a lot of hardware devices that are connected. However, what we have tried since Friday is to create a Docker environment and see if we could load test the server using only AMQP since our tests over STOMP seemed to crash the server all the time. Now with AMQP the same problems/crashes occurs. We are only talking about 100-500 connections with small messages.
 

Do those consumers show up in `rabbitmqctl list_consumers`?

No... the command just says "Listing consumers ..." and exits.
 

Do they re-appear if you reset management database as described
at the bottom of http://www.rabbitmq.com/management.html?

Running that command just returns "true" and exits. I ran it on the node where stats was running.
No changes to the consumers list.

Logs shows nothing so far...

[Michael Klishin]

On 2 May 2016 at 16:46:09, John F (john.f...@gmail.com) wrote:
> > Do those consumers show up in `rabbitmqctl list_consumers`?
>
> No... the command just says "Listing consumers ..." and exits.
>
> > Do they re-appear if you reset management database as described
> > at the bottom of http://www.rabbitmq.com/management.html?
>
> Running that command just returns "true" and exits. I ran it on
> the node where stats was running.
> No changes to the consumers list.
>
> Logs shows nothing so far...

but you know for a fact that there are consumers online, e.g. message rates are non-0
and your application logs report that messages are being consumed? 

[John F]

but you know for a fact that there are consumers online, e.g. message rates are non-0
and your application logs report that messages are being consumed? 

We have producers (hardware devices) online, yes.
We have consumers (backend services) online, yes.
We have the rabbitmq-event-exchange plugin enabled and have seen it working for connection.* bindings, yes.
We have a sniffer client that connects to all exchanges and consumes all messages and it sees the event connection.connection and connection.disconnect event messages, yes.
Our backend services (consumers) does not receive the events, no.

So... it looks like one of these consumers on a "unknown" channel actually receives these messages but does nothing with it.