[rabbitmq-discuss] 404, "NOT_FOUND - no queue 'foo' in vhost '/'"

[Matt Pietrek]

I'm encountering a strange situation that I don't think has been covered by other threads.

I'm running RabbitMQ 2.7.0 on Ubuntu 10.04 LTS with the stock PIKA 0.9.5 client, and taking advantage of mirrored queues.

My scenario:

• I have two apps that ping-pong messages to each other using two queues (a pseudo-RPC type of thing.)
• The apps talk to RabbitMQ via a keepalived monitored Virtual IP.
• Before starting the apps, I predeclare both queues.
• In the RabbitMQ management Web UI, I see both nodes and the Queues tab shows both queues as "HA D", so I believe they're mirrored.
• Start up the two ping-ponging apps, let run for few seconds. Note messages flowing back and forth.
• Stop the primary node (rabbitmqctl stop)
• Keepalived fails over the Virtual IP
• After a few seconds I see one of the apps resume processing messages as expected.
• Shortly thereafter the other app gets an exception from channel.basic_get(), which is the 404 "no queue found" exception.
• I don't believe it's a connection error, as during my queue reads, my logic catches AMQPConnectionErrors exceptions and retries after a short wait.

I've looked at my code carefully, and there's no calls to queue_declare other than the original creation (item 3, above.)

Any thoughts on what could be wrong?

Thanks much,

Matt

[Matt Pietrek]

Some additional information I've learned since posting the original, below.

After shutting down the primary node, the queue causing the 404 error has actually vanished from the broker. Keeping an eye on the Management Web UI, I literally see it disappear.

In addition, in the rabbit@<nodename>.log, I see a whole bunch of spew, including many instances of this:

=WARNING REPORT==== 20-Dec-2011::14:09:17 ===

Non-AMQP exit reason '{{{function_clause,

                            [{gb_trees,lookup,

                                 [<<150,83,52,247,194,247,140,35,130,167,168,

                                    35,224,203,183,186>>,

                                  {dict,0,16,16,8,80,48,

                                      {[],[],[],[],[],[],[],[],[],[],[],[],[],

                                       [],[],[]},

                                      {{[],[],[],[],[],[],[],[],[],[],[],[],

                                        [],[],[],[]}}}]},

                             {rabbit_amqqueue_process,

                                 '-confirm_messages/2-fun-0-',2},

                             {lists,foldl,3},

                             {rabbit_amqqueue_process,confirm_messages,2},

                             {rabbit_amqqueue_process,next_state,1},

                             {rabbit_amqqueue_process,noreply,1},

                             {gen_server2,handle_msg,2},

                             {proc_lib,wake_up,3}]},

                        {gen_server2,call,

                            [<0.1298.0>,

                             {basic_get,<0.2542.0>,false},

                             infinity]}},

                       [{gen_server2,call,3},

                        {rabbit_misc,with_exit_handler,2},

                        {rabbit_channel,handle_method,3},

                        {rabbit_channel,handle_cast,2},

                        {gen_server2,handle_msg,2},

                        {proc_lib,init_p_do_apply,3}]}'

Again, this is a stock 2.7.0 install.

[Matthias Radestock]

On 20/12/11 22:44, Matt Pietrek wrote:
> In addition, in the rabbit@<nodename>.log, I see a whole bunch of spew,
> including many instances of this:
>
> =WARNING REPORT==== 20-Dec-2011::14:09:17 ===
> Non-AMQP exit reason '{{{function_clause,
> [{gb_trees,lookup,
> [<<150,83,52,247,194,247,140,35,130,167,168,
> 35,224,203,183,186>>,
> {dict,0,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[],[]}}}]},
> {rabbit_amqqueue_process,
> '-confirm_messages/2-fun-0-',2},

> [...]

> Again, this is a stock 2.7.0 install.

The above got fixed in 2.7.1.

Matthias.
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Matt Pietrek]

Indeed! In my hour or so of testing I just did, 2.7.1 seems much happier,
and appears to handle VIP failover reasonably well in my scenario.

Thanks much!

Matt

[Dave Stevens]

I'm having a similar issue with 2.7.1 installed on ubuntu.

I'm running a cluster of three nodes. I have three consumers consuming
from a non-mirrored, non-exclusive queue. When I take down the node
containing the queue with rabbitmqctl stop_app, occasionally one of
the clients receives the following when attempting to reconnect and re-
declare the queue.

{#method<channel.close>(reply-code=404, reply-text=NOT_FOUND - no
queue 'foo' in vhost '/', class-id=50, method-id=10),null,""}

Is this a race condition caused by the clients all trying to redeclare
the queue at the same time? I'm using the spring-amqp java client,
although I'm probably going to ditch it soon for other reasons.

On Dec 20 2011, 6:58 pm, Matt Pietrek <mpiet...@skytap.com> wrote:
> Indeed! In my hour or so of testing I just did, 2.7.1 seems much happier,
> and appears to handle VIP failover reasonably well in my scenario.
>
> Thanks much!
>
> Matt
>

> On 12/20/11 3:12 PM, "Matthias Radestock" <matth...@rabbitmq.com> wrote:
>
>
>
>
>
>
>
>
>
> >On 20/12/11 22:44, Matt Pietrek wrote:
> >> In addition, in the rabbit@<nodename>.log, I see a whole bunch of spew,
> >> including many instances of this:
>
> >> =WARNING REPORT==== 20-Dec-2011::14:09:17 ===
> >> Non-AMQP exit reason '{{{function_clause,
> >> [{gb_trees,lookup,
> >> [<<150,83,52,247,194,247,140,35,130,167,168,
> >> 35,224,203,183,186>>,
> >> {dict,0,16,16,8,80,48,
> >> {[],[],[],[],[],[],[],[],[],[],[],[],[],
> >> [],[],[]},
> >> {{[],[],[],[],[],[],[],[],[],[],[],[],
> >> [],[],[],[]}}}]},
> >> {rabbit_amqqueue_process,
> >> '-confirm_messages/2-fun-0-',2},
> >> [...]
> >> Again, this is a stock 2.7.0 install.
>
> >The above got fixed in 2.7.1.
>
> >Matthias.
>
> _______________________________________________
> rabbitmq-discuss mailing list

> rabbitmq-disc...@lists.rabbitmq.comhttps://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Dave Stevens]

It is important to note that the queue is durable. I just read after
sending this that none of the clients should have been allowed to
reconnect. I'm curious why two out of the three clients were able to
come up when they apparently should have also received the same error.
I will experiment some more with this knowledge about how durable
queues work. I originally wrote this assuming we would be using HA
queues and I have them on my brain...

[Simon MacMullen]

On 18/01/12 16:32, Dave Stevens wrote:
> I just read after
> sending this that none of the clients should have been allowed to
> reconnect.

None of them should be allowed to declare the queue, yes. So if you can
reliably reproduce this I'd like to hear from you...

Cheers, Simon

--
Simon MacMullen
RabbitMQ, VMware